The reliability of infrastructure that is critical to society's functionality, survival and progression has gained significance for both national security and research because of its large-scale and interdependent nature. However, the theoretical basis of the relatively new research field of critical infrastructure is incomplete and the common parlance about the underlying concepts is ambiguous. This article addresses this issue and presents the results of a substantial review of scientific literature on the concepts of systems, infrastructure and governance. The results demonstrate that the concepts encounter a common challenge in characterising their key elements, structures and processes because of their recursive nature. The multi-level character of critical infrastructure systems provokes governance to systemically address the properties of adaption, emergence and entropy which the complex system-of-systems exhibits. This article contributes with both a conceptual study of the terms system, infrastructure and governance and a detailed review of the state of the art regarding these concepts in the current scientific literature to an enhanced understanding of the theoretical foundations of the associated fields. Subsequent research could interrelate other concepts, such as vulnerability, resilience, sustainability and feedback with the provided state of the art.