Book

Engineering a Safer World: Systems Thinking Applied to Safety

Authors:
Book

Engineering a Safer World: Systems Thinking Applied to Safety

... Escalating complexity of socio-technical systems along with emerging technology-related risks (new and unknown risks) denote an outstanding challenge for conventional system safety approaches. The rising complexity of socio-technical systems inevitably leads to a rise in emerging risks (Leveson, 2016). The effects of these risks in asset management should be studied considering the organization's external and internal context involving human performance and socio-economic as well as socio-cultural considerations. ...
... They perform best on mechanical elements or hardware. Though, they have serious limitations on for e.g., human operators, organizational and social considerations, software program-related aspects, etc. (Leveson, 2016;Underwood et al., 2013). On these arguments, both practitioners and scholars have been interested in relatively new advanced methods based on system theories, namely the Functional Resonance Analysis Method (FRAM) Hollnagel (2012) ...
... Model and Processes (STAMP-System Theoretic Process Analysis (STPA)) (Leveson, 2016), as well as the Risk-Informed Decision-Making Approach (RIDM) processes (Dezfuli et al., 2010c;Gaha et al., 2021;Komljenovic et al., 1 "Uncertainty is an intrinsic part of decisions about the prospective behavior of a complex system over long periods of time. (…). ...
Article
Full-text available
Suffice to say that long-established businesses have their own challenges. Furthermore, accurate systematic methods and tools for managing risks in the context of industry 4.0 are lacking or less efficient, spreading unrealistic awareness of risk (or situational awareness) in various domains where risk management is needed. Conventional methods have their own limits and might not identify all aspects that influence system safety. Once traditional industry challenges are combined with emerging risks along with new systemic and organizational risks as well as cognitive and motivational biases in human logic, there will be the necessity of building thorough Asset Management and Decision Support approaches accounting both for conventional and emerging risk safety management. Hence, innovative, and efficient approaches that can investigate issues from a broad systemic perspective to support asset management practitioners to deal with those threats associated with the complexity of socio-technical systems are of interest. On these grounds, this paper focuses on identifying and analyzing components of risk management approaches especially for new emerging safety risks within industry 4.0 (emerging technology-related risks), as well as the rising of extreme, rare, and disruptive events, at a time of continued uncertainty in the global economy, in conjunction with the highly insecure political situation caused by recent armed conflicts (for e.g., Russia vs Ukraine), and the coronavirus disease pandemic (COVID-19) that might create fatal disturbance of the performance of organizations. We opt for the relatively new methods that have been developed based on system theories, viz. the Functional Resonance Analysis Method (FRAM), the System-Theoretic Accident Model and Processes (STAMP, System Theoretic Process Analysis (STPA)) and the global risk-informed decision-making approach (RIDM) in asset management as the best suited approach for this research. We first discuss the benefits of these methods then outline the possibility of combining them to conduct high-level risk management and decision-making framework. Further research would validate their efficiency and practicality. Therefore, future research initiatives will be devoted to conducting case studies in order to obtain more accurate data.
... The aim of accident models is to identify accident causal factors, and hence determine what measures need to be implemented to avoid similar consequences or reduce their likelihood (Bugalia et al., 2020). The present accident reports are sometimes poorly defined when referring to causes, since accident analyses may focus on finding someone or something to blame: this situation leads to miss the opportunity to learn important lessons to improve system safety (Leveson, 2011). Currently, due to the increase in systems' complexity, many accidents do not result from a linear causal chain, but they are caused by non-trivial socio-technical interactions e.g., human factors, mission profile, equipment, financial pressures, and information that increase the normal operational variability of the system process (Rong and Tian, 2015). ...
... Therefore, other complexoriented accident analysis models seem necessary, possibly relying on systems' thinking. This latter focuses on a combination of thinking about the operation or/and management process related with the analyzed system (Leveson, 2011). More formally, systems thinking consists of three aspects: (i) elements' characteristics; (ii) interconnections between the elements; (iii) systems functional purpose. ...
... The interactions must be established to accomplish the control of the system's behavior by enforcing the safety constraints in its design and operation. The STAMP model is founded on three basic concepts (Leveson, 2011): 1. The safety constraints. ...
Article
Traditional safety risk analysis methods are rooted in event chain modeling and looking for individual points of failure. This approach allowed tremendous improvement in safety management but starts to be difficult to apply when dealing with large-scale systems constituted by a wide number of interactions among technical and social elements. Therefore, systemic safety management poses new challenges, demanding approaches capable of complementing techno-centric investigations with social-oriented analyses. For this purpose, this study adopts the Systems-Theoretic Accident Model and Processes (STAMP) as a new accident causation model based on systems theory. Such a model is the first element to gain a complete understanding of the system at hand, and subsequently to create a set of safety recommendations. STAMP can lead to both the development or evaluation of safety management systems and the identification of leading indicators related to hazards, in order to improve decision-making domains and strengthen accidents/loss analyses. The present research incorporates three basic components of systems theory for STAMP models: constraints, hierarchical control structure, and process loops. These items are meant to allow recognizing causes and preventing potential system failures as well as undesired events. In the proposed model, accidents are examined in terms of the ways controls fail and how they may not allow prevention or detection of hazards. This study proposes a hierarchical safety control structure on a demonstrative use case referred to an industrial plant for gas and oil production, The model consists of system-level safety constraints, and a preliminary investigation of system’s components with the purpose of supporting physical and organizational safety requirements elicitation.
... Yet, failures are very rarely caused only by a single component or operator flaw [42]. Rather, accidents in SSCs derive from complex dynamic processes involving interconnected sociotechnical systems and cannot be explained solely by static chains of failures [25]. Furthermore, the technological advancements deployed for the benefits of safety and the growing adoption of passive safety systems have reduced the number of active SSC failures and, therefore, the ability to model accidental scenarios based on acquired knowledge from past experience [25]. ...
... Rather, accidents in SSCs derive from complex dynamic processes involving interconnected sociotechnical systems and cannot be explained solely by static chains of failures [25]. Furthermore, the technological advancements deployed for the benefits of safety and the growing adoption of passive safety systems have reduced the number of active SSC failures and, therefore, the ability to model accidental scenarios based on acquired knowledge from past experience [25]. PRA methods may then not be sufficient to provide realistic risk assessments as they do not consider dynamic interactions and feedbacks [34]. ...
... In this perspective, System-Theoretic Accident Model and Processes (STAMP) [25] overcomes the limitations of PRA as it allows identifying hazards from the functional interaction among SSCs [23]. STAMP is a qualitative accident modeling framework based on system theory, rather than reliability theory, which treats the system as a whole rather than as separate parts [45]. ...
Article
Full-text available
Accidents may occur as a result of complex dynamic processes in interconnected socio-technical systems. Such accidents cannot be explained solely in terms of static chains of failures. Therefore, the traditional Probabilistic Risk Assessment (PRA) framework, which stands on the consideration that accidents are caused by direct failures or chains of events, is not apt to describe the dynamic behavior of the relevant Systems, Structures and Components (SSCs) and assess the risk. This work proposes a novel framework that embeds i) System-Theoretic Accident Model and Processes (STAMP) principles to guide a qualitative exploration of the SSC threats and hazards, ii) Modeling and Simulation (M&S) to investigate the SSC dynamic behavior during accidental scenarios, and iii) the Goal-Tree Success-Tree Master Logic Diagram (GTST-MLD) framework to assess risk quantitatively. The integration of STAMP, M&S and GTST-MLD allows a systematic analysis to provide risk insights, with due account to the SSC dependencies and interactions, and enables a dynamic assessment of the risk profile. The effectiveness of the proposed framework is shown by means of its application to the safety assessment of Nuclear Batteries (NBs), a unique class of nuclear micro-reactors which is gaining attention as a transportable, flexible, affordable, and distributed low-carbon power source.
... Its theoretical basis comes from system theory and control theory [60]. The basic structure of STAMP includes security constraints, a hierarchical control structure, and a process model [61]. The core idea of STAMP theory is to transform a safety problem into a control problem. ...
... Process models are derived from cybernetics to show the rules that controllers follow in a specific control process, i.e., what action to take under what circumstances and what following steps to take based on the feedback information. STAMP theory considers conflicts between process models and actual processes as an important cause of accidents [61]. Figure 1 shows a classic control loop: the controller obtains the current parameters of the system through sensors and adjusts the behavior of the actuators according to the parameter changes to ensure that the values of the parameters remain within acceptable limits, even when the system is disturbed. ...
... The CAST method investigates accidents by analyzing factors such as context, communication and coordination, control, and mental models, based on the STAMP structure. The "context" is used to analyze the environment in which the system components are located at that time; "communication and coordination" are used to analyze the relationship between the system components and to show it in the form of a diagram; "control" is used to analyze the failed control actions during the accident; the "mental model" is used to analyze the causes of human error in the system [61]. Leveson classified the causes of accidents into four categories: improper controller operation, improper actuator operation, control process failure, and inaccurate, missing, or delayed information feedback . ...
Article
Full-text available
Emergency management research is used to deal with the increasing number of extreme weather threats in urban areas. This paper uses causal analysis based on systems theory (CAST) to review the subway water ingress accident and the government’s emergency management actions in Zhengzhou, Henan Province, during the heavy rainstorm disaster on 20 July 2021. The aims of this article are to establish safety control structures at both the enterprise level and the government level, and to systematically analyze the problems in emergency management in Zhengzhou City. Our analysis found that the construction of disaster prevention facilities restricted emergency management. Therefore, we suggest that enterprises and governments not only pay attention to emergency management, but also to the construction of disaster prevention facilities. This article also points out that the system of chief executive responsibility that is implemented in China is becoming a double-edged sword in emergency management. Our study makes recommendations for enhancing the capacities of emergency management, points out the shortcomings of the existing emergency management structure, and provides knowledge gained for future emergency management research.
... Major accidents in safety-critical systems might be investigated for a number of reasons, both inside the participating organizations and in the larger context of dysfunctional relationships between them (S. Lee et al., 2017). Accidents, according to sociotechnical systems theory, are caused by highly interactive and collaborative processes, as well as the impact of decision-makers and policymakers at all levels of society (Leveson, 2016;Rising and Leveson, 2018). ...
... In other words, interactions between IoT devices, software, and humans are viewed as a type of social connection (Atzori et al., 2012). To manage the behavior of lower-level systems, each hierarchical level must place safety limitations on the activities of the level underneath it (Leveson, 2016). ...
... The concept of emergence implies that at a given level of complexity, some properties of that level (which have emerged at that level) are irreducible to any other properties of that level (Leveson, 2018). Because safety can only be determined in the context of the whole system, it is clear that safety is an emergent property of systems (Leveson, 2016 In STAMP, safety is treated as an emergent property at each of these hierarchy levels that arises as a result of the interaction of the system components with their environment when they interact. ...
Thesis
Full-text available
New technology is making fundamental changes in the etiology of accidents and is creating a need for changes in the explanatory mechanisms used. We need a better and less subjective understanding of why accidents occur and how to prevent future ones. The most effective models will go beyond assigning blame and instead help engineers to learn as much as possible about all the factors involved, including those related to social and organizational structures. In our increasingly complex and interrelated societal structure, responsibility for safety is shifting from the individual to the government. Individuals no longer have the ability to control the risks around them and are demanding that government assume greater responsibility for controlling behavior through laws and various forms of oversight and regulation. In this study, two taxonomically distinct methods: "Systems Theoretic Accident Model and Process" based "System-Theoretic Process Analysis" (STAMP-STPA) and Design with Intent (DwI), have been used to develop a framework for road crash mitigation measures in the context of Bangladesh. Where most traditional accident models view accidents as resulting from a chain or sequence of events, STAMP-STPA and DwI focus on the overall structure of the system: from the design level to the operation level. One road collision, the Uthalia accident, has been used as a case study to develop the framework in this research. STAMP-STPA is a structured, constrained, and systematic approach. In contrast, DwI is nonstructured and unconstrained, mainly in the form of brainstorming, idea-generation, or ideation sessions where participants are asked to generate concepts, individually or together, in response to a design problem. Both approaches aim to find the countermeasure or preventive measure for hazards. Accimap is a prevalent sociotechnical approach for accident analysis; however, STAMPSTPA has subtle advantages as it adequately indicates the irregularities or non-existence of the control-feedback loop in the system. On the other hand, DwI is an unconventional but handy method for analyzing and developing recommendations regarding system or design failure due to road collisions. This method allows for creative thinking and brainstorming. Traditional accident analysis methods are localized since they provide a site-specific solution. In comparison to traditional methods, STAMP-STPA and DwI have the potential to provide solutions from a broader perspective that can be transferred or applied to other sites. Firstly, an Actor Map for the Uthalia accident has been generated. Secondly, the produced actor map is used to build the STAMP and STPA model and perform DwI sessions for the selected road collision study. Information regarding the accident event has been collected from popular media reports, accident investigation reports, and local interviews to conduct a thorough investigation. A total of 171 actors have been identified across the eight levels of the Bangladesh road safety system. Actors at different levels have been found to be contributing to the overall road safety system. STAMP-STPA analysis of the case study revealed that while investigating accidents, the blame should not always be only on the end users' level; instead, the faults across different hierarchies in the system need to be identified. Exploring the lackings in the overall road safety system through this developed framework will guide policymakers to build a safer road infrastructure for the road users. Based on the developed STAMP-STPA model and DwI method, a number of recommendations corresponding to the collision events were proposed, which were later aggregated. These recommendations have been provided corresponding to different events occurring at different levels of the system, which emphasizes performing road safety system reformation. It is found that almost all of the recommendations or countermeasures found from DwI fall under Level 5: Operating Process and Environment indicating that DwI focuses on lower-level solutions, whereas STAMP-STPA offers solutions from a broader perspective: from the international context to the operational level. The aggregation of the proposed recommendations from the STAMP-STPA and DwI method were validated by three different subject matter experts with vast experience in the safety domain. It should be noted that methods like STAMP-STPA have primarily been applied to lane-based, homogeneous traffic systems; however, the traffic situation in Bangladesh is very different and, arguably, more complex, given the wide variety of road and road user types, as well as the chaotic nature of the system, when compared to those seen in high-income countries. More studies are needed for validation purposes in order to establish the use of sociotechnical methods in Bangladesh.
... Therefore, organizations can be interpreted as socio-technical systems due to the interrelated and interdependent structures where social and technical aspects remain intertwined [4,5]. Currently, accident reports are sometimes poorly written when referring to causes, since the analysis frequently stops after finding someone or something to blame and the opportunity to learn important lessons from the accident is lost [6,7]. In addition, it is left aside that these accidents involve different factors, e.g., human factors, mission, equipment, financial pressures, reputation, and information that increase the normal operational variability of the system [8]. ...
... One interesting stream of research in this sense is built up around the systems theoretic accident modelling and process (STAMP) model, which is rooted in control theory and previous experience of hierarchical safety control actions [8,11]. The STAMP model has been used to identify the systemic factors behind accident occurrence, as it provides the basis for maximizing learning from events [6]. In STAMP, an accident is regarded as a complex process, not just the sum of stand-alone events. ...
... In this framework, preventing future accidents requires shifting from a focus on preventing failures to the broader goal of designing and implementing controls to understand why an accident occurred and determining why the previous controls were ineffective [37]. The STAMP is based on these principles and its foundations are on three basic constructs, here introduced with relevance for railway operations [6,38,39]. ...
Article
Full-text available
Post mortem incident investigations are vital to prevent the occurrence of similar events and improve system safety. The increasing interactions of technical, human and organizational elements in modern systems pose new challenges for safety management, demanding approaches capable of complementing techno-centric investigations with social-oriented analyses. Hence, traditional risk analysis methods rooted in event-chain reactions and looking for individual points of failure are increasingly inadequate to deal with system-wide investigations. They normally focus on an oversimplified analysis of how work was expected to be conducted, rather than exploring what exactly occurred among the involved agents. Therefore, a detailed analysis of incidents beyond the immediate failures extending towards socio-technical threats is necessary. This study adopts the system-theoretic accident model and process (STAMP) and its nested accident analysis technique, i.e., causal analysis based on systems theory (CAST), to propose a causal incident analysis in the railway industry. The study proposes a hierarchical safety control structure, along with system-level safety constraints, and detailed investigations of the system’s components with the purpose of identifying physical and organizational safety requirements and safety recommendations. The analysis is contextualized in the demonstrative use of a railway case. In particular, the analysis is instantiated for a 2011 incident in the United Kingdom (UK) railway system. Hence, the CAST technique requires information regarding incidents, facts and processes. Therefore, the case study under analysis provided the information to analyze the accidents based on system theory, in which the results of the analysis prove the benefits of a CAST application to highlight criticalities at both element- and system-level, spanning from component failure to organizational and maintenance planning, enhancing safety performance in normal work practices.
... STPA is a hazard analysis method based on the idea of System-Theoretic Accident Model and Processes (STAMP) (Leveson 2011), in which the unsafe interactions between components are considered important contributors to an accident. Other types of hazard identification methods, Hazard and Operability Study (HAZOP) and Failure Mode and Effects Analysis (FMEA), focus on component or event failures. ...
... The steps of STPA are presented below. More information, including its advantages, disadvantages, and limitations, can be found in previous studies (Leveson 2011;Yang and Utne 2022). They are not further discussed here due to limited article length. ...
... In terms of losses, anything valuable to stakeholders should be included. System-level hazards are defined as the system's state to possibly lead to a loss under the worst-case environment (Leveson 2011). ...
Conference Paper
Full-text available
Operations with multiple autonomous marine systems (AMS) are becoming increasingly popular for a variety of applications. Some traditional challenges associated with single AMS operations may be relieved by the presence of a second AMS. However, the operation of multiple AMS may bring new challenges, possibly caused by the unsafe interaction between the participating AMS. Hence, this needs to be further analyzed to improve their safe and reliable operations. However, most previous risk-related works on AMS focuses on the operation of a single AMS and ignores the unsafe interaction between different participating AMS. The current study focuses on the operation with multiple AMS, aiming at identifying the potential hazards during the operation. System theoretic process analysis (STPA) is applied to capture the interaction between each AMS and the interaction between AMS and human operators. An integrated USV-AUVs operation is used as a case study in this study. The analysis results are expected to support future planning of operations with multiple AMS and increase awareness of the operators. In addition, it is expected that the analysis results and conclusions can also be used to develop an online risk model which can capture the rapid change of operating conditions of operations with multiple AMS and then enhance the intelligence of the AMS, its situation awareness, and decision-making during operation.
... At most, the aforementioned design concepts and guidelines help build a buffer to technical failures, but do not support the technical systems engineering that could resolve core issues in the original designs. This suggest that the scope of system engineering for safety has to be enlarged [21]. ...
... Principles behind, for instance, (CS)E must be adopted by engineers in engineering processes producing complex, safety-critical socio-technical systems [21]. This extension of the scope of human-centred design to engineering processes suggests that system engineers should ensure that engineering activities take care to address the implications on operations by complex socio-technical systems [22]. ...
Article
Full-text available
Commercial deployment of maritime autonomous surface ships (MASSs) is close to becoming a reality. Although MASSs are fully autonomous, the industry will still allow remote operations centre (ROC) operators to intervene if a MASS is facing an emergency the MASS cannot handle by itself. A human-centred design for the associated emergency response systems will require attention to the ROC operator workplace, but also, arguably, to the behaviour-shaping constraints on the engineers building these systems. There is thus a need for an engineer-centred design of engineering organisations, influenced by the current discourse on human factors. To contribute to the discourse, think-aloud protocol interviewing was conducted with well-informed maritime operators to elicit fundamental demands on cognition and collaboration by maritime autonomy emergency response systems. Based on the results, inferences were made regarding both design factors and methodological choices for future, early phase engineering of emergency response systems. Firstly, engineering firms have to improve their informal gathering and sharing of information through gatekeepers and/or organisational liaisons. To avoid a too cautious approach to accountability, this will have to include a closer integration of development and operations. Secondly, associated studies taking the typical approach of exposing relevant operators to new design concepts in scripted scenarios should include significant flexibility and less focus on realism.
... This definition takes safety, health, wellbeing and fulfilment under a single care, and safety management as an integrative attribute of the human resource management system of the project organisation. On this basis, drawing further from systems thinking (Rasmussen, 1983;Dekker, 2006Dekker, , 2011Hollnagel et al. 2006;Leveson 2016Leveson [2012), a set of IPD-embedded safety capabilities are identified in Table 1. Recognise productivity comes from a healthy and self-motivated workforce (in contrast to the exploitation approach) (Levitt and Samelson, 1993;Loosemore et al, 2003;Oxenburgh et al, 2004;Chan et al, 2022) Caring Attend to team member's characteristics; understand and accommodate different strengths, needs and difficulties (in contrast to the task-centred, command-andcontrol approach) (Hale et al, 2010;Lingard and Francis, 2009) Problem-framing Be able to comprehend a situation of ambiguity and uncertainty, frame the problems that point to effective solutions (Kvan and Gao, 2004;Snowden and Boone, 2007;Walker et al., 2017) Engagement Involve, consult and develop employees (instead of hire-and-fire) (Westrum, 1993;Lawani et al., 2017) Inclusiveness Embrace differences and diversity; motivate team members to participate (Lowe, 201;Zwetsloot et al., 2013) Mindfulness Stay authentic to the meaning of the work (Weick et al., 1999) No-blame See mistakes as learning opportunities; seek to solve problem rather than lay blames (Dekker, 2006;Love and Smith, 2016;Walker et al., 2017) Forgiveness Capability of absorbing the consequences of teammates' mistakes; go an extra mile to complement; keep the system working despite others' errors (Strang, 2001;Senge, 2006;Caldwell and Dixon, 2010) Resilience Be responsive to emergent issues (in contrast to non-reflexive practice); respond resiliently to interruptions to keep the system running as usual (Senge, 2006;Winwood et al, 2013;Turner et al, 2016;Abankwa,et al, 2021) Integration ...
... This definition takes safety, health, wellbeing and fulfilment under a single care, and safety management as an integrative attribute of the human resource management system of the project organisation. On this basis, drawing further from systems thinking (Rasmussen, 1983;Dekker, 2006Dekker, , 2011Hollnagel et al. 2006;Leveson 2016Leveson [2012), a set of IPD-embedded safety capabilities are identified in Table 1. Recognise productivity comes from a healthy and self-motivated workforce (in contrast to the exploitation approach) (Levitt and Samelson, 1993;Loosemore et al, 2003;Oxenburgh et al, 2004;Chan et al, 2022) Caring Attend to team member's characteristics; understand and accommodate different strengths, needs and difficulties (in contrast to the task-centred, command-andcontrol approach) (Hale et al, 2010;Lingard and Francis, 2009) Problem-framing Be able to comprehend a situation of ambiguity and uncertainty, frame the problems that point to effective solutions (Kvan and Gao, 2004;Snowden and Boone, 2007;Walker et al., 2017) Engagement Involve, consult and develop employees (instead of hire-and-fire) (Westrum, 1993;Lawani et al., 2017) Inclusiveness Embrace differences and diversity; motivate team members to participate (Lowe, 201;Zwetsloot et al., 2013) Mindfulness Stay authentic to the meaning of the work (Weick et al., 1999) No-blame See mistakes as learning opportunities; seek to solve problem rather than lay blames (Dekker, 2006;Love and Smith, 2016;Walker et al., 2017) Forgiveness Capability of absorbing the consequences of teammates' mistakes; go an extra mile to complement; keep the system working despite others' errors (Strang, 2001;Senge, 2006;Caldwell and Dixon, 2010) Resilience Be responsive to emergent issues (in contrast to non-reflexive practice); respond resiliently to interruptions to keep the system running as usual (Senge, 2006;Winwood et al, 2013;Turner et al, 2016;Abankwa,et al, 2021) Integration ...
Preprint
Full-text available
This article is aimed to introduce the potential of Elinor Ostrom and colleagues' Institutional Analysis and Development (IAD) framework to the context of construction safety management as an actionable theory in guiding the dynamic formation of action situations in projects that incentivise and nurture individual safety capabilities for integrated project delivery (IPD) practice. A case is presented to illustrate how the IAD framework can work for this purpose. A set of safety capabilities is defined through an in-depth review of the IPD and safety literature. The reflective analysis highlights three key perspectives in mobilising the IAD framework for safety capabilities development in IPD context: (1) a dynamic power division between top-down design and bottom-up development of institutions; (2) an inclusive project front-end that involves actors in problem-framing activities; and (3) capabilities development as a process of shifting logics. Practically, the results inform effective training and coaching of practitioners at the teambuilding stage of IPD projects to configure safety as an integral part of the project system. Theoretically, the research contributes to the development of institutional theories in the project safety management context.
... STAMP is a model of accident causation technique based on systems theory and the idea that all of the parts of a system are interconnected and their behavior affects one another [26]. The basic elements of STAMP are accidents, hazards, and safety constraints as well as a hierarchical safety control structure block diagram. ...
... Hazards under the right environmental conditions lead to a loss. STAMP uses fewer than a dozen high-level hazards [26]. Safety constraints are generated based on a hazard and are used as high-level safety requirements that shall be met by the system. ...
Preprint
Full-text available
This research considers the problem of identifying safety constraints and developing Run Time Assurance (RTA) for Deep Reinforcement Learning (RL) Tactical Autopilots that use neural network control systems (NNCS). This research studies a specific use case of an NNCS performing autonomous formation flight while an RTA system provides collision avoidance and geofence assurances. First, Systems Theoretic Accident Models and Processes (STAMP) is applied to identify accidents, hazards, and safety constraints as well as define a functional control system block diagram of the ground station, manned flight lead, and surrogate unmanned wingman. Then, Systems Theoretic Process Analysis (STPA) is applied to the interactions of the the ground station, manned flight lead, surrogate unmanned wingman, and internal elements of the wingman aircraft to identify unsafe control actions, scenarios leading to each, and safety requirements to mitigate risks. This research is the first application of STAMP and STPA to an NNCS bounded by RTA.
... This definition takes safety, health, wellbeing and fulfilment under a single care, and safety management as an integrative attribute of the human resource management system of the project organisation. On this basis, drawing further from systems thinking (Rasmussen, 1983;Dekker, 2006Dekker, , 2011Hollnagel et al. 2006;Leveson 2016Leveson [2012; Goh, 2020), a set of IPD-embedded safety capabilities are identified in Table 1. Recognise productivity comes from a healthy and self-motivated workforce (in contrast to the exploitation approach) (Levitt and Samelson, 1993;Loosemore et al, 2003;Oxenburgh et al, 2004;Chan et al, 2022) Caring Attend to team member's characteristics; understand and accommodate different strengths, needs and difficulties (in contrast to the task-centred, command-andcontrol approach) (Hale et al, 2010;Lingard and Francis, 2009) Problem-framing Be able to comprehend a situation of ambiguity and uncertainty, frame the problems that point to effective solutions (Kvan and Gao, 2004;Snowden and Boone, 2007;Walker et al., 2017) Engagement Involve, consult and develop employees (instead of hire-and-fire) (Westrum, 1993;Lawani et al., 2017) Inclusiveness Embrace differences and diversity; motivate team members to participate (Lowe, 201;Zwetsloot et al., 2013) Mindfulness Stay authentic to the meaning of the work (Weick et al., 1999) No-blame See mistakes as learning opportunities; seek to solve problem rather than lay blames (Dekker, 2006;Love and Smith, 2016;Walker et al., 2017) Forgiveness Capability of absorbing the consequences of teammates' mistakes; go an extra mile to complement; keep the system working despite others' errors (Strang, 2001;Senge, 2006;Caldwell and Dixon, 2010) Resilience Be responsive to emergent issues (in contrast to non-reflexive practice); respond resiliently to interruptions to keep the system running as usual (Senge, 2006;Winwood et al, 2013;Turner et al, 2016;Abankwa,et al, 2021) Integration ...
... This definition takes safety, health, wellbeing and fulfilment under a single care, and safety management as an integrative attribute of the human resource management system of the project organisation. On this basis, drawing further from systems thinking (Rasmussen, 1983;Dekker, 2006Dekker, , 2011Hollnagel et al. 2006;Leveson 2016Leveson [2012; Goh, 2020), a set of IPD-embedded safety capabilities are identified in Table 1. Recognise productivity comes from a healthy and self-motivated workforce (in contrast to the exploitation approach) (Levitt and Samelson, 1993;Loosemore et al, 2003;Oxenburgh et al, 2004;Chan et al, 2022) Caring Attend to team member's characteristics; understand and accommodate different strengths, needs and difficulties (in contrast to the task-centred, command-andcontrol approach) (Hale et al, 2010;Lingard and Francis, 2009) Problem-framing Be able to comprehend a situation of ambiguity and uncertainty, frame the problems that point to effective solutions (Kvan and Gao, 2004;Snowden and Boone, 2007;Walker et al., 2017) Engagement Involve, consult and develop employees (instead of hire-and-fire) (Westrum, 1993;Lawani et al., 2017) Inclusiveness Embrace differences and diversity; motivate team members to participate (Lowe, 201;Zwetsloot et al., 2013) Mindfulness Stay authentic to the meaning of the work (Weick et al., 1999) No-blame See mistakes as learning opportunities; seek to solve problem rather than lay blames (Dekker, 2006;Love and Smith, 2016;Walker et al., 2017) Forgiveness Capability of absorbing the consequences of teammates' mistakes; go an extra mile to complement; keep the system working despite others' errors (Strang, 2001;Senge, 2006;Caldwell and Dixon, 2010) Resilience Be responsive to emergent issues (in contrast to non-reflexive practice); respond resiliently to interruptions to keep the system running as usual (Senge, 2006;Winwood et al, 2013;Turner et al, 2016;Abankwa,et al, 2021) Integration ...
Chapter
Full-text available
This Chapter explores the potential of Ostrom's Institutional Analysis and Development (IAD) framework as an actionable theory in guiding safety capability development for integrated project delivery (IPD). It does so through the dynamic formation of action situations that incentivise and nurture the desirable capabilities for collaborative practice. A worked example is presented to illustrate how the IAD framework can work for this purpose. A set of safety capabilities is defined through an in-depth review of the IPD and safety literature. The reflective analysis highlights three key perspectives in mobilising the IAD framework for IPD-enabling safety capabilities development: (1) a sensible and dynamic power division between top-down design and bottom-up development of institutions; (2) an inclusive project front-end that involves actors in problem-framing activities; and (3) capabilities development as a process of shifting logics. Practically, the results inform training and coaching of practitioners at the teambuilding stage of IPD projects to configure safety as an integral part of the project system. Theoretically, the research contributes to the development of institutional theories in the project safety management context.
... Research has concluded that there is a need for investigations to look beyond individual actions to understand how a crash occurs so as to identify the underlying causes and to establish interventions that prevent the reoccurrence of a crash (Rasmussen, 1997;Perrow, 1999;Leveson, 2004;Leveson, 2012; The presence of laws, such as those found in the Heavy Vehicle National Law (2012) and the Road Traffic (Vehicles) Act (2012) and Regulations (2014), the latter being the Compliance & Enforcement legislation, help guide and influence the decision-making processes to develop the required rules, procedures and risk management processes to ensure compliance (Larsson et al., 2010;Schobel and Manzey, 2011;Foster et al., 2019). However, it has been said that the rules, laws and procedures can never account for all uncertainties and scenarios, so can never be sufficient for every context. ...
... the lens needs to focus on management systems and the decisions and actions of other actors within the system such as Government and Regulators (Rasmussen, 1997;Leveson, 2004;Leveson, 2012;Toft et al., 2012;Newnam and Goode, 2015). A safe system depends on actors involved in the heavy vehicle transport industry working in collaboration, showing concern for the safety of others anticipating threats to safety, and contributing to safety improvements (Stucky and Lamontagne, 2005). ...
... To address this gap, the current study aimed at investigating the potential patterns of safety climate change over time, and the antecedents of these change patterns based on a longitudinal study design. These aims are based on a key premise that workplace safety is not a static state but an emergent property (Leveson, 2012), and so is safety climate (Lee et al., 2019a, b). ...
Article
Full-text available
Safety climate evolves in reflection of the effectiveness of organizational safety management efforts. The present study identified patterns of safety climate change over time and examined the role of error disclosure climate and counterfactual sharing in relation to safety climate change patterns. Online surveys were administered in a Chinese hospital three times at approximately 1-month intervals; the final sample included 451 healthcare workers nested within 62 teams. A latent growth mixture modeling approach was adopted to identify representative patterns of safety climate change at both individual and team levels and the predictors of those patterns. Three patterns of safety climate trajectories, declining (16%), improving (39%), and maintaining (45%), were identified at the individual level. Positive error disclosure climate and counterfactual sharing were significantly associated with increased probability of membership in the improving and maintaining trajectories compared to the declining trajectories. Counterfactual sharing mediated the relation between error disclosure climate and membership of safety climate trajectories. At the team level, two patterns of safety climate change, declining (15%) and improving (85%), were identified. Team counterfactual sharing was significantly associated with increased probability of membership in the improving trajectories compared to the declining trajectories. The current study demonstrated that an open and non-judgmental culture and the practice of sharing errors can contribute to improving safety climate over time.
... Safety is sometimes viewed within the Artificial Intelligence community as only a matter of achieving high reliability (e.g., overcoming adversarial examples). However, as Nancy Leveson and others have argued [11], reliability is neither the only necessary condition nor a sufficient condition for safety. Instead, one needs a system-theoretic approach to understanding hazards and overcoming issues arising in the 'outer-loops'. ...
Preprint
Achieving safe and robust autonomy is the key bottleneck on the path towards broader adoption of autonomous vehicles technology. This motivates going beyond extrinsic metrics such as miles between disengagement, and calls for approaches that embody safety by design. In this paper, we address some aspects of this challenge, with emphasis on issues of motion planning and prediction. We do this through description of novel approaches taken to solving selected sub-problems within an autonomous driving stack, in the process introducing the design philosophy being adopted within Five. This includes safe-by-design planning, interpretable as well as verifiable prediction, and modelling of perception errors to enable effective sim-to-real and real-to-sim transfer within the testing pipeline of a realistic autonomous system.
... There are numerous approaches, techniques, and levels of rigor in carrying out a hazard analysis and a risk assessment, and we refer to existing literature for further detail [26]. Our approach is reminiscent of a preliminary System Hazard Analysis (SHA) that subsumes a further categorization and prioritization of the hazards across each "subsystem" 6 (i.e., Subsystem Hazard Analysis). ...
Preprint
Full-text available
Codex, a large language model (LLM) trained on a variety of codebases, exceeds the previous state of the art in its capacity to synthesize and generate code. Although Codex provides a plethora of benefits, models that may generate code on such scale have significant limitations, alignment problems, the potential to be misused, and the possibility to increase the rate of progress in technical fields that may themselves have destabilizing impacts or have misuse potential. Yet such safety impacts are not yet known or remain to be explored. In this paper, we outline a hazard analysis framework constructed at OpenAI to uncover hazards or safety risks that the deployment of models like Codex may impose technically, socially, politically, and economically. The analysis is informed by a novel evaluation framework that determines the capacity of advanced code generation techniques against the complexity and expressivity of specification prompts, and their capability to understand and execute them relative to human ability.
... The main goal of STPA is to consider both component failure and unsafe interactions of system components on the hazard analysis [9], including the human component and its behavior with the designed system. ...
Conference Paper
The earlier phases of any product development greatly influence its life cycle, especially in the aerospace field. Therefore, precise requirements are critical for good acquisition/development contract execution. Firstly, this study has made use of OPM (Object Process Methodology) to model the current Brazilian Air Force Policy for aerospace products' life cycle and a robust hazard analysis technique (STPA-System-Theoretic Accident Model and Processes) to investigate the causal factors which lead to negative impacts on the contract elaboration process for military aerospace products in Brazil. STPA uses System Theory to model any process as a feedback control structure. Focusing on the minimization of losses, the method considers the hazards, safety constraints, unsafe control actions, and causal factors. Based on that, it proposes requirements (which can be understood as recommendations), showing a path throughout the earlier phases of the Brazilian military aerospace products life cycle to improve the contract elaboration process.
... Other tourism scholars identify the patterns of safety management from the perspective of a system, adopting the safety system theory (Leveson 2011;Lower et al., 2018). Previous literature suggests that the safety system elements involve: human safety capabilities; the safety and reliability of equipment and environments; safety functions of the energy production process; and safety information flow (Xie et al., 2021). ...
Article
Full-text available
This research identifies safety practices to be adopted by organizations of peer-to-peer accommodation for different segments of tourists in a pandemic context. More specifically, it identifies the profiles of tourists based on their opinions on the safety practices they expect to find when booking peer-to-peer accommodation. Results from a Multiple Correspondence Analysis (MCA) and Cluster Analysis applied to a sample of 864 prospective tourists suggest two prominent dimensions of safety practices: information and hygiene, and protection; and four types of tourist segments: concerned tourists, indifferent tourists, forewarned tourists, and confident tourists. While the concerned tourists value all safety practices most, the indifferent tourists do not require access to information about safety measures, although they do want information on the Covid-19 regulations at their destination. The forewarned tourists attach the least importance to aspects such as information and hygiene, and the greatest to the protection aspect. In contrast, the confident tourists value all information practices and safety measures but do not appreciate the protection aspects. These results will be helpful for peer-to-peer accommodation providers wishing to customize services during and after the Covid-19 period.
... Nancy Leveson introduced Systems-Theoretic Accident Model and Processes (STAMP) [9] by looking at safety as a control problem. She stated that safety can be controlled by applying sufficient constraints to a system. ...
Article
Full-text available
The technological development ongoing in the maritime industry is making the ground for remotely and even autonomously operated vessels in the future. This is a result of increased data collection, processing and inter-connectivity capabilities. The industry is working towards increased safety, improved efficiency of the ship’s operation, improved environmental performance and a more cost-effective shipping. New technologies are developed in order to reach these goals, and DNV as a Class society is developing frameworks for assurance of such systems. The certification of ships and vessels with a high degree of automation or autonomy needs an increased focus on software, an understanding of the human-to-machine interaction and the resulting ability to solve complex operations in a secure way. In this paper, a method for high-level risk analysis of the safety aspects of autonomous vessels combined with automatic simulation-based testing of a control system, is proposed.
... An important issue in CPSoS engineering is emergence [22]. Emergence is a complex, somewhat philosophical concept, that has several interpretations around a central common understanding of "appearing as a result of interactions of subsystems". ...
Article
Characterinsing the nature of cybephysical systems is not easy task. What are core aspects and what are not? This is especially tricky in systems-of-systems aggregates. Some EU-funded cyberphysical systems projects have performed a roadmapping exercise over the domain of Cyber-Physical Systems-of-Systems. In particlular, the EU-CPSoS project roadmap has identified t hree m ajor c hallenges a nd e leven r esearch a nd i nnovation p olicies t hat shall be addressed to solve the three challenges. The third core challenge addresses Cognitive Cyber-physical Systems of Systems. In this article we address the role that knowledge and cognition are to play in future cyber-physical systems of systems from a life-cycle perspective of high autonomy systems.
... Hazards exist because they are inevitable (elements of hazards must be used in a system), and caused by inadequate safety considerations. Leveson (2011) defines a hazard as a condition that, together with the worst set of environmental conditions, will lead to an accident (loss). ...
Book
The challenges to which contemporary building design needs to respond grow steadily. They originate from the influence of changing environmental conditions on buildings, as well as from the need to reduce the impact of buildings on the environment. The increasing complexity requires the continual revision of design principles and their harmonisation with current scientific findings, technological development, and environmental, social, and economic factors. It is precisely these issues that form the backbone of the thematic book, Sustainable and Resilient Building Design: Approaches, Methods, and Tools. The purpose of this book is to present ongoing research from the universities involved in the project Creating the Network of Knowledge Labs for Sustainable and Resilient Environments (KLABS). The book starts with the exploration of the origin, development, and the state-of-the-art notions of environmental design and resource efficiency. Subsequently, climate change complexity and dynamics are studied, and the design strategy for climate-proof buildings is articulated. The investigation into the resilience of buildings is further deepened by examining a case study of fire protection. The book then investigates interrelations between sustainable and resilient building design, compares their key postulates and objectives, and searches for the possibilities of their integration into an outreaching approach. The fifth article in the book deals with potentials and constraints in relation to the assessment of the sustainability (and resilience) of buildings. It critically analyses different existing building certification models, their development paths, systems, and processes, and compares them with the general objectives of building ratings. The subsequent paper outlines the basis and the meaning of the risk and its management system, and provides an overview of different visual, auxiliary, and statistical risk assessment methods and tools. Following the studies of the meanings of sustainable and resilient buildings, the book focuses on the aspects of building components and materials. Here, the life cycle assessment (LCA) method for quantifying the environmental impact of building products is introduced and analysed in detail, followed by a comprehensive comparative overview of the LCA-based software and databases that enable both individual assessment and the comparison of different design alternatives. The impact of climate and pollution on the resilience of building materials is analysed using the examples of stone, wood, concrete, and ceramic materials. Accordingly, the contribution of traditional and alternative building materials to the reduction of negative environmental impact is discussed and depicted through different examples. The book subsequently addresses existing building stock, in which environmental, social, and economic benefits of building refurbishment are outlined by different case studies. Further on, a method for the upgrade of existing buildings, described as ‘integrated rehabilitation’, is deliberated and supported by best practice examples of exoskeleton architectural prosthesis. The final paper reflects on the principles of regenerative design, reveals the significance of biological entities, and recognises the need to assign to buildings and their elements a more advanced role towards natural systems in human environments.
... STAMP is a continuous control task to place the required constrains in order to ensure that the system will work to safe adjustments (Leveson, 2004). It treats safety as a control problem of the system and views systems as hierarchical structures with multiple control levels that each one represents constrains on the activities of the level below (Leveson, 2011). ...
Conference Paper
The paper presents a systematic analysis approach for the early stages of safety processes in large hydrocarbon fuel tanks, known as Systems Theoretic Early Concept Analysis (STECA), the method is based on the STAMP (Systems Theoretic Accident Model and Process) accident causation model. STAMP considers safety as emergent property and therefore every tool based on STAMP, including STECA, considers each system as a whole. STECA method was applied to oil storage tank farms. Results are presented and discussed, and it is concluded that the present systematic analysis method helps to identify the missing information that the procedures should have or safeguards that should be established in order to avoid major accidents, to improve safety measures and to assist the analysis in these aspects.
... This is in contrast with a belief in safety that is merely created by being compliant with technical specifications and static procedures [50]. This shifts the focus from only controlling safe outcomes to the potential to create an understanding of the adaptive capacity of systems and their interactions, fighting analytical reductionism [51][52][53]. ...
Article
Full-text available
Despite their undisputed potential, the uptake of collaborative robots remains below expectations. Collaborative robots (cobots) are used differently from conventional industrial robots. The current safety focus of collaborative workspaces is predominantly on the technological design; additional factors also need to be considered to cope with the emerging risks associated with complex systems. Cobot technologies are characterized by an inherent tradeoff between safety and efficiency. They introduce new, emergent risks to organizations and can create psychosocial impacts on workers. This leads to a confusing body of information and an apparent contradiction about cobot safety. Combined with a lack of safety knowledge, this impedes the introduction of cobots. A multi-step methodology was used, including a literature review and conceptual modeling. This article argues for the need for a system-wide safety awareness readiness assessment in the consideration phase of cobot implementation to alleviate the knowledge deficit and confusion. This work will benefit both researchers and practitioners. In addition, it defends the appropriateness of a maturity grid model for a readiness assessment tool. The building blocks for an easy-to-use and practically applicable tool are proposed, as well as an agenda for the next steps.
... Semi-formal methods rely on system models, (e.g., control structure diagrams or UML diagrams) to guide the user through hazard identification. Examples are STPA [25] and HAZOP-UML [10]. These methods provide a more rigorous analysis than approaches which are based on human reasoning alone. ...
Conference Paper
Full-text available
The use of human-robot collaboration (HRC) systems requires a risk assessment prior to commissioning. Simulations of collaborative workflows can support the risk assessment process and allow users to identify potential hazards at early stages of development. However, simulation-based risk assessment still has a number of open challenges, including modeling human workers, choosing appropriate risk metrics and creating relevant test scenarios. This paper presents an approach that addresses these challenges by using motion capture data, collision force estimation, and search algorithms. Preliminary results are presented and future research challenges are discussed. In the long term, the presented approach will be further developed into a tool for the simulator ABB RobotStudio.
... In System-Theoretic Process Analysis (STPA), the system is represented by a control structure and the hazardous conditions are generated by the lack, the presence or the improper timing of the control actions [57]. After the identification of the Unsafe Control Actions (UCAs), the safety engineer duty is to identify the underlying factors causing the occurrence of the UCAs. ...
Chapter
Cyber-physical systems are the technical foundation of our modern world. They combine a cyber-part (software-based computer control) and a physical part (real system, such as a car, an airplane, and a cardiac pacemaker). These software-controlled systems allow nearly unlimited functionality, flexible functional changes, and cost-effective development and implementation.
Article
This paper presents combining MBSE (Model‐Based System Engineering) and STPA (Systems‐Theoretic Process Analysis) to mitigate security risks at an early stage of system development and to increase agility when developing or modifying architectures. The MBSE approach states that the systems development process should have a system model or a set of models as the unique source of truth. From the system model or a set of models, systems engineers of different specialties should be able to extract the information needed to perform their job. However, some specialties usually create their artefact apart from the model to perform the analysis, breaking the premises of MBSE to have a unique source of truth leading to out‐of‐date artefacts. This article proposes extending the Unified Architecture Framework (UAF) Profile (UAFP) to enable safety and security systems engineers to perform their analysis from the early stage of a system development process.
Chapter
This chapter reviews the history of how humans became a species that creates and uses technology along with some important milestones. The key features of Homo sapiens include our brains, and our ability to use them to form abstractions through language, as well as the extraordinary dexterity of our hands as enablers of technological evolution. We argue that the replacement or augmentation of human and animal strength with machines, such as the steam engine, was one of those key milestones. We review several technological revolutions, including electrification starting in the nineteenth century and the information age which started in the twentieth century. We briefly discuss the role of national identity and conflicts in claiming or accelerating technological progress and speculate on what humanity’s next technological revolution might be in the future.
Chapter
Safety principles form the foundation for creating and operating trustworthy safety-critical cyber-physical systems. These principles have their roots in theoretical work and long-standing, time-tested, practical experience. Safety principles are a proven and successful way to teach, enforce, and implement safe systems.
Chapter
General principles for safety and security apply to the whole system; i.e., they cover many quality properties of the CPSs. Therefore, they are presented in this chapter.
Article
This article analyzes the policy options available to governments for addressing the very costly economic impacts of cybersecurity threats. It contributes to complexity thinking in public policy. Complexity involves self-organization, emergence, feedback loops, and adaptation. We show that these are present with cyberthreats. In contrast to plans that involve a series of linear steps to a specific outcome, complexity thinking recommends adaptive design, which creates processes that involve coordinated decentralized capacity for experimentation and resilience. These are illustrated with an examination of the 2007 nation-wide cyberattack on Estonia and the lessons learned from this attack.
Chapter
Vulnerabilities of the systems are at the core of safety accidents and security incidents. Threats and failures utilize such vulnerabilities to damage the systems. Therefore, vulnerabilities, threats, and failures can be considered the “three devils of safety and security”. These three devils accompany both the developers and the operators of the cyber-physical systems at all times.
Article
Achieving safe and robust autonomy is the key bottleneck on the path towards broader adoption of autonomous vehicles technology. This motivates going beyond extrinsic metrics such as miles between disengagement, and calls for approaches that embody safety by design. In this paper, we address some aspects of this challenge, with emphasis on issues of motion planning and prediction. We do this through description of novel approaches taken to solving selected sub-problems within an autonomous driving stack, in the process introducing the design philosophy being adopted within Five. This includes safe-by-design planning, interpretable as well as verifiable prediction, and modelling of perception errors to enable effective sim-to-real and real-to-sim transfer within the testing pipeline of a realistic autonomous system.
Article
We are concerned with the construction, formal verification, and safety assurance of dependable multiagent systems. For the case where the system (agents and their environment) can be explicitly modelled, we develop formal verification methods over several logic languages, such as temporal epistemic logic and strategy logic, to reason about the knowledge and strategy of the agents. For the case where the system cannot be explicitly modelled, we study multiagent deep reinforcement learning, aiming to develop efficient and scalable learning methods for cooperative multiagent tasks. In addition to these, we develop (both formal and simulation-based) verification methods for the neural network based perception agent that is trained with supervised learning, considering its safety and robustness against attacks from an adversarial agent, and other approaches (such as explainable AI, reliability assessment, and safety argument) for the analysis and assurance of the learning components. Our ultimate objective is to combine formal methods, machine learning, and reliability engineering to not only develop dependable learning-enabled multiagent systems but also provide rigorous methods for the verification and assurance of such systems.
Chapter
The increasing importance of software and rising level of connectivity of safety-critical products such as vehicles enable continuously improving and adding the functionality. DevOps development principles support such kind of continuous deployment. However, safety-critical products shall fulfill safety standards. In addition, it is impossible to show that a new or updated functionality is safe without considering the entire system. We introduce the SafeOps approach that leverages the DevOps principles automation, feature-driven development, and monitoring during operations to fulfill the requirements of the ISO 26262 when iteratively extending and improving safety-critical products. We present concepts and existing approaches to increase the level of automation of safety engineering tasks like safety analysis and generation of safety artifacts and we show how the management of these artifacts can be supported. Furthermore, we outline future research questions and propose a first concept to obtain quick and systematic feedback of the quality of the safety concept from the deployed products, enabling to enter the DevOps cycle from a safety point of view.KeywordsDependabilityAgileCDMBSEMBSAFTAFMEA
Chapter
Securing the supply chain of information and communications technology (ICT) has recently emerged as a critical concern for national security and integrity. With the proliferation of Internet of Things (IoT) devices and their increasing role in controlling real world infrastructure, there is a need to analyze risks in networked systems beyond established security analyses. Existing methods in literature typically leverage attack and fault trees to analyze malicious activity and its impact. In this chapter, we develop a security risk assessment framework borrowing from system reliability theory to incorporate the supply chain. We also analyze the impact of grouping within suppliers that may pose hidden risks to the systems from malicious supply chain actors. The results show that the proposed analysis is able to reveal hidden threats posed to the IoT ecosystem from potential supplier collusion.
Chapter
The main properties of trustworthy cyber-physical systems are safety and security. Both safety and security are the results of careful, responsible, risk-guided engineering. The real danger of cyber-physical systems is vulnerabilities in the protection mechanisms of the assets.
Chapter
Approaches based on Machine Learning (ML) provide novel and promising solutions to implement safety-critical functions in the field of autonomous driving. Establishing assurance in these ML components through safety requirements is critical, as the failure of these components may lead to hazardous events such as pedestrians being hit by the ego vehicle due to an erroneous output of an ML component (e.g., a pedestrian not being detected in a safety-critical region). In this paper, we present our experience with applying the System-Theoretic Process Analysis (STPA) approach for an ML-based perception component within a pedestrian collision avoidance system. STPA is integrated into the safety life cycle of functional safety (regulated by ISO 26262) complemented with safety of the intended functionality (regulated by ISO/FDIS 21448) in order to elicit safety requirements. These requirements are derived from STPA unsafe control actions and loss scenarios, thus enabling the traceability from hazards to ML safety requirements. For specifying loss scenarios, we propose to refer to erroneous outputs of the ML component due to the ML functional insufficiencies, while adhering to the guidelines of the STPA handbook.KeywordsSafety requirementsMachine LearningFunctional insufficienciesSTPAISO 26262ISO/FDIS 21448
Article
Formal risk assessment is a component of safety management relating to hazardous manual tasks (HMT). Systems thinking approaches are currently gaining interest for supporting safety management. Existing HMT risk assessment methods have been found to be limited in their ability to identify risks across the whole work system; however, systems thinking-based risk assessment (STBRA) methods were not designed for the HMT context and have not been tested in this area. The aim of this study was to compare the performance of four state-of-the-art STBRA methods: Net-HARMS, EAST-BL, FRAM and STPA to determine which would be most useful for identifying HMT risks. Each method was independently applied by one of four analysts to assess the risks associated with a hypothetical HMT system. The outcomes were assessed for alignment with a benchmark analysis. Using signal detection theory (SDT), overall STPA was found to be the best performing method having the highest hit rate, second lowest false alarm rate and highest Matthews Correlation Coefficient of the four methods.
Preprint
Full-text available
As learning machines increase their influence on decisions concerning human lives, analyzing their fairness properties becomes a subject of central importance. Yet, our best tools for measuring the fairness of learning systems are rigid fairness metrics encapsulated as mathematical one-liners, offer limited power to the stakeholders involved in the prediction task, and are easy to manipulate when we exhort excessive pressure to optimize them. To advance these issues, we propose to shift focus from shaping fairness metrics to curating the distributions of examples under which these are computed. In particular, we posit that every claim about fairness should be immediately followed by the tagline "Fair under what examples, and collected by whom?". By highlighting connections to the literature in domain generalization, we propose to measure fairness as the ability of the system to generalize under multiple stress tests -- distributions of examples with social relevance. We encourage each stakeholder to curate one or multiple stress tests containing examples reflecting their (possibly conflicting) interests. The machine passes or fails each stress test by falling short of or exceeding a pre-defined metric value. The test results involve all stakeholders in a discussion about how to improve the learning system, and provide flexible assessments of fairness dependent on context and based on interpretable data. We provide full implementation guidelines for stress testing, illustrate both the benefits and shortcomings of this framework, and introduce a cryptographic scheme to enable a degree of prediction accountability from system providers.
ResearchGate has not been able to resolve any references for this publication.