Content uploaded by Radu Calinescu
Author content
All content in this area was uploaded by Radu Calinescu on May 26, 2018
Content may be subject to copyright.
A preview of the PDF is not available
Engineering Trustworthy Self-Adaptive Software with Dynamic Assurance Cases
Abstract and Figures
Building on concepts drawn from control theory, self-adaptive software handles environmental and internal uncertainties by dynamically adjusting its architecture and parameters in response to events such as workload changes and component failures. Self-adaptive software is increasingly expected to meet strict functional and non-functional requirements in applications from areas as diverse as manufacturing, healthcare and finance. To address this need, we introduce a methodology for the systematic ENgineering of TRUstworthy Self-adaptive sofTware (ENTRUST). ENTRUST uses a combination of (1) design-time and runtime modelling and verification, and (2) industry-adopted assurance processes to develop trustworthy self-adaptive software and assurance cases arguing the suitability of the software for its intended application. To evaluate the effectiveness of our methodology, we present a tool-supported instance of ENTRUST and its use to develop proof-of-concept self-adaptive software for embedded and service-based systems from the oceanic monitoring and e-finance domains, respectively. The experimental results show that ENTRUST can be used to engineer self-adaptive software systems in different application domains and to generate dynamic assurance cases for these systems.
Figures - uploaded by Radu Calinescu
Author content
All figure content in this area was uploaded by Radu Calinescu
Content may be subject to copyright.
... Both of these capabilities (a) and (b) complement the monitoring capabilities discussed in the introduction of the main section above, with the main difference being the collection of the data for central or offline analyses. In [99], several of the assurance approaches discussed require (b) monitoring of the system, its operational contexts and/or some Key Performance Indicators (KPIs) [156], [157], [158]. Concrete measures and metrics underpinning such KPIs are elaborated upon in Sec. ...
... Also Calinescu et al. [157] suggest allocating some of the assurance tasks to run-time, by dynamically generating the assurance case throughout both design-time as well as runtime. This run-time assurance generation is predominantly dependent on formal methods and model checking [157], assuming formalisable system models and requirements. ...
... Also Calinescu et al. [157] suggest allocating some of the assurance tasks to run-time, by dynamically generating the assurance case throughout both design-time as well as runtime. This run-time assurance generation is predominantly dependent on formal methods and model checking [157], assuming formalisable system models and requirements. There is also a link between the formalisation-dependent ideas of run-time certification and ConSerts (discussed in Sec. ...
In recent years, enormous investments in Automated Driving Systems (ADSs) have distinctly advanced ADS technologies. Despite promises made by several high profile auto-makers, it has however become clear that the challenges involved for deploying ADS have been drastically underestimated. Contrary to previous generations of automotive systems, common design, development, verification and validation methods for safety critical systems do not suffice to cope with the increased complexity and operational uncertainties of an ADS. Therefore, the aim of this paper is to provide an understanding of existing methods for providing safety evidence and, most importantly, identifying the associated challenges and gaps pertaining to the use of each method. To this end, we have performed a literature review, articulated around four categories of methods: design techniques, verification and validation methods, run-time risk assessment, and run-time (self-)adaptation. We have identified and present eight challenges, collectively distinguishing ADSs from safety critical systems in general, and discuss the reviewed methods in the light of these eight challenges. For all reviewed methods, the uncertainties of the operational environment and the allocation of responsibility for the driving task on the ADS stand-out as the most difficult challenges to address. Finally, a set of research gaps is identified, and grouped into five major themes: 1) completeness of provided safety evidence, 2) improvements and analysis needs, 3) safe collection of closed loop data and accounting for tactical responsibility on the part of the ADS, 4) integration of AI/ML-based components, and 5) scalability of the approaches with respect to the complexity of the ADS.
... There are many reasons for this frequent use of PMC for the formal modelling and verification of software-intensive systems, e.g. [10,13,14,20,28,52]. The models supported by PMC (discreteand continuous-time Markov chains, Markov decision processes, partially observable Markov decision processes, etc.) capture key aspects of the uncertainty affecting such systems. ...
Given its ability to analyse stochastic models ranging from discrete and continuous-time Markov chains to Markov decision processes and stochastic games, probabilistic model checking (PMC) is widely used to verify system dependability and performance properties. However, modelling the behaviour of, and verifying these properties for many software-intensive systems requires the joint analysis of multiple interdependent stochastic models of different types, which existing PMC techniques and tools cannot handle. To address this limitation, we introduce a tool-supported UniversaL stochasTIc Modelling, verificAtion and synThEsis (ULTIMATE) framework that supports the representation, verification and synthesis of heterogeneous multi-model stochastic systems with complex model interdependencies. Through its unique integration of multiple PMC paradigms, and underpinned by a novel verification method for handling model interdependencies, ULTIMATE unifies-for the first time-the modelling of probabilistic and nondeterministic uncertainty, discrete and continuous time, partial observability, and the use of both Bayesian and frequentist inference to exploit domain knowledge and data about the modelled system and its context. A comprehensive suite of case studies and experiments confirm the generality and effectiveness of our novel verification framework.
... Denney & Pai (2024) building on their earlier work introduced above, propose a framework to facilitate what they term 'dynamic assurance', i.e. 'continued, justified confidence that a system is operating at a safety risk level consistent with an approved risk baseline' using an example of autonomy in aviation. A similar notion of 'dynamic safety management' and 'dynamic assurance cases' utilising self-adaption autonomous systems to maintain safety are developed in, for example, Trapp & Weiss (2019) and Calinescu et al. (2018). However, none of these state-of-the-art approaches have been applied to frontier AI systems. ...
Frontier artificial intelligence (AI) systems present both benefits and risks to society. Safety cases - structured arguments supported by evidence - are one way to help ensure the safe development and deployment of these systems. Yet the evolving nature of AI capabilities, as well as changes in the operational environment and understanding of risk, necessitates mechanisms for continuously updating these safety cases. Typically, in other sectors, safety cases are produced pre-deployment and do not require frequent updates post-deployment, which can be a manual, costly process. This paper proposes a Dynamic Safety Case Management System (DSCMS) to support both the initial creation of a safety case and its systematic, semi-automated revision over time. Drawing on methods developed in the autonomous vehicles (AV) sector - state-of-the-art Checkable Safety Arguments (CSA) combined with Safety Performance Indicators (SPIs) recommended by UL 4600, a DSCMS helps developers maintain alignment between system safety claims and the latest system state. We demonstrate this approach on a safety case template for offensive cyber capabilities and suggest ways it can be integrated into governance structures for safety-critical decision-making. While the correctness of the initial safety argument remains paramount - particularly for high-severity risks - a DSCMS provides a framework for adapting to new insights and strengthening incident response. We outline challenges and further work towards development and implementation of this approach as part of continuous safety assurance of frontier AI systems.
... Our proposed approach divides the model into components. When the components are updated periodically, if they are subject to any change, the affected components need to be updated and verified again [15,48]. In this research, the main approach is to keep the number of updates as low as possible. ...
In dynamic environments, safety-critical autonomous systems must adapt to environmental changes without violating safety requirements. Model verification at runtime supports adaptation through the periodic analysis of continually updated models. A major limitation of the technique is the high overhead associated with the regular analyses of large state-space models. Our paper introduces an adaptive approximation strategy that tackles this limitation by delaying unnecessary model updates, significantly reducing the overheads of these analyses. The strategy is applicable to Markov decision processes (MDPs) and is partitioned into components that can be analyzed independently and approximately. Each component is assigned a priority that depends on its impact on the accuracy of verification, and only the highest-priority components affected by changes are scheduled for updating/approximating. A complete update and verification of the entire model is triggered infrequently when the accuracy drops below a predefined threshold. We provide theoretical guarantees and proofs which ensure that our strategy can be applied without impacting the overall safety of the verified autonomous system. The experimental results from a case study in which we applied the strategy to a rescue robot team show that it is fully robust against safety-critical errors and can achieve a decision accuracy of over 97%.
... ese capabilities and methods are collectively referred to as assurances. e eld of formal Validation and Veri cation (V&V) also uses the term assurances to refer to structured evidence that indicates whether or not a system is functioning according to a priori design speci cations [17]. ese assurances will be referred to here as 'hard assurances'. ...
People who design, use, and are affected by autonomous artificially intelligent agents want to be able to \emph{trust} such agents -- that is, to know that these agents will perform correctly, to understand the reasoning behind their actions, and to know how to use them appropriately. Many techniques have been devised to assess and influence human trust in artificially intelligent agents. However, these approaches are typically ad hoc, and have not been formally related to each other or to formal trust models. This paper presents a survey of \emph{algorithmic assurances}, i.e. programmed components of agent operation that are expressly designed to calibrate user trust in artificially intelligent agents. Algorithmic assurances are first formally defined and classified from the perspective of formally modeled human-artificially intelligent agent trust relationships. Building on these definitions, a synthesis of research across communities such as machine learning, human-computer interaction, robotics, e-commerce, and others reveals that assurance algorithms naturally fall along a spectrum in terms of their impact on an agent's core functionality, with seven notable classes ranging from integral assurances (which impact an agent's core functionality) to supplemental assurances (which have no direct effect on agent performance). Common approaches within each of these classes are identified and discussed; benefits and drawbacks of different approaches are also investigated.
With the rapid advancements in Artificial Intelligence (AI), autonomous agents are expected to handle increasingly complex situations. However, learning‐enabled algorithms, which are critical to these systems, present significant challenges, including complexity, difficulty in verification, and a lack of certification pathways. A systematic approach integrating architectural analysis with human–machine interaction is crucial to ensuring the safety of these systems. This research emphasizes the early incorporation of human interactions in the design of architectural models to meet safety requirements. These interactions are modeled in the Soar cognitive architecture, which combines symbolic decision logic and numeric decision preferences, refined by reinforcement learning. The agent is then automatically translated into the formal verification environment, nuXmv, where its properties are verified. Our framework integrates systems modeling, formal verification, and simulation to check operational correctness, enhancing the reliability and safety of learning‐enabled autonomous agents.
We present our Balanced, Integrated and Grounded (BIG) argument for assuring the safety of AI systems. The BIG argument adopts a whole-system approach to constructing a safety case for AI systems of varying capability, autonomy and criticality. Whether the AI capability is narrow and constrained or general-purpose and powered by a frontier or foundational model, the BIG argument insists on a meaningful treatment of safety. It respects long-established safety assurance norms such as sensitivity to context, traceability and risk proportionality. Further, it places a particular focus on the novel hazardous behaviours emerging from the advanced capabilities of frontier AI models and the open contexts in which they are rapidly being deployed. These complex issues are considered within a broader AI safety case that approaches assurance from both technical and sociotechnical perspectives. Examples illustrating the use of the BIG argument are provided throughout the paper.
Cloud-based service certification extends traditional certification schemes to address the peculiarities of dynamic distributed environments. It pursues three main objectives: (i) greater flexibility of the certification process (i.e., models for certification life cycle, target, and process), (ii) adaptability to service evolution and environmental changes (i.e., incremental certification), (iii) soundness of the chain of trust, reducing the involvement of chartered (and costly) Certification Authorities.
Self-adaptation equips a software system with a feedback loop that resolves uncertainties during operation and adapts the system to deal with them when necessary. Most self-adaptation approaches today use decision-making mechanisms that select for execution the adaptation option with the best-estimated benefit expressed as a set of adaptation goals. A few approaches also consider the estimated (one-off) cost of executing the candidate adaptation options. We argue that besides benefit and cost, decision-making in self-adaptive systems should also consider the estimated risk the system or its users would be exposed to if an adaptation option were selected for execution. Balancing all three concerns when evaluating the options for adaptation to mitigate uncertainty is essential for satisfying stakeholders’ concerns and ensuring the safety and public acceptance of self-adaptive systems. In this paper, we present a reference model for decision-making in self-adaptation that considers the estimated benefit, cost, and risk as core concerns of each adaptation option. Leveraging this model, we then present an ISO/IEC/IEEE 42010 compatible architectural viewpoint that aims at supporting software architects responsible for designing robust decision-making mechanisms for self-adaptive systems. We demonstrate the applicability, usefulness, and understandability of the viewpoint through a case study where participants with experience in the engineering of self-adaptive systems performed a set of design tasks in DeltaIoT, an Internet-of-Things exemplar for research on self-adaptive systems.
Providing assurances for self-adaptive systems is challenging. A primary underlying problem is uncertainty that may stem from a variety of different sources, ranging from incomplete knowledge to sensor noise and uncertain behavior of humans in the loop. Providing assurances that the self-adaptive system complies with its requirements calls for an enduring process spanning the whole lifetime of the system. In this process, humans and the system jointly derive and integrate new evidence and arguments, which we coined perpetual assurances for self-adaptive systems. In this paper, we provide a background framework and the foundation for perpetual assurances for self-adaptive systems. We elaborate on the concrete challenges of offering perpetual assurances, requirements for solutions, realization techniques and mechanisms to make solutions suitable. We also present benchmark criteria to compare solutions. We then present a concrete exemplar that researchers can use to assess and compare approaches for perpetual assurances for self-adaptation.
Self-aware computing systems are envisaged to exploit the knowledge of their own software architecture, hardware infrastructure and environment in order to follow high-level goals through proactively adapting as their environment evolves. This chapter describes two classes of key enabling techniques for self-adaptive systems: automated synthesis and formal verification. The ability to dynamically synthesize component connectors and compositions underpins the proactive adaptation of the architecture of self-aware systems. Deciding when adaptation is needed and selecting valid new architectures or parameters for self-aware systems often requires formal verification. We present the state of the art in the use of the two techniques for the development of self-aware computing systems and summarize the main research challenges associated with their adoption in practice.
Providing assurances for self-adaptive systems is challenging. A primary underlying problem is uncertainty that may stem from a variety of different sources, ranging from incomplete knowledge to sensor noise and uncertain behavior of humans in the loop. Providing assurances that the self-adaptive system complies with its requirements calls for an enduring process spanning the whole lifetime of the system. In this process, humans and the system jointly derive and integrate new evidence and arguments, which we coined perpetual assurances for self-adaptive systems. In this paper, we provide a background framework and the foundation for perpetual assurances for self-adaptive systems. We elaborate on the concrete challenges of offering perpetual assurances, requirements for solutions, realization techniques and mechanisms to make solutions suitable. We also present benchmark criteria to compare solutions. We then present a concrete exemplar that researchers can use to assess and compare approaches for perpetual assurances for self-adaptation.
Modern software applications are subject to uncertain operating conditions, such as dynamics in the availability of services and variations of system goals. Consequently, runtime changes cannot be ignored, but often cannot be predicted at design time. Control theory has been identified as a principled way of addressing runtime changes and it has been applied successfully to modify the structure and behavior of software applications. Most of the times, however, the adaptation targeted the resources that the software has available for execution (CPU, storage, etc.) more than the software application itself. This paper investigates the research efforts that have been conducted to make software adaptable by modifying the software rather than the resource allocated to its execution. This paper aims to identify: the focus of research on control-theoretical software adaptation; how software is modeled and what control mechanisms are used to adapt software; what software qualities and controller guarantees are considered. To that end, we performed a systematic literature review in which we extracted data from 42 primary studies selected from 1512 papers that resulted from an automatic search. The results of our investigation show that even though the behavior of software is considered non-linear, research efforts use linear models to represent it, with some success. Also, the control strategies that are most often considered are classic control, mostly in the form of Proportional and Integral controllers, and Model Predictive Control. The paper also discusses sensing and actuating strategies that are prominent for software adaptation and the (often neglected) proof of formal properties. Finally, we distill open challenges for control-theoretical software adaptation.
An assurance case provides a structured argument to establish a claim for a system based on evidence about the system and its environment. I propose a simple interpretation for the overall argument that uses epistemic methods for its evidential or leaf steps and logic for its reasoning or interior steps: evidential steps that cross some threshold of credibility are accepted as premises in a classical deductive interpretation of the reasoning steps. Thus, all uncertainty is located in the assessment of evidence. I argue for the utility of this interpretation.
An increasingly important concern of software engineers is handling uncertainties at design time, such as environment dynamics that may be difficult to predict or requirements that may change during operation. The idea of self-adaptation is to handle such uncertainties at runtime, when the knowledge becomes available. As more systems with strict requirements require self-adaptation, providing guarantees for adaptation has become a high-priority. Providing such guarantees with traditional architecture-based approaches has shown to be challenging. In response, researchers have studied the application of control theory to realize self-adaptation. However, existing control-theoretic approaches applied to adapt software systems have primarily focused on satisfying only a single adaptation goal at a time, which is often too restrictive for real applications. In this paper, we present Simplex Control Adaptation, SimCA, a new approach to self-adaptation that satisfies multiple goals, while being optimal with respect to an additional goal. SimCA offers robustness to measurement inaccuracy and environmental disturbances, and provides guarantees. We evaluate SimCA for two systems with strict requirements that have to deal with uncertainties: an underwater vehicle system used for oceanic surveillance, and a tele-assistance system for health care support.
Software is often governed by and thus adapts to phenomena that occur at runtime. Unlike traditional decision problems, where a decision-making model is determined for reasoning, the adaptation logic of such software is concerned with empirical data and is subject to practical constraints. We present an Iterative Decision-Making Scheme (IDMS) that infers both point and interval estimates for the undetermined transition probabilities in a Markov Decision Process (MDP) based on sampled data, and iteratively computes a confidently optimal scheduler from a given finite subset of schedulers. The most important feature of IDMS is the flexibility for adjusting the criterion of confident optimality and the sample size within the iteration, leading to a tradeoff between accuracy, data usage and computational overhead. We apply IDMS to an existing self-adaptation framework Rainbow and conduct a case study using a Rainbow system to demonstrate the flexibility of IDMS.