Available via license: CC BY 4.0
Content may be subject to copyright.
EI Systems Challenges
1
fortiss View on
Systems Challenges
for
Trustworthy Embodied Intelligence
1
Harald Rueß
(ruess@fortiss.org)
fortiss
Research Institute of the Free State of Bavaria
Guerickestr. 25
D-80805 München
10th of January, 2022.
“I am an optimist and I believe
that we can create AI for the good of the world.
That it can work in harmony with us.
We simply need to be aware of the dangers,
identify them,
employ the best possible practice and management,
and prepare for its consequences
well in advance”
Stephen Hawkins at Web Summit in Nov. 2017
1
Disclaimer: this research is supported by the BMWi-funded project Embodied Intelligence - The Next Big Thing, and the CASSAI project
as funded by the Bavarian Ministry of Economics in the context of the fortiss AI Center. This report heavily draws on the results on the Agenda
CPS and ongoing discussions and constructive feedback to earlier versions by Prof. Dr. Dr. h.c. Manfred Broy, is strongly influenced by
ongoing work by SRI colleagues on the rigorous design of increasingly autonomous machines, in particular Dr. John Rushby and Dr. Natarajan
Shankar, and also exchanges during a research stint in November 2019. It is also based on regular exchanges with fortiss colleagues, in
particular, Dian Balta, Dr. Markus Duchon, Dr. Johannes Kroß, Prof. Dr. Daniel Mendez, Prof. Dr. Rute Sofia, and Dr. Henrik Putzer. It has
also benefitted from a recent BIG workshop on challenges for embodied systems with participating colleagues from Siemens and fortiss. The
views in the section on robust AI have been developed while preparing the fortiss-IBM joint Center for AI, and the section on human-centered
engineering is heavily based on the corresponding fortiss whitepaper by Dr. Yuanting Liu and Dr. habil. Hao Shen on this topic, and the views
on uncertainty quantification have mainly been formed through interactions within the CASSAI projects with the fortissians Carmen Carlan,
Amit Sahu, Tewodros Beyene, and Julian Bernhard. It is a safe assumption, however, that most probably none of the contributors mentioned
above could agree with all the described hypotheses on systems challenges for embodied systems as outlined here.
EI Systems Challenges
2
Table of Contents
TABLE OF CONTENTS .............................................................................................................................. 2
1. INTRODUCTION ........................................................................................................................... 3
2. EMBODIED SYSTEMS ................................................................................................................. 7
SOFTWARE ACTORS ................................................................................................................................. 8
ROBOTIC COMPANIONS ........................................................................................................................... 9
SERVICE FEDERATIONS ......................................................................................................................... 11
3. CHARACTERISTICS .................................................................................................................. 14
COGNITIVE SYSTEMS ............................................................................................................................ 15
INTENT-DRIVEN SYSTEMS ..................................................................................................................... 17
FEDERATED SYSTEMS ........................................................................................................................... 18
AUTONOMOUS SYSTEMS ....................................................................................................................... 19
SELF-LEARNING SYSTEMS .................................................................................................................... 20
4. TRUSTWORTHINESS ................................................................................................................ 22
5. CHALLENGES ............................................................................................................................. 27
ROBUST AI/ML ..................................................................................................................................... 28
HUMAN-CENTERED AI/ML .................................................................................................................. 31
COGNITIVE ARCHITECTURES ............................................................................................................... 34
UNCERTAINTY QUANTIFICATION ........................................................................................................ 36
SELF-INTEGRATION .............................................................................................................................. 40
ANALYSIS ............................................................................................................................................... 43
TESTING .................................................................................................................................................. 43
SYMBOLIC ANALYSIS ............................................................................................................................. 45
RUNTIME VERIFICATION AND RECOVERY .............................................................................................. 46
ASSURANCE ............................................................................................................................................ 48
6. CONCLUSIONS ........................................................................................................................... 56
EI Systems Challenges
3
1. INTRODUCTION
A new generation of increasingly autonomous machinery is about to be developed and
embodied into all kinds of aspects of everyday life. This machinery is used beyond mere
automation and assistance to humans, as manufacturing robots make way for autonomous
machine workers, business and administrational services are performed by autonomous virtual
organizations, and processes and value chains in the real and the virtualized worlds are executed
by coalitions of autonomous machine actors. In this way, computational machines are acting
as self-sufficient entities in our very economic and societal fabric.
Next generation autonomous machines may also acquire and improve necessary skills and
behavior in real world contexts by means of action, reaction, and interaction in and with
physical and social environments.
2
But how exactly do the required cognitive skills and
behavior that could possibly be termed intelligent
3
unfold in time?
Various answers to this fundamental quest on the emergence of intelligence, including
Descartes’ brain-body duality, have been developed throughout the history of philosophy.
These considerations have also been motivating the field of embodied intelligence (EI), which
is the computational approach to the design and understanding of intelligent behavior in
embodied and situated actors through the consideration of the strict coupling between the actor
and its environment (situatedness), mediated by the constraints of the actor’s own body,
perceptual and motor system, and brain (embodiment).
4
Based on this paradigm it could be demonstrated that computational machines may develop and
improve at least some physical dexterities and cognitive skills by means of interactions between
2
Striedinger, Kaspar, der „rätselhafte Findling“, Lebensläufe aus Franken, Herausgegeben im Auftrag der Gesellschaft für Fränkische
Geschichte von Anton Chrous, III. Band, 1927.
3
Deary, Intelligence: A very short introduction, Oxford University Press, 2020.
4
Cangelosi, Bongard, Fischer, Nolfi. Embodied Intelligence. Handbook of Computational Intelligence, Springer, 2015.
EI Systems Challenges
4
the physical self and the physical environment.
5
6
7
8
9
10
11
12
13
14
In this way, EI robots have
been built that can move, see, speak, and interact with other robots effectively.
15
16
Current
approaches to EI also rely on training actors in virtualized playgrounds for accelerated learning
and for enabling non-destructive learning from failure.
17
18
19
20
Despite all this progress on EI,
however, it still remains open to speculation if and how “higher-level” thinking, symbol
grounding, natural language understanding, consciousness, and emotions may emerge in
machines through embodiment.
21
22
23
A central but largely untouched challenge of EI is concerned with embodying computational
actors into real-world environments with all their intricacies, including uncertainty, complexity,
and unpredictability. This is not an easy task as actors may demonstrate unexpected and even
emergent behavior when deployed in new operating environments; for example, when being
moved from a model-based virtual playground into a real-world physical context.
24
5
There is growing evidence that at least some sensorimotor behavior and cognitive skills are realized in the body and that social interactions
can bootstrap learning (for example, Ballard et al, Deictic codes for the embodiment of cognition. Behavioral and Brain Sciences, 1997;
McGeer. Passive dynamic walking. International Journal of Robotics Research, 1990; O’Regan, Noe. A sensorimotor account of vision and
visual consciousness. Behavioral and Brain Sciences, 2001)
6
Pfeifer, Scheier. Understanding intelligence, MIT Press, 2001.
7
Breazeal. Designing sociable robots. MIT Press, 2002.
8
Beer: A dynamical systems perspective on agent-environment interaction. Artificial Intelligence 72, 1994.
9
Brooks: Elephants don’t play chess, Robotics and Autonomous Systems 6(1), 1990.
10
Cangelosi: Grounding language in action and perception: From cognitive agents to humanoid robots, Physics of Life Reviews, 7(2), 2010.
11
Chiel, Beer: The brain has a body: Adaptive behavior emerges from interactions of nervous system, body and environment. Trends in
Neurosciences 20, 1997.
12
Keijzer: Representation and Behavior, MIT Press, 2001.
13
Nolfi, Floreano: Evolutionary Robotics: The Biology, Intelligence, and Technology of Self-Organizing Machines, MIT Press, 2000.
14
Pfeifer, Bongard: How the Body Shapes the Way We Think: A New View of Intelligence, MIT Press, 2006.
15
Workshop on Embodied AI at CVPR’21 (https://embodied-ai.org)
16
Pfeifer, Iida, Embodied Artificial Intelligence, 2003.
17
Such as visual exploration, visual navigation, and embodied question-answering. In particular, the question ”Is there still milk in the fridge?”
may require the embodied actor to unlock new insights that are momentous in answering and to initiate corresponding tasks such as moving to
the kitchen and opening the fridge.
18
For example, workshop on Embodied AI, http://https://embodied-ai.org/, CVPR, 2020.
19
For example, ongoing Embodied AI programmes, for example, at Intel Labs or Facebook AI Research.
20
These simulation environments currently include SUNCG, Matterport3D, iGibson, Replica, Habitat, and DART.
21
Smith, Gasser, The development of embodied cognition: Six lessons from babies, Artificial life, vol. 11, no. 1-2, 2005.
22
Rushby, Sanchez, Technology and Consciousness, SRI Technical Report, 2018.
23
For example, Integrated information theory (IIT) asserts that it is the intrinsic causal powers of the brain that really matter. And those powers
cannot be simulated but must be part and parcel of the physics of the underlying mechanism.
24
Remark: the very notion of a “digital twin” suggests an unequivocalness of the model with respect to the real world no model can actually
deliver; as Plato already realized such a model is nothing more than a “shadow on the wall”.
EI Systems Challenges
5
Another key point of moving actors from innocent training playgrounds into a physical or social
context is that actions have real consequences. Tay,
25
a pretrained chatbot, who quickly turned
nasty, when put into the context of a (so-called) social network, may serve as cautionary tale of
what can go wrong all too easily and quickly. While this problem was solved by simply taking
Tay offline, it is easy to imagine other cases of self-learning actors neglecting human moral
expectations with long-lasting and perhaps even more severe impact. The sad tale of Tay also
reminds us on the importance of a meaningful human control of EI.
We clearly face serious social, economic, and legal challenges when deploying EI in the real
world,
26
as it is crucial to coordinate the behavior of EI actors in a beneficial manner, ensure
their compatibility with our human-centered social norms and values, and design verifiably safe
and reliable human-machine interaction.
Notice that interpretations of the terms “beneficial” and “social norms and values” are heavily
context- and application-specific. A simple robot companion, for example, might be called
beneficial if it acts according to the intents of its human companion. This does not necessarily
imply, however, that the robot companion acts according to the intents of others, which may
even be contradictory. Let alone that it acts according to relevant human-centered social values.
In our quest for developing and deploying EI in a meaningful way we therefore need to find
adequate solutions to central engineering challenges:
§ How can we ensure that an EI actor behaves beneficially? That is, it functions as
intended and it behaves in accordance with higher-level societal goals and standards.
§ How can we ensure that learning-enabled EI actors are robust across their whole
lifecycle? That is, the behavior or EI-enabled actors is dependable, safe, and predictable
(up to quantified tolerances) in uncertain, complex, and unpredictable environments.
§ How can we ensure that a meaningful (human) control over an EI actor is enabled during
operation?
Traditionally, the field of systems engineering tackles these kinds of questions for assuring
quality-of-service and the dependable functioning of software-intensive systems.
Systems engineering, however, has so far mainly been concerned with relatively small-scale,
centralized, deterministic, non-evolvable, automated, and task-specific embedded and cyber-
physical systems (CPS), which are operating in well-defined and largely predictable operating
environments. The state-of-the-practice in safety engineering,
27
for instance, is restricted to
deterministic systems operating in well-defined operating contexts, and it usually relies on
25
https://www.theverge.com/2016/3/24/11297050/tay-microsoft-chatbot-racist
26
Vazdanan et al., Responsibility Research for Trustworthy Autonomous Systems, 2021.
27
For instance, industrial safety engineering standards such as DO 178C in aerospace and ISO 26262 in the automotive industry.
EI Systems Challenges
6
ultimate fallback mechanisms to a human operator (as is the case, for example, for current
airplane autopilots). Current safety engineering practice therefore does not support certification
and corresponding operating readiness of the envisioned new generation of EI acting in real-
world environments.
In Section 2 we introduce the notion of embodied systems. This class of systems is based on the
central EI concepts of situatedness and embodiment. As such, the design of embodied systems
may benefit from any progress on EI, but their success doesn’t hinge on reaching EI’s ultimate
quest for human-like machine intelligence. We illustrate the defining features and the disruptive
potential of embodied systems by means of three different scenarios, namely software
assistants, robot companions, and federations of services. In Section 3 we analyze the main
characteristics of embodied systems, and in Section 4 we discuss how to possibly increase trust
into embodied systems. In Section 5 we deduce system challenges for designing trustworthy
embodied systems from the main characteristics of embodied systems. We conclude with some
final remarks in Section 6.
EI Systems Challenges
7
2. EMBODIED SYSTEMS
In contrast to traditional EI, which is largely driven by its fundamental quest for emerging
intelligent behavior in computational machines, we are embracing a seemingly utilitarian stance
in that the embodiment of actors in a real-world context, physical or social, serves
§ the development of flexible, inventive, and optimized actions
§ for beneficial, goal-oriented, and robust behavior
§ in uncertain, complex, and unpredictable real-world environments.
The central challenge is to enable the beneficial, dependable, safe, and predictable, at least up
to acceptable tolerances, operation of embodied actors when woven into our very economic and
social fabric.
Actors
28
are supposed to purposefully cooperate with other actors, both human and robot. As
such, embodied systems are supporting humans in various real-world tasks, and in an
increasingly autonomous, self-guided manner. Embodied systems, however, should not be
restricted to an assistance role. Instead, their self-learning capabilities should well support the
co-evolution of humans and machine as necessary for bootstrapping overall capabilities.
29
Embodied systems
30
may be viewed as increasingly autonomous and self-learning cyber-
physical systems.
31
More specifically, embodied systems are decentralized, dynamic, self-
learning, and self-organizing federations of actors that collaborate to accomplish complex tasks
and missions in partially unpredictable real-world operating contexts; these systems therefore
are:
§ Situational in that they are aware of the operating context and itself.
§ Embodied in that real-world actions, reactions, and interactions are instrumental for the
flexible development of inventive actions for goal-oriented and robust behavior in
uncertain, complex, and unpredictable real-world environments.
§ Open to interact and collaborate with others in a mutually synergistic manner, while
still operating as self-sufficient individually purposeful systems.
28
Notice that we use the terms embodied system and embodied actor almost interchangeably.
29
Bardini, Bootstrapping : Douglas Engelbart, Coevolution, and the Origins of Personal Computing, Stanford University Press, 2000.
30
Notice that we use the terms embodied system and embodied actors almost interchangeably; if the embodied system is regarded as a self-
sufficient component, however, we tend to call it an actor. Moreover, populations of actors are usually refered to as embodied systems.
31
Broy, Geisberger (Eds.), Agenda CPS, acatech, 2013 (https://www.fortiss.org/en/results/scientific-publications/details/agenda-cps-
integrierte-forschungsagenda-cyber-physical-systems)
EI Systems Challenges
8
§ Adaptive in that they may adjust and improve behavior through experience and targeted
exploration.
Some scenarios might help to illustrate the main concepts, benefits, and challenges of embodied
systems.
SOFTWARE ACTORS
Consider a mission- and communication-oriented software-based actor
32
which autonomously
gathers information, for example, from seminar organizers, composes announcements of next
week's seminars, and mails them each week to a list that it keeps updated, all without the
supervision of a human. Similarly, consider a software companion for automated assignment
of tasks to employees in a large organization that automates the workload of hundreds of human
counterparts. Automated software assistants typically are “embodied” in and interact with a
largely virtual world of information sources, with a dedicated interface to a human operator.
These kinds of autonomous software assistants do exist, with their main architectural principles
rooted in cognitive architectures such as the global workspace theory
33
or multi-actor
communication frameworks such as KQML
34
or OAA.
35
They have often been shown to
outperform human counterparts on many real-life exploration and complex planning,
scheduling, and dispatching tasks such as customer service, risk evaluation, product inspection,
and data mining. Their actions are based on a situational awareness of the operating
environment together with built-in intents and goals for managing the unpredictable.
Traditional autonomous software assistants, however, have been lacking main attributes of
embodied systems as they are purpose-built for solving specific tasks only, and their ability to
adapt to changing environments through interactions with their environment, including human
operators, is limited.
We can easily imagine a new generation of integrated and more versatile software-based actor,
which are embodied in the sense that they are learning our intents through collaborative
interactions, and which are able to automatically detect and adapt to changing conditions (for
example, a specific source of information is temporarily unavailable), thereby also optimizing
future problem-solving strategies. Such a personal software assistant should be able to largely
automate managerial tasks such as doing taxes, planning trips, ordering food, or controlling
household appliances according to an overall situational assessment and intents.
32
The actor paradigm in AI is based upon the notion of reactive, internally motivated entities embedded in changing, uncertain worlds which
they perceive, and in which they act.
33
Franklin, Patterson, The LIDA Architecture: Adding New Modes of Learning to an Intelligent, Autonomous, Software Agent, IDPT, 2006.
34
Finin,Labrou,Mayfield, KQML as an agent communication language, Software Agents, MIT Press, 1997.
35
Martin, Cheyer, Moran, The Open Agent Architecture: A Framework for Building Distributed Software Systems, Applied AI, 1999.
EI Systems Challenges
9
ROBOTIC COMPANIONS
36
Figure 1 depicts an embodied system with the sole purpose of landing an airplane. In a slightly
more complex scenario, we envision such a robot to act as a co-pilot in a single-pilot cockpit.
In this way, on long flights with two pilots, one can sleep while the other flies with the assistance
from the robotic companion. Such a robot needs to be more like a human copilot than
conventional flight management system or functionally automated autopilot. In particular, the
robot companion needs to perform heterogeneous and complementary tasks including, say,
radio communications, interpreting weather data and making route adjustments, pilot
monitoring tasks, shared tasks (flaps, gear), ground taxi, and communication with the cabin-
crew (emergency evacuation). The robot companion also needs to integrate these tasks to
accomplish a safe flight, it needs to base its decisions and actions on an overall situational
assessment. In case things go wrong, the robot companion needs to find effective explanations
based on fault diagnosis, and it needs to engage in an effective resolution process with the
(human) pilot, based on a model of the pilot’s beliefs.
Figure 1. Robotic copilot.
37
In extreme situations the automated robot copilot might even take over control; for instance, if
there is smoke in the cockpit. The robotic copilot might also be better suited to maintain pitch
and thrust in extreme situation as the human pilot.
38
In these rare cases, the “robot” must now
also cope with inconsistencies (for example, in sensor readings) based on flight laws, training
procedures, models of the physical environment, and unforeseen situations without the
possibility of a structured hand-over to the human pilot. In extreme cases where flight laws
36
This scenario is heavily based on the one presented by J. Rushby on increasingly autonomous systems.
37
Source: HTTPS://WWW.AEROTIME.AERO/23374-ROBOT-CO-PILOT-FLIES-LANDS-BOEING-737
38
For instance, http://understandingaf447.com/extras/18-4_minutes__23_seconds_EN.pdf
EI Systems Challenges
10
suddenly change, due, for example, to severe damage to the body of the airplane, the
mechanized co-pilot can even relearn to fly the aircraft under these new circumstances on the
spot and in real-time.
Altogether, the robotic copilot in Figure 1 augments sensing, acting, and mental capabilities of
the pilot (see Figure 2). For its embodied “self” it may be operated in different cockpit settings.
However, if the robotic copilot is only supposed to augment decision and acting capabilities of
a pilot then an embodied “self” is superfluous. In these cases, it suffices to deploy an
autonomous and self-learning software companion into the cockpit control system.
FIGURE 2. AREAS OF HUMAN AUGMENTATION.
The robotic copilot is a particular instance of a larger class of embodied companions. Personal
companions for supporting and taking over tedious household chores and for assisting with tax
declarations, including the communication with tax authorities, and suggesting new
possibilities based on our intents are an old dream. Embodied companions are also designated
to assist, say, truck drivers, ship captains, caregivers, investors, managers, workers, lawyers,
and, in fact, everybody. Potential benefits of these kinds of embodied companions include
increased safety, reliability, efficiency, affordability, and previously unattainable capabilities.
EI Systems Challenges
11
FIGURE 3. FEDERATED RETAIL.
SERVICE FEDERATIONS
When using current shopping websites, the consumer selects an order from a large, but
ultimately fixed set of product options and configurations, and the ordered goods are, by-and-
large, already produced and stored in a system of warehouses as to guarantee speedy delivery.
As such, the traditional shopping experience has just moved from a physical store onto a retail
web site, but the whole process builds on established stakeholder roles of consumers, producers,
logisticians, and chains of retailers.
We might think, however, of more flexible setups which start with a software-defined custom
design instead of a custom order. An often-cited example are custom-designed sneakers which
might even be 3-D knitted, say, in a store.
39
Other obvious examples are software-defined
custom cars or mission-specific drones. The journey thus starts with a custom design, which is
usually constructed by instantiating available design patterns and by composing them into the
desired design and a corresponding production plan.
This new design and the production plan can be made available to others to build upon;
analogous to an “app” store. Let’s assume, however, that the design now needs to be realized,
that is the production plan needs to be executed. Instead of identifying a suitable production
site upfront, a bidding process is started among several production sites and logistics provider
to dynamically set up a suitable distributed production chain for realizing the custom design.
39
See Speed Factory (https://www.digitale-
technologien.de/DT/Redaktion/DE/Standardartikel/AutonomikFuerIndustrieProjekte/autonomik_fuer_industrie_projekt-speedfactory.html )
EI Systems Challenges
12
In Figure 3 this process of connecting the digital production plan with real-world production
entities such as 3D printers, robots, flexible and production lines together with storage and
transport capabilities for moving partly finished products is referred to as deployment.
40
Some
of these entities, such as the shipping service, may operate autonomously and is possibly
organized as its own virtual company.
Such a deployment might be chosen according to different criteria, including the traditional
triad of costs, quality, and time-to-delivery. For example, the ordering customer is provided
with the option of a reduced price if she accepts a, say, two-day delay of shipping. In addition,
flexible deployment might also take other all-important attributes such as resilience to faults or
even partial collapse of selected production or logistics capabilities, sustainability (for example,
climate neutrality) and other ethical considerations (for example, no child labor). These
attributes might now even be part of the ordering process itself, thereby enabling everybody to
act on behalf of her own personal beliefs and social values, and to directly influence the world
through seemingly mundane every-day acts such as ordering goods.
Altogether, the customer is not (only) a customer anymore as she can order the manufacturing
of the goods she wants or needs. Also, integrated production and logistics chains are broken up
into highly specialized services, which are orchestrated flexibly and in a highly distributed
fashion. The production and logistics services might be realized through increasingly
autonomously acting actors, who might do their own bidding for providing services with the
purpose of forming dynamic coalitions. From this point of view, there does not seem to be a
direct need for a traditional retailer
41
as an intermediary between customers and producers
anymore.
By-and-large we might be able to realize a proof-of-concept prototype of the embodied retail
system as depicted in Figure 3 with currently available technology. In fact, the attentive reader
might already have noticed that the embodied retailer is structured according to the main
principles of model-driven design, namely, platform independent model (PIM), platform
dependent model (PDM), and a deployment relation between those two models.
However, the extra flexibility of the embodied retail system comes with a cost, and we do not
yet have a foundational understanding of the necessary operators for composing the required
designs and service nor do we have anything that comes close to the required large-scale
deployment schemes. There are also serious questions regarding the resilience, that is tolerance,
isolation, and recovery from faults, breakdowns, and cyberattacks.
42
From a business and societal perspective the main open questions involve feasibility of
corresponding value chains, incentivization and regulation, (product) liability issues,
40
In general, such a deployment is an n:m relation between services in the virtual world and their physical embodiments.
41
Maybe as an embodied system?
42
Fault Detection, Isolation, Recovery (FDIR)
EI Systems Challenges
13
intellectual property, payment systems, and tax laws. Indeed, systems such as the outlined
federation for retail do not only have the potential of disrupting established economic cycles
but also shake up established social, economic, and governmental structures and infrastructures.
The embodied retail scenario has been designed to also demonstrate the possible elimination of
middlemen between consumer and service providers, thereby enabling completely new value
chains in the upcoming era of post-platform ecosystems. Particularly, these kinds of dynamic
federations of embodied service systems may well make currently well-established retail
platforms and middlemen such as Amazon, Uber, or AirBnB superfluous. They also rely on
flexible infrastructural services for, say, payment systems, insurance, and arbitration boards.
We can easily abstract from the main building principles of embodied retail to other societal-
scale life worlds to create similar scenarios; in particular, critical supply infrastructures (for
example, energy/water), mobility (for example, integrated transportation), medical (for
example, personalized medicine), law (for example, judicial Q&A advisors, attorneys at law,
and software judges in line with Leibniz’ rational actors), and government (e.g. seamless and
proactive government services).
Numerous other examples of embodied federated systems can be found across many application
domains. For instance, collaborative IoT systems for supply-demand interaction (e.g. machine-
machine, human-machine) in industrial applications, prosumer interaction and collaboration
under permanently changing conditions in the energy domain, ad hoc collaboration between
smart infrastructure and autonomous vehicles, self-engineering and re-configuration of
automation components in building automation, or self-managed production logistics and
warehouse logistics with a heterogeneous fleet of AGVs which are able to self-adapt to
heterogeneous production floors and tasks.
Moreover, next-generation communication networks are expected to sense, compute, learn,
reason, act on business intent, and to manage the ongoing explosion of data from an ever-
increasing number of connected intelligent devices and a multiplicity of new use cases, along
with ever-changing network topologies.
43
Zero-touch network operations for coping with the
growing system’s complexity, size, and reduced decision-making lead time lead to an increased
level of network autonomy and self-adapting automation, where unforeseen situations, intents,
and requirements can usually be addressed without human intervention.
43
Ericsson, Artificial intelligence in next-generation connected systems, Whitepaper, 2021.
EI Systems Challenges
14
3. CHARACTERISTICS
Embodied systems are comprised of federations of collaborating actors, they operate in largely
unpredictable environments, physical or not, and they recognize this environment through
sensors. Moreover, they are informed about the intentions of other actors in their respective and
immediate operating environments; they take non-trivial decisions based on reasoning, they
influence their environment, including other actors, via actuators; they interact and cooperate
with the elements of their operating environment, they influence elements in their environment
to better meet own goals
44
; and they show a certain behavior based on skills; and they learn
new and improved behavior during operation and through interactions.
45
Altogether, an
embodied system is characterized as being:
§ Cognitive, as actions are based on situational awareness, model-building, and planning.
§ Intent-driven, as actions are based on capturing actors’ intents, tasks, and goals.
§ Federated, as actions of decentralized actors are coordinated in a collaborative manner
between stakeholders and on an intentional level to accomplish joint tasks or missions.
§ Autonomous, as actions are increasingly determined by an actor’s, or federations of
actors’, own knowledge, beliefs, intents, preferences, and choices.
§ Self-learning, as actions are adapted and improved through experience, exploration, and
reasoning, both inductive and deductive, of a situated actor.
Notice that embodied systems may therefore be viewed as autonomous and self-learning cyber-
physical systems with intent-driven, goal-oriented, and collaborative behavior.
Embodied systems therefore go well beyond current functionally automated systems, such as
autopilots for landing an airplane, parking a car, or manufacturing robots, which are all designed
to handle limited tasks in precisely specified operational contexts. Task execution in
functionally automated systems is planned offline or during design time, and they usually do
not learn during operation. Moreover, collaboration is restricted to the exchange of information
about the system context, and they rely on fallback mechanism to a human operator when
encountering unforeseen and difficult situations. Most current systems, including production or
household robots, are functional automated systems, and we are on the edge of deploying intent-
and mission-oriented systems. Simple collaborative systems start to be established. Self-
learning during operation, however, is not possible now for mission-critical and safety-related
applications.
44
For example, Mechanism Design (see also: Roughgarden, Algorithmic Game Theory, CACM, 2010).
45
Putzer, Wozniak, Trustworthy Autonomous/Cognitive Systems, fortiss, 2021. (see also:
https://www.fortiss.org/fileadmin/user_upload/05_Veroeffentlichungen/Informationsmaterialien/fortiss_whitepaper_trustworthy_ACS_web.
pdf)
EI Systems Challenges
15
Cognitive Systems
Cognition may be defined as the mental action or process of acquiring knowledge and
understanding through experience and the senses.
46
Clearly, it is only in conjunction with
cognitive faculties that systems and machines develop their full potential and address a
constantly growing number of new challenges in daily use.
In a system with cognitive faculties, individual or interconnected objects therefore cannot only
perceive the physical environment but are also able to learn from the wealth of experience
gained, derive new insights, understand contexts, and make important decisions in a supportive
or autonomous manner. Essential cognitive faculties are:
47
48
§ Perception. Fusion and interpretation of a multitude of sensors, stimuli, and observed
behavior; removal of vagueness and ambiguity in input data, and synthesis of relevant
information such as the detection, localization, and classification of relevant
surrounding objects.
49
§ Interpretation. Construction and update of faithful representations (“digital twins”) of
the exogeneous operating environment and the endogenous “self”, based on perceived
inputs and other knowledge sources.
§ Imagination. Model-based capability of situational awareness, inductive and deductive
reasoning, planning, and for the projection into the future (and the past) based on both
exogeneous and endogenous world models, derived knowledge, and perceived input.
§ Action. Selecting and prioritizing appropriate goals for a given configuration of the
environmental models, and current beliefs and intents to achieve the selected goals by
balancing optimized performance with the need for resilience.
§ Learning. Ability to adapt and optimize situational behavior, to adapt internal models,
goal management, and planning processes dynamically, and to acquire new knowledge
through inductive and deductive reasoning.
Situational-aware reasoning, planning, and learning therefore are the main ingredients of the
sense-plan-act cycle (see Figure 4), which is the central architectural concept of cognitive
actors in the field of Artificial Intelligence (AI).
50
In other words, AI is concerned principally
46
Lexico.com
47
The selection of these cognitive capabilities is inspired by “axioms” which are considered to be necessary for machine intelligence (see
also: Aleksander, Machine Consciousness, Progress in Brain Research, Elsevier, 2005). The big questions, however, centers around
cognitive capabililies sufficient for machine consciousness.
48
See also: Metzler, Shea, Taxonomy of Cognitive Functions, Engineering Design, 2011.
49
Guidance, Navigation, and Control (GNC)
50
Russell, Norvig, Artificial Intelligence: a modern approach, Pearson, 2021.
EI Systems Challenges
16
with designing the internals of a stream-transforming cognitive actors for mapping from a
stream of raw perceptual data to a stream of actions, whereas EI primarily focuses on perception
of and actions on a physical world.
FIGURE 4. SENSE-PLAN-ACT CYCLE.
Designs for cognitive actors vary enormously depending on the nature of the operating
environment, the nature of the perceptual and motor connections between actors and
environment, and the requirements of the task. For instance, cognitive faculties have been
categorized into System 1 (fast) capabilities for performing intuitive, automated tasks that we
can do instinctively, and System 2 (slow) capabilities for performing tasks that require
conscious decision making and may be described verbally.
51
There has always been a tendency in AI to use a heterogeneous set of techniques for realizing
these two categories of cognitive capabilities, as machine implementations of System 1
functionality are often based on logic and probability theory (“symbolic”), and the
implementation of System 2 functionality primarily uses connectionist (“sub-symbolic”)
approaches, which are inspired by the networked neural structure of the brain. Building on these
two pillars of AI, cognitive architectures such as Soar
52
integrate logic with connectionist
techniques for realizing autonomous robotic (both virtual and physical) systems that have some
basic cognitive abilities of humans, but also in search of a unified theory of cognition
53
. With
a similar motivation in mind, neuro-symbolic programming proposes integrated frameworks
which have the pure neural, logical, and probabilistic methods as special cases.
54
Symbolic
reasoning capabilities, however, may also be encoded with connectionist networks.
51
According to the global workspace theory of consciousness.
52
Laird, The SOAR Cognitive Architecture, MIT Press, 2012.
53
Newell, Unified theories of cognition, Cambridge 1990.
54
Raedt, Manhaeve, Dumancic, Neuro-symbolic = neural + logic + probabilistic, IJCAI, 2019.
30RESEARCH LINES
Joint Action Planning
AI Engineering @ fortiss
2.3
Joint Action Planning
Authors:
Klemens Esterle, Patrick Hart, Tobias Kessler
With robotic systems (drones, cars, or medical robots) being developed with
increasing capabilities to act autonomously, they will usually rely on a recogni-
ze-act-cycle (sense-plan-act or perception-deliberation-execution) as shown in
Figure 6. Specifically, the deliberation module calculates the system’s action ba-
sed on its prior or perceived knowledge. As part of the deliberation component,
joint action planning is an advanced research line dealing with action planning in
multi-agent settings, such as systems with multiple interacting intelligent agents
(humans or autonomous systems). These methods find the robot’s best action
(which we will denote as “ego agent”) by planning, predicting, and evaluating all
other agents’ actions and reactions in conjunction with the ego agent. Ot-
her agents may interact with the ego agent, for example, in a cooperative way
(swarms of service drones) or a more competitive manner. If the action plan
does not account for these interactions accurately, the overall system’s safety is
at risk.
When engineering autonomous systems, the system requirements that need
to be defined are manifold. Generally speaking, autonomous systems are expec-
ted to operate safer than humans. Broken down to the deliberation component,
Perception
• Multisensor
Data Fusion
• Object Detection
• Object Classification
• Intent Recognition
Deliberation
• Desires & Goals
• Strategy
• Plan
• Trajectory
Execution
• Action Selection
• Priority
• Reflexes
• Precise Control
In Out
Context
Model
Dependable Cognitive System (based on dependable AI)
Action
Plan
World / Environment
Figure 6.
-
EI Systems Challenges
17
Intent-driven Systems
Everything an actor needs to know about its goals and expected actions must be defined by
means of intents of both human and machine actors. The aim of intent-driven systems is to
capture relevant intents and to act in accordance with these intents in an optimized and resilient
manner.
55
Intent-driven networks, for example, can predict faults to proactively optimize
performance and to carry out repairs.
56
Intents of actors are ideally expressed explicitly and declaratively – that is, as utility-level
goals
57
that describes the properties (what?) of a satisfactory outcome rather than prescribing a
specific solution (how?). This gives the system the flexibility to explore various solution
options and find an optimized one. It also allows the system to optimize by choosing its own
goals that maximize utility. One of the benefits of expressing intents as utility-level goals is
that it supports the system to cope with conflicting objectives of multiple intents. This is vital,
because an embodied actor often must take multiple intents into account before taking a
decision.
Unlike traditional software-intensive systems, where requirements are analyzed offline to
detect and resolve conflicts prior to implementation, intents are added to an embodied system
and modified during operation. Adaptation to changed intent as well as conflict detection and
resolution are therefore essential capabilities of embodied systems.
One of the main challenges for specifying goals and intents of embodied systems operating in
the real world, however, is that there is very little chance that we can specify our objectives
completely and correctly in such a way that the pursuit of those objectives by more capable
machines has beneficial outcomes.
58
In particular, we can expect a sufficiently capable machine
pursuing a fixed objective to take preemptive steps to ensure that the stated objective is
achieved, including acquiring physical and computational resource and defending against any
possible attempt to interfere with goal achievement.
59
This ability to recognize plans, goals, and intents of other actors enables basic cognitive
capabilities to reason about what other actors are doing, why they are doing it, and what they
will do next. Consider, for example, a robotic butler for doing all the boring, repetitive tasks
that we wish we didn’t have to do. Ultimately, we want household assistants that can anticipate
our intentions and plans, and provably act in accordance with them. Building these kinds of
55
Silvander, Wnuk, Svahnberg, Systematic literature review on intent-driven systems, 2020.
56
Huawei. Huawei Launches the Intent-Driven Network Solution to Maximum Business Value, 2018. (see:
http://www.huawei.com/en/press-events/news/2018/2/Huawei-Launches-the-Intent-Driven-Network-Solution )
57
Including those that may have been considered “common sense” in human-operated systems.
58
“So tell me what you want, what you really, really want…“ (Spice Girls, Wannabe)
59
Russell, Artificial Intelligence and the Problem of Control, 2021.
EI Systems Challenges
18
proactive assistant systems requires plan, action, and intent recognition for accurately capturing
and tracking the operator’s requirements.
60
Plan, goal, and intent recognition also enables different actors to negotiate with each other on
behalf of their peers’ intent, as the basis for a potential mutually beneficial collaboration. For
example, one actor may have the intent to deliver high-quality service, while another may want
to minimize resource spending. Current AI technology resolves such conflicts either explicitly
from weights that introduce relative importance or implicitly from properties of preferential
outcomes as defined in utility-level goals.
Federated Systems
A federated system consists of an ensemble of interacting actors, both machines and humans,
which are collaborating to jointly achieve a common task or goal through the mutual provision
of actions that individual systems alone cannot achieve.
61
62
In doing so, these ensembles exchange relevant information, negotiate their goals, plans and
intentions, and they adapt their own actions to the negotiated plan. This aspect of negotiation,
that is the ability to confer with another as to arrive at the settlement of some matter
63
is a central
activity, and the realization of “confer” and “arrive at settlement” are among the challenges for
designing collaborative, federated systems.
64
Especially the community of distributed AI has
developed a range of models of collaborative plans in the face of resource constraints and
uncertainty.
Collaborative federated systems are often safety-critical, operate in dynamic contexts, and must
be capable of reacting to unforeseen situations without human intervention. On the one hand,
they must be able to handle uncertainties due to the imprecision of sensors and the behavior of
data-driven components for perceiving and interpreting the context to enable decisions to be
made during operation. On the other hand, uncertainties can emerge from the collaboration in
a collaborative group, related to the exchange of information (e.g., context knowledge) between
collaborative systems. Uncertainty that can occur during operation should be considered
systematically during engineering to enable collaborating systems to cope with uncertainties
autonomously.
Effective collaboration strongly relies on the assumption that most, if not all, actors operate as
expected. Therefore, a level of trust and distrust between actors needs to be established as part
60
Sukthankar et al (Eds.)., Plan, action, and intent recognition, 2014.
61
Böhm et al., Model-based Engineering of Collaborative Embedded Systems, Springer Nature, 2021.
62
Grosz, Collaborative Systems, AAAI-94 Presidential Address, AI Magazine, 1996.
63
Mish (Ed), Ninth New Collegiate Dictionary, 1988.
EI Systems Challenges
19
of the collaboration.
65
Measures need to be taken to tolerate a certain number of non-
collaborating and ineffective actors, but also actors with malicious intentions. In particular, the
security design of current decentralized systems with their heterogeneous components,
including components-off-the-shelf (COTS) and software of unknown pedigree (SOUP) start
with a zero-trust model, which is then mitigated with the right mix of security measures to
create challenging barriers for attackers, including pervasive authentication and corresponding
checks of all interactions.
Autonomous Systems
Current industrial practice is mainly concerned with developing remotely operated vehicles
(“teleoperated driving”), self-operated systems for restricted time periods and for restricted
objectives such as remotely piloted air systems in case of a lost data link, pilotless underwater
vehicles, and driverless metros in controlled urban environments. Yet we are only at the very
beginning of a new generation of autonomous systems, which are characterized by increasingly
autonomous behavior in increasingly complex environments, fulfilling missions of increasing
complexity, the ability to collaborate with other machines and humans, and the capability to
learn from experience and to adopt their behavior appropriately.
As it is designed to perform the equivalent operational tasks of understanding through
experiencing and sensing, an autonomous system therefore may be viewed as a technical
implementation of cognition. These systems perform and integrate heterogeneous sets of tasks
based on an overall situational assessment.
In contrast to mere automation, increasingly autonomous systems employ a never-give-up
strategy even in the face of real difficulties, say inconsistencies, unforeseen situations, and
authority limits (see Robotic companions). Potential benefits include increased safety,
reliability, efficiency, affordability, and previously unattainable mission capability. Clearly,
with more autonomy comes more and different forms of responsibility.
In the absence of an adequately high level of autonomy that can be relied upon, substantial
involvement by human supervisors and operators is required. Increasingly autonomous systems
support humans in daily routine tasks, have humans in the loop for continuous control of
evolution of subsystems, and ask humans for high-level decision-making. This kind of mixed
human-machine systems creates significant new challenges in the areas of human-machine
collaboration and mixed initiative control.
The overarching goal is in achieving a sufficient mutual understanding of state and intent of
both humans and technical systems as to optimally blend their competences in jointly acting
towards overarching objectives, while respecting privacy. The challenge here is to model
EI Systems Challenges
20
human behavior interactions and to provide the appropriate uncertainty characteristics related
to the largely unpredictable behavior of humans under unforeseen circumstances. Moreover, as
individual spheres of control may overlap arbitrarily, there is a pronounced need for
orchestrating these processes such that they jointly serve, say, not only a single human, but can
best-possibly multi-task in serving arbitrarily large groups at the same time despite uncorrelated
requests and uncoordinated missions.
Self-Learning Systems
One of the distinctive features of embodied systems is their ability to continuously improve
their knowledge and capabilities through experience, both positive and negative, and targeted
exploration in the real world. This ability of self-learning through exploration is, of course, a
far cry from supervised machine learning schemes for synthesizing, say, neural network
representations for approximating functions from given input-output samples. The “learned”
behavior may often not be transferred to other operating contexts. An “end-to-end” autonomous
controller, for example, might work well for, say, driving around in the outskirts of Phoenix for
which it was trained. But we most probably expect it to fail miserably when put into the streets
of Algiers or to navigate around the Gate of India. This is because the neural network controller
has not actually learned to drive, as we as humans are supposed to when visiting driving school.
It was just trained to mimic a context-specific set of driving scenarios, and there is no reason to
expect it to generalize its behavior to radically new contexts.
In contrast, autopoietic systems extend autonomously their perception and attention, their
situational representation and interpretation of the perceived world, their actions and their
collaboration patterns, and they are able to communicate such learned capabilities with other
systems:
66
“An autopoietic machine is a machine organized (…) as a network of processes of
production (transformation and destruction) of components which: (i) through their
interactions and transformations continuously regenerate and realize the network of processes
(relations) that produced them; and (ii) constitute it (the machine) as a concrete unity in space
in which they (the components) exist by specifying the topological domain of its realization as
such a network”. The ability of unsupervised learning by means of interacting with its operating
context (including the “self”) is the major characteristic of autopoietic systems. This is close to
human behavior and possibly also the ultimate dream of AI.
Achieving higher levels of autonomy in open, that is uncertain, unstructured, and dynamic,
environments and terrain increasingly involves data-driven machine learning techniques with
many open systems science and engineering challenges. The prevalent approach in autonomous
driving, for example, aims at reducing the uncertainty of the operating context by compiling
and continuously extending huge sets of driving scenarios which sufficiently (up to tolerable
quantities?) cover all possible situations; as the basis for continuous self-learning ecosystems
of a global scale.
66
Maturana, Varela, Autopoiesis and cognition: the realization of the living, Kluwer, 1980.
EI Systems Challenges
21
New and more efficient control regimes for reliable and safe exploration of unknown terrain
are clearly needed, as embodied systems must necessarily act with complete, uncertain, and
even inconsistent models of the world. Possible approaches that are being currently pursued
include the learning and use of causal models, employment of an ensemble of models, and
multifaced understanding.
67
67
Minsky: “You don’t really understand something if you only understand it one way”.
EI Systems Challenges
22
4. TRUSTWORTHINESS
Embodied systems are a new generation of increasingly autonomous systems operating in real-
world societal contexts. Thus, actions of embodied systems do matter.
The autonomous behavior of embodied systems also implies a real danger of losing control as
self-learning systems may exhibit emergent behavior, they evolve much faster than we as
humans may even comprehend, and they are able to self-organize in increasingly powerful
dynamic federations. These essential features of embodied systems make it even harder to
ensure that a meaningful (human) control over an embodied system is enabled in the field.
Without embodied systems – and the human beings behind them – being demonstrably worthy
of trust, unwanted consequences may ensue, and their uptake might be hindered, preventing the
realization of the potentially vast social and economic benefits that they can bring.
Trustworthiness therefore is a prerequisite for people and societies to develop, deploy and use
embodied systems in a meaningful manner.
Trust may be viewed either as a belief, attitude, intention, or behavior, and as such it is a
complex notion in itself. It is most generally understood as a subjective evaluation of a truster
on a trustee about something in particular; for example, the completion of a task.
68
A classical
definition from organization theory defines trust as the willingness of a party to be vulnerable
to the actions of another party based on the expectation that the other will perform a particular
action important to the trustor, irrespective of the ability to monitor or control that party.
69
An
expert group commissioned by the EC has recently identified three main ingredients for
trustworthy AI-based systems;
70
specifically, they recommend trustworthy systems to be at
least:
§ Lawful; that is, complying with applicable laws and regulations,
§ Ethical; that is, ensuring adherence to applicable ethical principles and values,
§ Robust; that is, predictable and sustained functionality in the face of uncertainty, faults,
and malicious attacks.
In addition, respecting the vast amount of, largely, unwritten norms for accepted social behavior
is also instrumental for building up trust.
68
Hardin, Trust and trustworthiness. Russell Sage Foundation, 2002.
69
Mayer, Davis, Schoorman, An integrative model of organizational trust. Academy of management review, Academy of Management
Review, 1995.
70
https://digital-strategy.ec.europa.eu/en/policies/expert-group-ai
EI Systems Challenges
23
Explicit lawful and ethical actors have been proposed based on implementing legal theories,
humanlike competence, and ethical theories predicated on virtue ethics, deontology, and
consequentialism.
71
For example, when ethical principles are in conflict, they attempt to work
out reasonable resolutions. For contexts where informing others of one‘s intention and
reasoning is crucial, these actors communicate and even defend their reasoning.
Explicit lawful and ethical actors, however, are inevitably tied to a specific societal context.
Societal-scale CPS therefore have been proposed that are parameterized by social contexts. This
approach is based on (1) understanding the nature, scope, and evolution of policies in the
operation of societal-scale CPS in different societies, (2) investigating methods for the explicit
formal representation of societal context, and (3) developing architectures that guarantee the
enforcement of policy requirements.
72
Human agency and
oversight
Including fundamental rights, human agency, and human
oversight.
Technical robustness
and safety
Including resilience to attack and security, fall back plan and
general safety, accuracy, reliability, and reproducibility.
Privacy and data
governance
Including respect for privacy, quality and integrity of data, and
access to data.
Transparency
Including traceability, explainability, and communication.
Diversity, non-
discrimination, and
fairness
Including the avoidance of unfair bias, accessibility and
universal design, and stakeholder participation.
Societal and
environmental wellbeing
Including sustainability and environmental friendliness, social
impact, society, and democracy.
Accountability
Including auditability, minimization and reporting of negative
impact, trade-offs, and redress.
TABLE 1. PRINCIPLES BY THE EC FOR RESPONSIBLE AND TRUSTWORTHY AI.
73
We shortly describe different accounts in the EU and the US for trustworthy AI-based systems.
Table 1 summarizes the lawful, ethical, and robustness attributes for responsible and trustworthy
AI as developed on behalf of the European Commission. Figure 5displays these practices,
which are recommended to be implemented and continuously evaluated throughout the
system’s lifecycle. These are strong recommendations indeed, as current, largely manual,
auditing frameworks are way to inefficient, slow, and error-prone for self-learning and ever-
evolving systems, and we currently do not have adequate technical means for automated
compliance audits. Moreover, trade-offs usually need to be made in real-life engineering for
addressing seemingly contradictory attributes such as privacy and transparency.
71
Scheutz, The case for explicit ethical agents, AI Magazine, 2017.
72
NSF PIRE 16-571: Science of Design for Societal-Scale Cyber-Physical Systems
73
https://www.aepd.es/sites/default/files/2019-12/ai-ethics-guidelines.pdf
EI Systems Challenges
24
FIGURE 5. ATTRIBUTES OF TRUSTWORTHY AI.
74
In a similar vein, the principles in Table 2 by the ACM for transparency and accountability
75
are designed to increase trust in all kinds of algorithmic systems.
Awareness
Owners, designers, builders, users, and other stakeholders of analytic
systems should be aware of the possible biases involved in their design,
implementation, and use and the potential harm that biases can cause to
individuals and society.
Access and
redress
Regulators should encourage the adoption of mechanisms that enable
questioning and redress for individuals and groups that are adversely
affected by algorithmically informed decisions.
Accountability
Institutions should be held responsible for decisions made by the
algorithms that they use, even if it is not feasible to explain in detail how
the algorithms produce their results.
Explanation
Systems and institutions that use algorithmic decision-making are
encouraged to produce explanations regarding both the procedures
followed by the algorithm and the specific decisions that are made. This
is particularly important in public policy contexts.
74
Source: EU High-Level Expert Group on AI (https://www.aepd.es/sites/default/files/2019-12/ai-ethics-guidelines.pdf)
75
ACM Code of Ethics (http://ethics.acm.org)
EI Systems Challenges
25
Data
Provenance
A description of the way in which the training data was collected should
be maintained by the builders of the algorithms, accompanied by an
exploration of the potential biases induced by the human or algorithmic
data-gathering process. Public scrutiny of the data provides maximum
opportunity for corrections. However, concerns over privacy, protecting
trade secrets, or revelation of analytics that might allow malicious actors
to game the system can justify restricting access to qualified and
authorized individuals.
Auditability
Models, algorithms, data, and decisions should be recorded so that they
can be audited in cases where harm is suspected.
Validation and
Testing
Institutions should use rigorous methods to validate their models and
document those methods and results. In particular, they should routinely
perform tests to assess and determine whether the model generates
discriminatory harm. Institutions are encouraged to make the results of
such tests public.
TABLE 2. PRINCIPLES BY THE ACM FOR TRANSPARENCY AND ACCOUNTABILITY.
76
Due to the relative similarity of the underlying systems of social values we may observe a strong
overlap of the trustworthiness requirements in the EU and the US. However, the ACM
principles on transparency and accountability tend to be phrased more in technical terms, and
consequently they seem to be more amenable to automated compliance. On the other hand, the
ACM principles on transparency and accountability have clearly been formulated with current
AI/ML techniques in mind. As such they do not adequately address the characteristics of the
upcoming generation of increasingly autonomous and self-learning systems. Particularly, there
might not even be clearly identifiable “institutions” anymore for operating an embodied system,
and embodied systems might eventually need to be held responsible for their very own actions.
Also, the demand on data provenance as formulated above may not be applicable to continuous,
unsupervised learning. Depending on the intended role of embodied systems in societal
contexts – ranging from mere assistants to fully autonomous actors - the ACM principles on
transparency and accountability consequently therefore need to be revised, developed further,
and agreed upon to fit the characteristics of a new generation of increasingly autonomous and
self-learning systems. We also need to develop techniques for ensuring such principles of
algorithmic transparency and accountability for dynamic federations of increasingly
autonomous, learning-enabled, and embodied systems.
76
ACM Code of Ethics (http://ethics.acm.org)
EI Systems Challenges
26
FIGURE 6. MAPPING FROM MAIN CHARACTERISTICS OF EMBODIED SYSTEMS TO CORRESPONDING
SYSTEM CHALLENGES.
EI Systems Challenges
27
5. CHALLENGES
Traditional system engineering comes to a juncture from assuring quality-of-service,
dependability, and safety attributes for relatively small-scale, centralized, deterministic and
predictable, non-evolvable, automated embedded systems operating in well-defined and
predictable environments to assuring the trustworthiness of larger-scale, federated, non-
predictable, self-learning, and increasingly autonomous embodied systems operating in
uncertain and largely unknown environments. These differences between embedded and
embodied systems are also summarized in Figure 7.
Embedded Systems
Embodied Systems
Architecture
centralized
federated
Behavior
deterministic
largely unpredictable
Context
well-defined
uncertain
Maintenance
updates
self-learning
Requirement
dependability
trustworthiness
Human control
yes
increasingly no
FIGURE 7. FROM EMBEDDED TO EMBODIED SYSTEMS.
Thereby, we can build up trust by engineering embodied systems which are lawful, ethical, and
robust. Compared with traditional embedded systems engineering we face additional
challenges; in particular, embodied systems:
§ Learn continuously and they adapt and optimize their behavior based on experience
and targeted exploration.
§ Need to safely operate in partially unknown or uncertain environments, and they need
to be robustness in the presence of inaccuracies, uncertainty, and errors in their world
models (“known unknown”) and also in the presence of non-modeled phenomena
(“unknown unknown”).
§ Increasingly lack the fallback to a responsible human being.
§ Offer a variety of new attack surfaces due to data-driven programming.
77
§ Exhibit largely unpredictable and emergent behavior due to data-driven programming.
§ Cannot be certified as current certification regimes require the system’s behavior and
its intended operating context to be fully specified and verified prior to commissioning.
77
vulgo: machine learning.
EI Systems Challenges
28
For the continuous evolvability and self-organizing capabilities of embodied systems, the
traditional design-build-commission-decommission life cycle of embedded systems is
inadequate and clearly needs to be replaced with a design-operation continuum based on the
combined functionality for design, simulation/verification, deployment, operation, and
maintenance.
Given the expected lifetime of embodied systems, they must be designed to cope with changing
underlying technologies and hardware, changing regulatory settings, changing societal
contexts, changing system requirements, and yet provide the same high level of dependability
and quality of service throughout their evolution. It is even perceivable that certain design and
engineering steps, including situational risk management, are eventually performed by the
embodied systems themselves.
Based on the characteristics of embodied systems as outlined in the Section 3 we are now
deriving all-important systems challenges for developing, deploying, and operating trustworthy
embodied systems. A high-level summary of this derivation is provided in Figure 6. Notice,
however, that Figure 6 only depicts the most obvious and possibly the most important relations
between characteristics and derived challenges for engineering trustworthy embodied systems.
ROBUST AI/ML
Machine learning (ML) techniques are ubiquitous with lots of successful applications. The main
attraction is that functional requirements are stated in terms of data only, and a corresponding
program for approximating such a function is synthesized in an automated fashion. Moreover,
the engineering steps from data wrangling to architectural selection and optimization of the
function approximation is increasingly being automated.
78
However, these advantages also
come with some downsides.
§ Current learning techniques based on, say, artificial neural networks are only as good
as the available data (and their resource-intensive pre-processing and labeling
requirements).
§ There is uncertainty on the input-output behavior of the learned function approximators
are usually they are not robust in the sense that small changes to the input might result
in unexpected behavior; for example, one-pixel attacks of trained classifiers are
successful for many artificial neural networks.
§ The learned function is usually restricted to the context as encoded in the learning data
with limited transferability.
79
78
Xin, Zhao, Chu. AutoML: A Survey of the State-of-the-Art. Knowledge-Based Systems, 2021.
79
“The soccer bot lines up to take a shot at the goal. But instead of getting ready to block it, the goalkeeper drops to ground and wiggles its
legs. Confused, the striker does a weird little sideways dance, stamping its feet and waving one arm, and then falls over. 1-0 to the goalie…
EI Systems Challenges
29
§ The input-output behavior of the learned function is implicit and not transparent in that
the reasons for proposed decisions are opaque; for example, a neural network for
classifying tumors as benign or not might show human-like performance, but there
usually is no further explanation to the human diagnostician.
§ Current supervised learning algorithm based on, say stochastic gradient descent, are
rather data-intensive and inefficient; small children, in particular, have the ability to
form concepts such as “cow” based on only a few encounters.
As we gain more experience with developing and deploying machine learning-based systems
for real world challenges we realize some short-comings.
80
§ Adequate data often is not or not readily available, and crucial data for specifying the
intended behavior might only become available during development and operation.
Traditional process models with clearly defined requirements engineering phases,
however, do not adequately support the extra flexibility needed by data-driven AI
development approach.
§ Building data-driven ML systems requires a comprehensive wealth of experience, even
though this kind of expert knowledge (“which learning algorithm?”, “which
architecture?”, “which hyperparameters?”) is increasingly being captured in automated
meta-learning processes.
81
§ An ML system is often more than just machine learning and building and running it is
a serious software and systems engineering undertaking. However, generally accepted
or even standardized processes, methods, and tools for the development and operation
of predictable and transparent ML are largely missing.
§ Current ML applications usually do not adequately address trustworthiness attributes as
outlined in Section 4.
Moreover, most machine learning applications nowadays are based on supervised learning,
which does not support the required self-learning capabilities of embodied systems.
Reinforcement learning, which is based on optimizing objective functions, however, is a good
starting point, where inverse reinforcement learning might be used for intent recognition.
However, current approaches to reinforcement learning require a huge number of trials. Failed
attempts in the real world which might result into undesired behavior, accidents, or other
catastrophic events.
AI trained using reinforcement learning can be tricked by … an adversarial policy.“ (quoted from: Heaven, Reinforcement learning AI are
vulnerable to a new kind of attack, MIT Technology Review, Artificial Intelligence, 2020)
80
Standards on engineering trustworthy autonomous systems are currently emerging; in particular VDE-AR-E 2842-61 of the standardization
organization DKE (cmp. https://www.dke.de/de/normen-standards/dokument?id=7141809&type=dke|dokument )
81
For example, AutoML.
EI Systems Challenges
30
Techniques for safe and predictable self-learning and exploration of uncertain and
unpredictable real worlds are still in their infancy. Promising approaches for tackling the added
complexities of autonomous actors in real world settings include hybrid combinations
82
of
classical and learning-based algorithms and the incorporation of prior knowledge.
83
How can one learner who does not know what there is to learn manage to learn anymore?
Current machine learning approaches usually start with what needs to be learned for learning.
We as humans, however, can discover both the tasks to be learned and the solution to those
tasks through exploration, or non-goal-directed action.
Machine learning has mainly concentrated on non-incremental learning tasks, tasks in which
the entire training set is fixed at the start of learning and then is either presented in its entirety
or randomly sampled. Embodied actors, however, need to learn incrementally and continuously
through exploration.
Machine learning also is increasingly being augmented with domain-specific knowledge and
rules for increasing the efficiency and effectiveness of machine learning, rules and decision
trees might also be compiled from learned behavior, which themselves can be used for
improving further learning but also for making decisions transparent, say, to a human operator.
In this way, domain knowledge, such as physical laws, is currently integrated into machine
learning, by using techniques such as regularization, data augmentation, or post-processing.
A recent survey on knowledge-augmented machine learning
84
reviews the role of knowledge
in machine learning, and it discusses its relation to the concept of invariance. Among others,
neuro-symbolic integration (with logic, probability theory, and neural structures as projections)
has been proposed as the basis for a new generation of dependable, predictable, transparent,
and efficient data-driven programming techniques for realizing increasingly autonomous
human assistants and/or for mission- and safety-relevant applications.
Altogether, despite technological advances that have led to the proliferation of data-driven
machine learning systems, there still is the question of the level of trust that we can put on these
systems. A new generation of robust machine learning algorithms therefore is needed that,
85
§ in uncertain and largely unpredictable environments,
§ can make timely and confident decisions,
§ whose results are understandable and explainable to a human operator,
§ that are resilient to erroneous inputs and targeted attacks,
82
Chaplol et al, Learning to explore using active neural slam, ICLR, 2020.
83
Xin Ye and Yezhou Yang, From seeing to moving: As survey on learning for visual indoor navigation, arXiv:2002.11310, 2020.
84
https://www.fortiss.org/fileadmin/user_upload/05_Veroeffentlichungen/Whitepaper/fortiss_whitepaper_knowledge_as_invariance_web.pdf
85
Stoica et al., A Berkeley View of Systems Challenges for AI, 2017 (https://arxiv.org/pdf/1712.05855.pdf).
EI Systems Challenges
31
§ that can process ever-increasing amounts of data,
§ from decentralized and heterogeneous data sources,
§ but can also extract useful insights from small amounts of data and sparse rewards,
§ without significant compromises in confidentiality and privacy in federated multi-actor
settings.
Traditionally, ML modeling techniques have relied on unsiloing data from multiple sources
into a single data lake. Centralized data sources, however, pose serious privacy, data misuse,
and security challenges for federated systems. Also, aggregating diverse data from multiple
sources needs to meet regulatory concerns such as GDPR, HIPAA, or CCPA (we will be
reconsidering these issues in the subsection below on Assurance). To overcome these
challenges, several pillars of privacy preserving machine learning have been developed for
unsiloing ML models with specific techniques that reduce privacy risk and ensure that data
remains reasonably secure, namely federated machine learning, secure multi-party
computation, differential privacy, and homomorphic encryption.
Altogether, the prevailing methods for machine learning do not map to the ways that humans
learn, as humans learn by seeing, moving, interacting, and speaking with others. Humans learn
from sequential experiences, not from shuffled and randomized experiences. We need to come
up with a new generation of machine learning techniques, possibly mimicking the ways humans
learn, as to enable efficient self-learning for trustworthy embodied systems through targeted
exploration and experience.
HUMAN-CENTERED AI/ML
Embodied systems are machines to support humans in daily routine tasks, have humans in the
loop for the continuous supervision of the evolution of subsystems, and ask humans for high-
level decision-making. The central challenge, as addressed in the field of human-centered
engineering, is to enable symbiotic relationships, in which embodied systems and humans
augment each other reciprocatively,
86
and as the basis for co-evolutionary improvement of both
machines and humans.
87
Human and machine need to avoid “mode confusions” based on a mutual understanding of state
and intent of both humans and machine as to optimally blend their competences in jointly acting
towards overarching objectives, while respecting privacy. Moreover, in the absence of an
adequately high level of autonomy that can be relied upon, substantial involvement by human
86
Cmp. IBM high-level AI framework.
87
Engelbart, The Bootstrap Paradigm (https://dougengelbart.org/content/view/248/)
EI Systems Challenges
32
supervisors and operators is required. This creates significant new challenges in the areas of
human-machine interaction and mixed initiative control.
Embodied systems are learning-enabled. But, as discussed above, current ML techniques offer
new attack surfaces, are largely non-transparent (implicit models), tend to be energy and data
hungry, and they lack basic transferability capabilities as required for navigating unknown or
uncertain terrain.
88
It is therefore unclear if and how these technologies can be used beneficially
in real-world applications requiring human-machine interaction or in mission- and safety-
critical applications.
89
Moreover, the apparent success of ML in producing seemingly intelligent decisions brings
along dangerous causes for misunderstandings in the communication between humans and
machines. If we compare the behavior of ML systems and humans in decision making,
significant differences are obvious. ML essentially provides efficient algorithmic solutions for
optimizing a well-defined target function, enabling the learning of task- and data-specific
patterns from a huge number of samples or observations.
In contrast, a human would rather make decisions based on ground truth rules like causality and
can transfer known solutions to new situations and domains. Although both types of decision
making can be called forms of generalization, the human way of decision making is a harder
form of generalization, sometimes termed horizontal, strong, or out-of-distribution
generalization. Human decision making takes advantage of heterogeneous information sources
such as interventions, domain shifts and temporal structures, which ML typically discards or
even fails to model in learning processes.
These shortcomings of ML lead to serious challenges in designing trustworthy systems based
on machine learning for human users; in particular:
§ Low explainability: the decision-making mechanism of ML algorithms such as ANNs
cannot be made fully transparent to humans and is difficult to interpret for humans.
§ Miscalibration of trust: ML seems to be both highly effective to humans and also largely
predictable, thereby luring humans into accepting these technical systems as human-
like partners (anthropomorphization) which are trusted more than actually justified.
90
§ Low level of human control and involvement: most ML algorithms rely on either a
hypothetical model of the distribution of data or concrete interpretation (labeling) of
data. Such constructions have become one major hurdle to enabling ML systems with
high levels of human control, such as human-like reasoning and generalization.
88
E.g. Keynote of AAAI President, http://web.engr.oregonstate.edu/~tgd/talks/dietterich-aaai-presidents-address-final.pdf
89
See: https://www.fortiss.org/fileadmin/user_upload/05_Veroeffentlichungen/Whitepaper/fortiss_whitepaper_HCML_web.pdf
90
This phenomenon has previously also been demonstrated by Feigenbaum’s Eliza program.
EI Systems Challenges
33
Specifically, it is rather difficult to find the right level of human control on which the
system can effectively communicate with humans to obtain such input.
The challenge here is to model human behavioral interactions with the technical system and to
provide the appropriate uncertainty characteristics related to the largely unpredictable behavior
of humans under unforeseen circumstances. Moreover, as individual spheres of control may
overlap arbitrarily, there is a pronounced need for orchestrating these processes such that they
jointly serve, say, not only a single human, but can best-possibly multi-task in serving arbitrarily
large groups at the same time despite uncorrelated requests and uncoordinated missions.
This is particularly challenging for complex mission tasks calling for collaboration and teaming
among humans and machines. In these cases, AI-enabled systems may need to identify the
“real” intent of human operators and their goals, interact with them in a goal-oriented manner
based on models of human behavior, and, in extreme cases, also tolerate and adequately
mitigate seemingly irrational behavior.
Building trust based on explainability (“how?”, “why?”, “what-if?”) is essential for human
operators to accept ML-based solutions and those systems incorporating decisions made by
them. Explainability therefore is particularly useful
§ for increasing the confidence of human operators,
§ for building trust by supporting an increased understanding of the transferability of
results to other problems of interest,
§ for avoiding misconception and ensuring that humans understand outcomes of learning-
enabled components, as a solid basis for intervening human actions, and
§ for increasing human confidence in the decisions and predictions made by a learning-
enabled component.
There are, however, significant challenges in developing adequate methods for explainability.
One of them is the trade-off between attaining the simplicity of algorithmic transparency and
impacting the high-performing nature of complex but opaque ML models. Yet another
challenge is to identify the right information for the user, where different levels of knowledge
will come into play. Beyond selecting the level of knowledge retained by the user, generating
a concise explanation also becomes a challenge. Most existing methods for explainability,
however, focus on explaining the processes behind an ML-based decision, which is often
useless in a particular application domain. In addressing these issues, current research is
integrating ML algorithms with domain-specific (for example, laws of physics) and, possibly
learned, knowledge.
EI Systems Challenges
34
COGNITIVE ARCHITECTURES
The field of cognitive architectures creates programs than can reason about problems across
different domains, develop insights, adapt to new situations, and even reflect on themselves.
These programs realize cognitive functions including perception, memory, attention, social
interaction, planning, motivation, actuation, reasoning, communication, learning, emotion,
modeling self/other, building/creation and arithmetic capabilities.
Prominent cognitive architectures include Soar, ACT-R, LIDA, CLARION, and EPIC.
91
However, with no clear definition and general theory of cognition, there are several hundreds
more based on different sets of premises and assumptions, also coming from various
backgrounds (computer science, psychology, philosophy, and neuroscience).
92
It is not even clear at all what constitutes a cognitive architecture. Newell’s criteria for a
cognitive architecture, for instance, include flexible behavior, real-time behavior, rationality,
large knowledge base, learning, development, linguistic abilities, self-awareness, and brain
realization.
93
Sun’s desiderata are broader and include, among others, cognitive realism,
adaptation, modularity, routineness, and synergistic interaction.
94
Many of these criteria for
cognitive system are clearly of general interest also for the class of embodied systems as defined
above.
The obvious question therefore is, if and how principles of cognitive architectures are aiding in
the design of embodied systems.
95
For example, it has been recognized that machinery which
is expected to behave “correctly” in a complex world may be akin to a model-based reflective
predictive controller of a machine with a mission.
96
97
Some cognitive architectures use one uniform representation and corresponding learning
method yielding “grand unification and functional elegance”,
98
but loosing expressiveness.
Others utilize quite general knowledge representations and many inference strategies
99
that
result in higher expressiveness, but they cause difficulties with integrations of different
components of the cognitive architecture. A substantial number of cognitive architectures are
hybrid (for example, Soar, ACT-R, LIDA, CLARION, EPIC) in that they are combining both
91
Nancy, Balamurugan, Vijakkumar, A Comparative Analysis of Cognitive Architecture, JARTET, 2016.
92
Kotseruba et al. A Review of 40 Years of Cognitive Architecture Research, arXiv:1610.08602, 2016
93
Anderson, Lebiere, The Newell Test for a Theory of Cognition, Beh. Brain Sci., 2003.
94
Sun, Desiderata for cognitive architectures, Philos. Psychol., 2004.
95
Irrespective of their relative in explaining the development of higher-level intelligent behavior and consciousness.
96
Sanz, Lopez, Rodriguez, Hernandez, Principles for consciousness in integrated cognitive control, Neural Networks Society, 2007.
97
Sanz, Thinking with the body: towards hierarchical scalable cognition, Handbook of Cognitive Science, An Embodied Approach, 2008.
98
Rosenbloom, Extending Mental Imagery in Sigma, LNAI 7716, 2012.
99
Goertzel, Pennachin, Geisweiller, Engineering General Intelligence, Atlantis Press, 2014.
EI Systems Challenges
35
symbolic and sub-symbolic reasoning, thereby providing architectural concepts for integrating
connectionist
100
with logic-based AI technologies.
Probabilistic programming provides yet another framework in which basic components of
cognitive architectures are represented in a unified and elegant fashion.
101
This probabilistic
model cognition is destined to support aleatoric uncertainty, that is the “known unknown”.
Notice also that probabilistic programming suggests a programming model for embodied
systems based on well-known concepts program construction in computer science for
specifying, developing, analyzing, synthesizing, and composing programs.
There are numerous demonstrations of cognitive architectures for performing real-world
including navigation, obstacle avoidance, object manipulation, and fetch-and-carry tasks for
trash collecting
102
or soda collecting
103
mobile robots. Applications from industrial domains
include robotic crane operation,
104
bridge construction,
105
autonomous cleaning and deburring
workstation,
106
an automated stamp distribution center,
107
and an analytics engine as inspired
by the HTM cognitive architecture.
108
Cognitive architectures have also proven to be useful
for human performance modeling, human-robot interaction, natural language processing,
categorization and clustering, and computer vision.
Cognitive architectures might be able to support active perception
109
for coupling perception
with action of an embodied actor. For example, an actor may be spawned anywhere in the
environment and may not immediately “see” the pixels containing the answer to its visual goal
(for example, the car/goal may not be visible). Thus, the actor must move to succeed —
controlling the pixels that it will perceive. The actor must learn to map its visual input to the
correct action based on its perception of the world, the underlying physical constraints, and its
understanding of the question. The observations that the actor collects are a consequence of the
actions that the actor takes in the environment, and the actor is controlling the data distribution
that is coming in. The actor controls the pixels it gets to see. One of the challenges of active
perception is to be generally robust to variation.
100
Connectionist architectures are supposed to exhibit intelligent behavior without storing, retrieving, or otherwise operating on structured
symbolic expressions.
101
Potapov, A Step from Probabilistic Programming to Cognitive Architectures, arXiv, 2016
102
Firby et al., An Architecture for Vision and Action, IJCAI, 1995.
103
Brooks, A robot that walks: emergent behaviors from a carefully evolved neural network, Neural Comp., 1989.
104
Lytle, Saidi, NIST research in autonomous construction, Auton. Robots, 2007.
105
Bostelman, Bunch, Delivery of an Advanced Double-Hull Ship Welding, ICSC, Symposia on Intelligent Industrial Automation and Soft
Computing, 1999.
106
Murphy, Norcross, Proctor, CAD directed robotic deburring, Robotics and manufacturing research, education, and applications,
1988.
107
Albus, The NIST Real-time Control System (RCS): an approach to intelligent systems research, J. Exp. Theor. Artif. Intell. 1997
108
https://numenta.com/grok
109
Aloimonos (Ed.), Active Perception, Psychology Press, 1993.
EI Systems Challenges
36
Finally, cognitive architectures and theories from psychology, such as cue theory, might serve
as the basis and inspiration for designing novel control regimes for embodied actors capable of
safely exploring and navigating the “unknown unknown”. In this way, careful terrain
exploration has been approached by minimizing surprises, for example, based on active
inference
110
and the free energy principle,
111
112
or, alternatively, by maximizing predictive
information.
113
UNCERTAINTY QUANTIFICATION
There is indeed a multitude of sources for uncertainty in the design and operation of embodied
systems, as there is uncertainty about their operational context (for example, how many and
which objects and actors are in the environment), there is uncertainty about corresponding
hazards and risks, there is uncertainty about the behavior of learning-enabled components, there
is uncertainty about safety envelopes, there is controller uncertainty due to nondeterminism and
also probabilistic control algorithms, there is uncertainty on the internal models, and, last but
not least, there is also uncertainty about the intentions, behaviors, and strategies actions of other
embodied actors, both human
114
and machines.
Learning in the sense of replacing specific observations by general models is a process for
inductive inference. Such models are never provably correct but only hypothetical and therefore
uncertain, and the same holds true for the predictions produced by a model. For example, the
input-output behavior of ANNs heavily relies on the selection of “complete” and “correct” sets
of training and support data for faithfully specifying relevant operating contexts (input) and
their intended internal representation (output). Another source of uncertainty for ANNs is due
to the use of stochastic search heuristics, which may lead to incorrect recall even for inputs
from the training data, and the largely unpredictable capability of generalizing from given data
points. Uncertainty on the faithfulness of the training data representing operating contexts and
uncertainty on the correctness and generalizability of training also combine in a, well, uncertain
manner.
One usually distinguishes between aleatoric and epistemic sources of uncertainty, whereas
aleatoric
115
uncertainty refers to the variability in the outcome of an experiment which is due to
110
Active inference: maintaining a model and its predictions through action to change the sensory inputs to minimize prediction error
indirectly (if the sound is not getting louder, moving closer towards the train in order to hear the train getting louder).
111
Friston’s Free Energy Principle (FEP) is a leading formal theory of self-organizing system dynamics. It basically asserts that living
systems must minimize the entropy of its sensory exchanges with the world; for example: Friston, The free-energy principle: a unified brain
theory?, Nature reviews neuroscience, 2010).
112
Smith, A unified Framework for Intelligence based on the Free Energy Principle, 2019.
113
Ay, Bertschinger, Der, Güttler, Olbrich, Predictive information and explorative behavior of autonomous robots, European Physical
Journal B, 2008.
114
Compare with the Human-Centered AI/ML challenge above.
115
Aka statistical, experimental, or “known unknown”
EI Systems Challenges
37
inherently random effects, and epistemic
116
uncertainty refers to uncertainty caused by a lack
of knowledge.
117
For example, incomplete knowledge of an embodied actor’s operating context
is an epistemic source of uncertainty. As epistemic uncertainty refers to the ignorance of an
actor, and hence to its epistemic state, it can in principle be reduced with additional information.
The central challenge is uncertainty quantification,
118
that is, to systematically reduce
uncertainty to acceptable level, and as the basis for trustworthy and (up to tolerable quantities)
predictable embodied systems. Uncertainty quantification involves:
§ Identifying all relevant sources of uncertainty.
§ Adequately quantifying and estimating uncertainty.
§ Understanding how uncertainty accumulates, forward and inverse, along chains of
computations.
§ Reducing overall uncertainty below acceptable levels.
119
§ Managing incremental change of uncertainty.
Table 3 lists corresponding challenges for the rigorous design of embodied systems and for
managing uncertainties throughout its lifecycle. The analysis and assurance challenges are also
addressed below in the corresponding subsections on Analysis and Assurance.
There are different uncertainty-reducing techniques for robust AI systems depending on the
aleatoric or epistemic nature.
120
The basic principle of uncertainty reduction also plays a key
role in active learning
121
and in learning algorithms.
122
For example, indirect cues
123
may cause
the system to hypothesize the existence and certain objects of a relevant object, which needs to
be confirmed by additional actions. In addition, uncertainty quantification approaches in
engineering have been designed to demonstrate that, with high probability, a real-valued
response function of a given physical system does not exceed a given safety threshold.
124
!
116
Aka systematic, structural, or “unknown unknown”
117
Hüllermeier, & Waegeman, Aleatoric and epistemic uncertainty in machine learning: an introduction to concepts and methods,. Machine
Learning, 110(3), 2021.
118
Uncertainty quantification (UQ) is the science of quantitative characterization and reduction of uncertainties in both computational and
real world applications. It tries to determine how likely certain outcomes are if some aspects of the system are not exactly known.
119
For example, less than one hazardous behavior for 10^9 operational time
120
Dietterich, Steps Toward Robust Artificial Intelligence. AI Magazine, 38(3), 2017.
121
Aggarwal, Kong, Gu, Han, & Philip. (2014). Active learning: A survey. Data Classification: Algorithms and Applications. CRC Press.
122
Mitchell. (1980). The need for biases in learning generalizations. Tech. Rep. TR CBM–TR–117, Rutgers University.
123
Björkman. (1994). Internal cue theory: Calibration and resolution of confidence in general knowledge. Organizational Behavior and
Human Decision Processes.
124
Owhadi, Scovel, Sullivan, McKerns, & Ortiz. (2013). Optimal uncertainty quantification. Siam Review, 55(2), 271-345.
EI Systems Challenges
38
Specification
Challenge
• Provide means for constructing (and maintaining) safety envelopes,
either deductively from safety analysis or inductively from safe nominal
behavior.
• Provide means for minimizing uncertainties related to safety envelopes
with a given level of effort.
• Provide means for deriving safety requirements for learning-enabled
components, which are sufficient for establishing AI system safety.
• Provide means for reducing specification uncertainty by means of
deriving data requirements for learning-enabled components.
Prediction
Challenge
• Identify all relevant sources of uncertainty for an AI system.
• Provide adequate means for measuring uncertainty.
• Calculate forward propagation of uncertainty, where the various sources
of uncertainty are propagated through the model to predict overall
uncertainty in the system response.
• Identify and solve relevant inverse 125 uncertainty quantification
problems for safe AI (using, for example, a Bayesian approach).
• Predict (up to tolerable quantities) the unsafe behavior of AI systems
operating in uncertain environments.
Assurance
Challenge
• Provide adequate measures of uncertainty for assuring AI system
safety.
• Construct and maintain evidence-based arguments for supporting the
certainty and for rebuting the uncertainty of safety claims.
• Identify useful safety case patterns for safe AI systems together with
compositional operators on safety cases for managing uncertainty.
Design
Challenge
• Develop safety case patterns for different architectural designs of AI
systems.126
• Compositionally construct safe and quasi-predictable AI systems
together with their safety cases.
Analysis
Challenge
• Provide adequate means for measuring and for reducing uncertainty on
the input-output behavior of learning-enabled components.
• Define and measure the respective contribution of static and dynamic
analysis techniques for learning-enabled systems, towards reducing
safety-related uncertainty to tolerable levels.
Maintenance
Challenge
• Identify incremental change operators for maintaining uncertainty and
safety assurance of self-learning AI systems.
• Safely adapt and optimize the situational behavior of an AI system
(together with its safety cases based on the principle of minimizing
uncertainty.
TABLE 3. ENGINEERING CHALLENGES FOR UNCERTAINTY QUANTIFICATION.
125
That is, calculating from a set of observations the causal factors that produced them.
126
In analogy to, say, Mils separation kernel protection profile.
EI Systems Challenges
39
Uncertainty quantification also plays a pivotal role in reducing uncertainties for learning-
enabled components such as ANNs.
127
128
Establishing resilience
129
and other invariance
130
properties, for example, are an important means for reducing behavioral uncertainty of ANNs.
Moreover, measuring and estimating the uncertainty of the input-output behavior of learning-
enabled components is essential for, say, switching between performant and safe channels in a
Simplex architecture, and uncertainty information is useful input for planning safe actions.
Proposals for measuring behavioral uncertainty of learning-enabled components include:
§ The distance between neuron activations observed during training and the activation
pattern for the current input are used for estimating the input-output uncertainty.
131
§ Ensemble learning techniques are used for estimating input-output uncertainty by
training a certain number of ML components from different initializations and
sometimes on differing versions of the dataset; the variance of the ensemble’s
predictions is then interpreted as its epistemic uncertainty.
§ Certain instances of ensemble learning techniques such as Bayesian neural networks
measure both epistemic uncertainty on model parameters, and the aleatoric
uncertainty of the input-output behavior with respect to model parameters.
132
What we should focus on, however, is not so much about reducing behavioral uncertainty of
individual components but of the embodied system itself. Such an uncertainty on the system-
level behavior is obtained, for example, by forward propagation
133
of component uncertainties
along chains of computation.
Uncertainties can also be explicitly managed through assurance cases.
134
These structured
arguments are comprehensive, defensible, and valid justification that the system fulfills crucial
properties, at least up to a tolerable level of uncertainty, with the goal of increasing confidence
and building up trust in the behavior of an embodied system. The purpose is, broadly, to
demonstrate that the crucial risks associated with specific system concerns
135
have been
127
Czarnecki, Salay, Towards a framework for managing perception uncertainty for safe automated driving. Computer Safety, Reliability
and Security. 2018, Springer.
128
Abdar. A Review of Uncertainty Quantification in Deep Learning: Techniques, Applications and Challenges. 2020 .
129
Cheng, Nührenberg, & Rueß, Maximum resilience of artificial neural networks, ATVA, 2017, Springer.
130
With respect to certain classes of input transformations such as stretching.
131
Cheng, Nührenberg, Yasuoka, Runtime monitoring neuron activation patterns, DATE, 2019, IEEE.
132
Jospin, Buntine, Boussaid, Laga, Bennamoun, Hands-on bayesian neural networks-a tutorial for deep learning users, arXiv:2007.06823,
2020.
133
For example, based on Bayesian inference.
134
We will be revisiting assurance cases in the subsection describing the Assurance challenge.
135
Including safety and security, but also applies to all the other attributes of trustworthiness.
EI Systems Challenges
40
identified, are well-understood, have been appropriately mitigated, and that there are
mechanisms in place to monitor the effectiveness of defined mitigations. Of particular interest
is to capture how the influence of a learning-enabled component is captured and reasoned
within the control structure of an embodied system. Recent extensions of assurance cases for
reasoning about confidence and uncertainty seem to be a good starting point for a more
thorough investigation into uncertainty quantification for embodied system.
136
137
Altogether, there is an increasing interest on various aspects of uncertainty quantification for
embodied systems. What is still missing, however, is a comprehensive set of methods and tools
for the rigorous design of embodied systems based on the principle of uncertainty
quantification.
SELF-INTEGRATION
Figure 3. How and why do all the embodied actors, ranging from design, production, and
logistics, form a collaborative federation in a productive manner, thereby supporting the intent
of the buyer? Moreover, how does this federation tolerate real-world mishaps, such as a ship
getting stuck in a channel?
Intent-driven formation of purposeful federations of embodied systems requires the individual
systems to be open to collaborate with others, while still operating as self-sufficient individually
purposeful systems. Formation of these federations is based on self-integration, which seeks
out other systems to support to meet their local and global intents and goals, which cannot be
accomplished on their own.
The Semantic Interoperability Logical Framework (SILF),
138
for example, facilitates
dependable machine-to-machine information exchange, based on an extensive ontology to
describe the content of messages, and an intent-aware mediation mechanism to translate
messages as needed. These adapters may be synthesized automatically from ontological
descriptions, whereas the purpose of the integration is represented in a task ontology.
139
Notice
that SILF focuses on the composition
140
of systems but not compositionality,
141
for enabling
novel capabilities.
More recently self-integration based on contract theory and negotiation has been used to
purposefully self-integrate, for example, drones and wearable (IoT) devices. More precisely, a
136
Duan et al., Reasoning about confidence and uncertaintyin assurance cases: A survey, Software Engineering in Health Care, 2014.
137
Bloomfield, Littlewood, Wright, Confidence: its role independability cases for risk assessment, Dependable Systems and Networks, 2007.
138
NATO Science and Technology Organization, Neuilly-sur-Seine, Framework for Semantic Interoperatility, TR-IST-094.5, 2014.
139
Ford et al., Purpose-aware interoperability: the ONISTT ontologies and analyzer, Simulation Interoperability Workshop, 2007.
140
Requiring the preservation of local properties.
141
Compositionality requires the analysis of emergent properties of compositions, some of which are vital, as in safety and security.
EI Systems Challenges
41
trust negotiation protocol for IoT devices has been developed to create an assume-guarantee
contract that also includes a set of assessment procedures.
142
The contract yields additional
assurance for dynamic integration from a shared, historical record of adaptation assessment.
This additional assurance might also be managed using the concept of assurance cases.
Other examples for self-integrating systems, including mobility scenarios in which cars and,
say, traffic lights are purposefully interacting and adjusting their behavior for improving the
flow of traffic, and intensive care unit scenarios in which, say, heart-lung machines and X-ray
cameras recognize each other and negotiate their safe interaction.
143
Some of these integrated
systems could, of course, be readily constructed as bespoke one pony trick systems by suitably
skilled teams. Automated self-integration, however, promises to be more flexible, more
efficient, and less error prone. The scenarios above also demonstrate that, beyond automation
of the integration, the challenge is to provide assurance for safety of the integrated systems.
Forming intent-driven federations of increasingly autonomous embodied systems is a
challenging endeavor. Indeed, composition of more traditional systems can often introduce new
vulnerabilities,
144
as in, say, exposed crypto keys and privacy violations. We therefore need to
come up with suitable architectural principles and composition operators for constructing
resilient and safe embodied systems from a (possibly dynamically changing) set of
heterogeneous, and even untrusted, constituent systems. In this way, embodied systems may
tolerate certain failures, unexpected events, and even malicious attacks. Modeling attacks and
other hardware and software defects is an issue, since, almost by definition, cyber-attacks are
very hard to predict.
145
Yet providing some degree of resilience and to continuously improve
resilience is a must for societal acceptance of embodied systems. And for the most advanced
kinds of systems, it may be that what is needed is agreement on a shared system of ethics.
Since embodied systems are acting in the real world with their wickerwork of societal norms,
rules and laws, smart contracts are a central concept towards intent-driven dynamic federations
of embodied systems. In this way, self-integration and self-orchestration might be approached
as follows.
§ Software-based (“smart”) contracts define the service interfaces and service-level
agreements for embodied actors.
§ Federations of embodied actors are formed through conclusion of contracts; for
instance, through bidding in auctions and/or using a mediator.
§ Smart contracts are executed until the purpose have the contract has been met.
142
Riley et al, Toward a Negotiation Framework for Self-Integration, Autonomic Computing & Self-Organizing Systems Companion, 2020.
143
Rushby, Trustworthy Self-Integrating Systems, Distributed Computing and Internet Technology, 2016.
144
Neumann, How might we increase trustworthiness?, CACM, 2019.
145
Dutertre et al., Intrusion-Tolerant Enclaves, Security and Privacy, 2002.
EI Systems Challenges
42
For example, a “ship” embodied actor offers smart contracts for “shipping A from B to C in
exchange for D”. If there is a “customer” who needs to ship “a in A” from “b in B” to “c in C”,
and is willing to provide “d in D” in exchange, then the “ship” and the “customer” might want
to conclude a corresponding contract. If “c noting C”, for example, that is the “ship” did not
intend to call at harbor “c”, then the “ship” might be willing to change her route for a small
extra fee. Alternatively, mitigations of common mishaps might already be defined in the initial
set of contracts. This is, of course, just how the current contract-based economy is designed to
work.
The offering and conclusion of contracts may be realized, for instance, by means of distributed
execution of logic programs. Global invariants need to be maintained on a set of contracts. For
example, the federation in Figure 3 needs to ensure that their mutual service level agreements
enable timely and orderly delivery of the customer’s order. Other invariants need to be ensured
such as regulatory rules, desiderata such as climate neutrality, or resiliency to some breach of
contract.
Not all contracts are being served to completion. What happens, for example, if the “ship” not
able to fulfill the contract, since it got stuck, say, in some shipping channel. Now, the federation
of actors in Figure 3 may need to reorganize to still be able to satisfy the customer order on
time. Such a reorganization of the federation is based on successful renegotiation and
cancellation of contracts. This process of resilient execution of smart contracts generalizes the
fault-detection, isolation, and recovery (FDIR) cycle of fault-tolerant systems. Again, if
contracts are negotiated by means of distributed logic programs, then resilient execution and
renegotiation of contracts might, for example, be realized through backtracking and
mechanisms for distributed incremental maintenance.
A contract-based reconfiguration of a federation might also involve a change of the embodiment
of a certain service. For example, if the ship refuses or is unable to call port “c”, the delivery
federation might decide to replace this ship with other means of transportation. Using the slang
of model-driven design, such a replacement involves changing the deployment of a PDM
federation of virtualized services) to a PIM (embodiment of virtualized services in the physical
world).
Most importantly, actors and/or federations of actors need to be incentivized to honor contracts.
also need to be held responsible
146
for breach of contract, possible by some empowered higher
instance, who identifies, collects evidence, and penalizes breach of contract.
Clearly, the suggested contract-based composition and execution operator for embodied
systems mimics a contract-based societal organization. The obvious question is, if and how
these “smart contracts” may be integrated into existing judicial systems or variants thereof.
These considerations point to a multitude of serious systems programming challenges. For
instance, how do we specify smart contracts? What is the right framework for negotiating
146
Vazdanan et al., Responsibility Research for Trustworthy Autonomous Systems, Autonomous Agents and Multiagent Systems, 2021.
EI Systems Challenges
43
contracts? How can we verify smart contracts? How can we provide evidence of the conclusion
or breach of contract? How to incentivize/penalize embodied actors as to ensure beneficial
behavior? It is also open to discussion if such federations should be deployed in social contexts
without an orchestrating higher instance.
ANALYSIS
Analysis is the process of assuring that a system meets a set of given requirements. The
verification challenge therefore involves identifying what kind of properties are expected of
embodied systems and how to establish them. Analysis of embodied systems is challenging,
among others, for their openness, adaptivity, situatedness, and for their largely non-predictable
behavior in uncertain operating contexts. In addition to functional correctness, performance,
dependability, and safety requirements as in classical embedded system, the analysis of
embodied systems also focuses on establishing properties on their lawful, ethical, and robust,
that is trustworthy, behavior. The verification of learning-enabled components of embodied
systems poses yet another challenge. The verification of learning-enabled components of
embodied systems poses yet another challenge.
Consider, for example, the embodied retail system in Section 2, and assume that, in reaction to
the customer’s request a dynamic federation of production and logistics services have been set
up for cooperatively serving this request, then analysis might establish that this federation
actually is able to fulfill the request within the agreed time frame, that the delivered product is
according to functional, quality, and safety agreements. The federation might also be shown to
be robust to common faults, such as breakdowns of logistics chains, and even malicious attacks.
Moreover, analysis may also be used to demonstrate that the federation complies with
applicable laws (for example, tax laws) and that certain social values such as climate neutrality
are adhered to.
State-of-the-practice techniques for safety analysis require deterministic behavior in well-
defined operating contexts, and they usually rely on fallback mechanisms to a human operator.
Clearly, these prerequisites are not fulfilled for embodied systems, and consequently current
safety analysis methodology, as encoded in industrial standards such as DO 178C in aerospace
or ISO 26262 in the automotive domain are not applicable – at least not directly so.
There are three main analysis techniques, namely testing, symbolic verification, and runtime
verification. We briefly describe some of associated challenges when applied to analyzing
embodied systems with learning-enabled components.
TESTING. This is the most widely used and, arguably, also the most successful technique for
analyzing software-intensive systems. Non-deterministic systems, however, are usually
considered to be untestable because of the overwhelming number of cases to be considered.
System tests are also performed with the assumption of fixed and well-described operating
contexts. Embodied systems, however, need to be analyzed with respect to uncertain operating
EI Systems Challenges
44
contexts, which may not even be known at design time. Finally, the analysis for learning-
enabled components requires establishing properties for all possible evolutions of such a
component. For all these reasons, testing methodologies as developed for embedded systems
are not directly applicable to embodied systems. Novel approaches to testing embodied systems
are urgently needed. For example, scenario-based testing dynamically classifies relevant
scenarios by means of automated clustering, and it generates a sufficient set of test cases from
the classes thus obtained.
147
More generally, probabilistic programs might be synthesized for
capturing relevant scenarios,
148
since probabilistic programs assign distributions to features of
scenarios, and they impose hard and soft constraints over scenarios.
Testing is usually decomposed into testing individual components of a system followed by
testing the integrated system. But then: how can we test systems with learning-enabled
components? For artificial neural networks (ANNs), traditional structural coverage criteria
from software testing can usually not be applied directly to ANN. For example, neuron
coverage is trivially fulfilled for an ANN by a single test case. Moreover, branch coverage,
when applied to ANNs, may lead to an exponential (in the number of neurons) number of
branches to be investigated, and are therefore not practical as typical ANNs are comprised of
millions of neurons. As usual in testing, the balance between the ability to find bugs and the
computational cost of test case generation is essential for the effectiveness of a test method.
149
Therefore, ANN-specific non-structural test coverage criterion for the robustness,
interpretability, completeness, and correctness of an ANN have been developed.
150
A scenario
coverage metric, for example, partitions the possible input space according to N attributes (e.g.
snow, rainy, …), and proposes, based on existing work on combinatorial testing, efficient k-
projection (for k = 0,…,N-1) coverage metrics as approximations of the exponential number of
input partitions.
The generation of falsifying/adversarial test cases is generally using search heuristics based on
gradient descent or evolutionary algorithms.
151
152
153
154
These approaches may be able to find
falsifying examples efficiently, but they usually do not provide an explicit level of confidence
about the nonexistence of adversarial examples in case the algorithm fails to find one.
Various traditional techniques for test case generation such as fuzzing, symbolic execution,
concolic testing, mutation testing, and metamorphic testing have been extended to ANNs.
147
Hauer, On Scenario-Based Testing of Automated and Autonomous Driving Systems, TUM, 2021.
148
Fremont et al. Scenic: Language-Based Scene Generation, UCB/EECS-2018-8, 2019.
149
Sun, Huang et al, Testing Deep Neural Networks, arXiv:1803.04792v4, 2019.
150
Cheng, Nührenberg, Rueß, Yasuaoka, Towards dependability metrics for neural networks, MEMOCODE, 2018, IEEE.
151
Goodfellow, Shlens, & Szegedy, Explaining and harnessing adversarial examples. arXiv:1412.6572, 2014.
152
Papernot, McDaniel et al., The limitations of deep learning in adversarial settings, Security & Privacy, 2016, IEEE.
153
Carlini, Wagner, Towards evaluating the robustness of neural networks, Security&Privacy, 2017, IEEE.
EI Systems Challenges
45
Despite their effectiveness in discovering various defects of ANNs together with their data-
centric requirement specifications, however, it is not exactly clear how testing-based
approaches can be efficiently integrated into the construction of convincing safety
argumentations for learning-enabled components, let alone embodied systems.
Altogether, testing methods seem to be effective at discovering defects of learning-enabled
components such as ANNs. It is unclear, however, how to measure the effectiveness of test
coverage metrics in building up sufficient confidence, or, dually, raising doubts. Also, most
testing-based approaches assume a fixed ANN. However, ANNs are learning-enabled and
trained continuously on new data/scenarios. The challenge is to come up with methodologies
for efficiently - depending on the application context also in real-time - retesting safety
requirements for continuously evolving ANNs.
SYMBOLIC ANALYSIS. These analysis techniques generalize testing in that sets of test cases are
evaluated on a system at once. These test sets are usually encoded as logical constraints for
describing, possibly infinite, test sets. Symbolic analysis neither requires a complete system
implementation nor a fully specified operational context, since unknown behavior may be
represented logically by means of uninterpreted functions. Logical constraints on these
uninterpreted function are used for expressing known (or learned) facts about these behaviors.
In contrast to testing, symbolic analysis therefore may be applied to demonstrating that certain
requirements hold for embodied systems operating in uncertain and only partially known
operating contexts. Another use of symbolic analysis is to support the generation of safe
trajectories during runtime.
155
Recently, many different symbolic analysis techniques have been adapted to learning-enabled
components such as ANNs.
156
In particular, verification problems for ANNs have been reduced
to constraint solving problems such as satisfiability in propositional logic,
157
158
satisfiability
modulo theories,
159
160
161
162163
and mixed-integer linear programming.
164
These approaches,
however, typically do not scale up to the size of real-world ANNs with millions of neurons.
Approximation techniques are applied to improve efficiency, but usually at the expense of
155
Althoff, Dolan, Online verification of automated road vehicles using reachability analysis, Transactions on Robotics, 2014, IEEE.
156
Huang et al., A Survey of Safety and Trustworthiness of Deep Neural Networks: Verification, Testing, Adversarial Attack and Defence, and
Interpretability, Computer Science Review, 2020.
157
Cheng, Nührenberg, Rueß, Verification of binarized neural networks, VSTTE, 2018.
158
Narodytska, Verifying properties of binarized deep neural networks, AAAI, 2018.
159
Huang, Kwiatkowska, Safety verification of deep neural networks, CAV, 2017
160
Pulina, An Abstraction-Refinement approach to verification of artificial neural networks, CAV, 2010.
161
Katz, Barrett e al., Reluplex: An efficient SMT solver for verifying deep neural networks, CAV, 2017.
162
Tuncali, Ito, Kapinski, Deshmukh, Reasoning about safety of learning-enabled components in autonomous cyber-physical systems, IEEE
DAC, 2018.
164
Cheng, Nührenberg, Rueß, Maximal Resilience of Artificial Neural Networks, ATVA, 2017.
EI Systems Challenges
46
precision. Recent approaches based on global optimization have the potential of dealing with
larger networks.
165
Compositional verification techniques for scaling up symbolic analysis are
largely missing. Since symbolic analysis technologies work on abstract models, they might
miss certain defects due to implementation issues (for example, rational numbers vs. IEEE
floating points). It is also unclear how to efficiently apply these techniques to continuously
evolving learning-enabled components.
RUNTIME VERIFICATION AND RECOVERY. In runtime verification a monitor observes the
concrete execution of the system in question and checks for violations of stipulated properties.
When the monitor detects a violation of a property, it notifies a command module which then
isolates the cause of the violation, followed by an attempt to recover from the violation. In this
way, runtime verification is a central element of FDIR-based
166
fault-tolerant systems. For the
multitude of sources for uncertainty in AI systems, stringent real-time requirements, and ever-
changing learning-enabled components, runtime verification is an essential element for
analyzing embodied systems.
Architectural design principles for monitoring distributed systems are needed to ensure that
monitoring does not perturb the system (at least, not too much).
167
In particular, the tutorial
discusses challenges on instrumenting real-time systems so that timing constraints of the system
are respected.
168
A recent tutorial describes state-of-the-practice technology for generating
runtime monitors that capture the safe operational environment of systems with AI/ML
components.
169
Altogether, runtime verification is an essential and attractive technique of any verification
strategy for embodied systems. Unlike static verification techniques such as testing or symbolic
analysis, there is no need for adaptation to learning-based components. In this way, runtime
monitoring is an enabling verification technology for continuous assurance, based on the
MAPE-K
170
loop from autonomic computing. The main challenge in deploying runtime
monitoring, as is the case for any other cyber-physical system, is to embed monitors in an
efficient (for example, energy-efficient) way, without perturbing the behavior of the embodied
system too much.
Runtime monitoring may also be used for measuring uncertainties in input-output behavior of
learning-enabled components. For example, if an input is out-of-distribution of the training set,
then one may conclude that the output may not actually be a “correct” one. Such information
about the uncertainty of a perception result is useful input for deliberatively planning
165
Ruan, Wu, Sun, Huang, Reachability analysis of deep neural networks with provable guarantess, IJCAI, 2018.
166
Fault Detection, Isolation, and Recovery
167
Goodloe, Pike, Monitoring distributed real-time systems: a survey and future directions, NASA, 2010.
168
Bonakdarpour, Runtime-Monitoring of Time-Sensitive Systems, Runtime Verification, 2011.
169
https://uva-mcps-lab.github.io/RV21/paper10.1.html
170
Measure, Analyze, Plan, Execute; the K stands for Knowledge
EI Systems Challenges
47
meaningful and safe actions. Uncertainty information about the perception unit is also used in
Simplex architectures for switching to a safe(r) perception channel whenever the ANN output
is doubtful. Clearly, the distance (in some given metric) of the input to the set of training input
may serve as a measure of uncertainty of the input-output behavior of the learning-enabled
component. Notice, however, that such a measure returns uncertainty zero even for “incorrect”
behavior on training inputs. Alternatively, it has been proposed to monitor the neuronal
activation pattern in ANN-based components, and to compare it with neuronal activation
patterns as learned during the ANN training phase.
171
In addition, applicable background
knowledge and physical laws may also be used in monitoring the plausibility of the input-output
behavior of an ANN.
In summary, due to the multitude of sources of uncertainty of embodied systems with learning-
enabled components and the partially unknown environments in which they operate, even if all
the challenges for specification and verification are solved, it is likely that one will not be able
to prove unconditional safe and correct operation. There will always be situations in which we
do not have a provable guarantee of correctness. Therefore, techniques for achieving fault
tolerance and error resilience at run time must play a crucial role. There is however not yet a
systematic understanding of what kind of analysis cand be achieved at design time, how the
design process can contribute to safe and correct operation of the embodied system at run time,
and how the design-time and run-time analysis techniques can interoperate effectively.
The distributed and dynamic nature of federations of embodied actors and their goals is
particularly challenging for runtime verification. A runtime monitoring framework for
embedded systems must support reasoning under uncertainty,
172
173
and also partially
observable systems with nondeterministic and probabilistic dynamics.
174
VerifAI is a runtime
monitoring framework for autonomous systems with learning-enabled components.
175
It
includes formal modeling of the autonomous system and its environment (in terms of
probabilistic programs), automatic falsification of system-level specifications as well as other
simulation-based verification and testing methods, automated diagnosis of errors, and
automatic specification-driven parameter and component synthesis. Safety of systems with
learning-enabled components in Simplex architectures
176
often relies on a runtime monitor-
based switch between a performant and a safe channel. Runtime monitoring of typical security
171
Cheng, Nührenberg, Yasuoka, Runtime monitoring neuron activation patterns, IEEE DATE, 2019.
172
Zheng, Julien, Verification and validation in cyber physical systems: Research challenges and a way forward. IEEE Software Engineering
for Smart Cyber-Physical Systems, 2015.
173
Ma, Meiyi, et al., Predictive monitoring with logic-calibrated uncertainty for cyber-physical systems, TECS, 2021.
174
Viswanadha, Kim et al., Parallel and Multi-Objective Falsification with Scenic and VerifAI. Runtime Verification, 2021.
175
Torfah, Junges, Fremont, Seshia, Formal Analysis of AI-Based Autonomy: From Modeling to Runtime Assurance, Runtime Verification,
2021.
176
Desai, Ghosh, Seshia, Shankar, Tiwari, SOTER: a runtime assurance framework for programming safe robotics systems, IEEE DSN,
2019.
EI Systems Challenges
48
hyperproperties,
177
178
privacy policies,
179
and contextual integrity
180
have also been
considered. Given a set of properties and an embodied system the challenge is to generate and
maintain sound and possibly complete runtime monitors, which are woven into the embodied
system, as to keep interferences with the core behavior at a minimum.
ASSURANCE
A trustworthy embodied system operating in our very economic and social fabric is required to
obey applicable regulation and laws, to act according to human-centered social values, and to
be robust and safe (see Section 4). But how can we be assured that an embodied system indeed
is worth of the trust we may put in it? Clearly analysis (see Subsection
Analysis) of embodied systems is a central component of any mechanism for inspiring
confidence into the lawful, ethical, and robust behavior of any technical system, and for
building up trust. In addition, as an actor in a real-life context, an embodied system is also
required to provide explicit evidence that indeed it is acting as required.
Requirements. Embodied systems have the possibility of autonomously acting in regulated
sectors such as healthcare, finance, insurance, accounting, or retail. As such they need to
comply with applicable regulations and national laws; for example, the General Data Protection
Rights (GDPR), the Health Insurance Portability and Accountability Act (HIPAA), the Gramm-
Leach-Bliley Act (GLBA), the Sarbanes-Oxley Act (SOA), or the Children’s Online Privacy
Protection Act (COPPA). There is indeed no lack of these kinds of regulations and laws. For
example, German law now obliges organizations to demonstrably ensure that basic labor and
human rights standards are respected throughout global supply chains.
181
The length of these
regulations and laws, the opacity of the legal language, the complexity of these acts, and their
contradictions
182
however make it very difficult to determine and demonstrate compliance.
183
Regulations have been formalized using a variety of policy languages.
184
185
186
Formalisms
and systems for expressing and checking privacy policy include P3P,
187
EPAL,
188
and
177
Finkbeiner, Hahn, Stenger, Tentrup, Monitoring hyperproperties, Formal methods in system design, 2019.
178
Bonakdarpour, Sanchez, Schneider, Monitoring hyperproperties by combining static analysis and runtime verification, Leveraging
Applications of Formal Methods, 2018.
179
Chowdhury, Jia, Garg, Datta, Temporal mode-checking for runtime monitoring of privacy policies. CAV 2014.
180
Barth, Datta, Mitchell, Nissenbaum, Privacy and contextual integrity: Framework and applications, IEEE Security&Privacy, 2006.
181
https://www.bmz.de/de/entwicklungspolitik/lieferkettengesetz
182
Lam, Mitchell, Sundaram, A formalization of HIPAA for a medical messaging system, Trust, Privacy and Security in Digital Business,
2009, Springer.
183
Ness, A year is a terrible thing to waste: early experience with HIPAA, Annals of Epidemiology, 2005.
184
Anton, Earp, Reese. Analyzing website privacy requirements using a privacy goal taxonomy. Requirements Engineering 2002.
185
Anton, Eart, Vail, Jain, Gheen, Frink, HIPAA’s effect on web site privacy policies. Security & Privacy, 2007, IEEE.
186
Anton, He, Baumer. Inside JetBlue’s privacy policy violations, Security & Privacy, 2004, IEEE.
187
Cranor, P3P: Making privacy policies more useful, Security & Privacy, 2003. IEEE.
188
Ashley, Hada, Karjoth, Powers, Schunter, Enterprise privacy authorization language (EPAL), IBM Research, 2003.
EI Systems Challenges
49
XACML,
189
DKAL,
190
and the Logic of Privacy and Utility, which is a formalization of the
concept of contextual integrity.
191
In addition to compliance with applicable laws and norms, embodied systems are required to
be demonstrably robust, safe, and secure. More precisely, embodied systems are expected
§ to be resilient to common and possibly also new kinds of breakdowns and malicious
attacks,
§ the risk of unintended harm to humans, machinery and the environment is demonstrably
below acceptable levels, and
§ identified confidentiality, integrity, and availability requirements are satisfied.
Other requirements include transparency (for example, traceable use of data and information
sources, published decision policies), demonstrable fairness of decisions (for example, all
applicants are demonstrably being served equal), inverse privacy,
192
or contextual integrity.
193
194
The latter requirement is based on the hypothesis that privacy is a right to appropriate flow
of information. In case of non-compliance to any of these requirements, culprits need to be
identified and called to account.
The challenge is to come up with adequate formalizations of adequate subsets of applicable
laws and norms as a prerequisite for demonstrable compliance to these requirements. Previous
attempts in this direction have often been based on some variant of temporal logic and/or logic
programming.
We now describe possible approaches and corresponding challenges for assuring compliance
to these kinds of requirements. The challenge for a federation of embodied actors is to assure,
and possibly demonstrably so, that it is indeed worth of our trust (Section Trustworthiness). For
example, the largely autonomously acting federation of embodied retail actors (Section Service
federations) is required to answer questions such as: is it able to deliver the requested product
in time and according to order specs? Is it resilient to common planning breakdowns, failures,
and malicious attacks? Are there contingency plans and never-give-up strategies in case of, say,
operational, judicial, and financial quagmires? Are the actors of the federation acting safely?
189
Ramli, Nielson, Nielson, The logic of XACML, Science of Computer Programming, 2014.
190
Gurevich, Neeman, DKAL 2—A simplified and improved authorization language, Technical Report MSR-TR-2009–11, 2009.
191
Barth, Datta, Mitchell, Nissenbaum, Privacy and contextual integrity: Framework and applications, Security and privacy, 2006, IEEE.
192
Gurevich, Hudis, Wing, Viewpoint: Inverse privacy, CACM Volume 59, Number 7, 2016.
193
Nissenbaum, Privacy in Context: Technology, Policy, and the Integrity of Social Life, Stanford University Press, 2010.
194
Datta, Blocki, Christin, DeYoung, Garg, Jia, Sinha, Understanding and protecting privacy: Formal semantics and principled audit
mechanisms, Information Systems Security, 2011. Springer.
EI Systems Challenges
50
Does the federation obey applicable tax laws? Can the federation demonstrate that the delivered
product is not based on child labor? Does the federation demonstrably act in a climate neutral
or even climate positive manner? This list of questions to be answered is necessarily incomplete
as there is indeed no lack of requirements when acting in real-world social contexts.
Analysis-based assurance. Static analysis techniques such as testing ensure the system’s
compliance to requirements, but only at a selected points of time in their evolution. Even if one
succeeds in testing the compliance of an embodied system to its requirements, these results
usually are outdated in further evolutions of the self-learning embodied system operating in a
dynamic context, thereby requiring retesting. In a similar vein, traditional formal verification
techniques may be applied to demonstrate, a priori during design time, conformance
requirements of an embodied system operating in largely unknown environments and all its
possible evolutions. Hereby, proof objects serve as evidence,
195
and reflection principles are
used to generate specific evidence for each possible evolution.
196
But it seems unrealistic that,
still largely manual, formal verification is applied to the analysis of non-trivial evolving
embodied systems. By-and-large we therefore need to go beyond current static analysis
techniques such as testing and formal verification to realistically assure compliance of ever-
evolving embodied systems with their complex real-world requirements.
Process-based assurance. Assurance has traditionally been approached in software and systems
engineering through design processes following rigorous development standards, and by
demonstrating compliance through analysis. Well-proven and successful assurance standards
such as DO 178C in aerospace or ISO 26262 in the automotive domain are predicated on the
assumptions that system behavior is deterministic, there is a clearly defined operating
environment, there is always a fallback to a human-in-the-loop operator, and, once the system
is deployed, it does not learn and evolve. These basic assumptions of traditional safety
engineering do not hold anymore for embodied systems. Safety engineering therefore comes to
a grand climacteric moving from deterministic, non-evolving embedded and cyber-physical
systems operating in well-defined contexts to embodied systems. Process-based design for
many embodied systems seems to be difficult, if not impossible, as these systems continuously
integrate into dynamic coalitions, they are increasingly autonomous, and they are evolving
based on previous experience. But changes in process-based assurance is notoriously slow and
expensive.
197
Therefore, assurance for embodied systems increasingly needs to rely on
assurance artifacts collected at runtime instead on process-based design-time assurance.
Continuous assurance. It has been recommended to continuously audit trustworthiness of an
embodied system throughout its lifecycle ( Table 1) to assure compliance to applicable laws,
social norms, and technical robustness and safety requirements. This is easier said than done,
as embodied systems
195
Necula, Proof-carrying code, POPL, 1997.
196
Rueß, Meta-Programming in the Calculus of Construction, U Ulm, 1995.
197
Changes of only one line of code in safety-cricital code in the aerospace industry have reportedly cost seven-figure sums (in US Dollars)
in some cases.
EI Systems Challenges
51
§ are dynamic federations based on changing priorities and intent,
§ are composed of ever-evolving embodied systems with self-learning capabilities,
§ operate in largely unknown contexts based on experience and targeted exploration, and
§ applicable laws and regulations may change over time and are also location dependent.
The challenge therefore is continuous assurance, an assurance system provided provisionally
at design time, and continually monitored, updated, and evaluated at run-time as the system and
its environment evolves. But continuous assurance for regulations such as GDPR has become
a costly burden.
198
199
The culprit is the heavy reliance on manual, checklist-based auditing in
today’s compliance process, which is expensive, slow, and error-prone. It is also restricted to
the compliance of static snapshots in the evolution of a system. Therefore, runtime analysis and
assurance are all-important for continuously assuring compliance of ever-evolving embodied
systems.
Runtime assurance is a dynamic analysis technique based on a log for recording system events,
which are relevant for establishing conformance with given requirements. These requirements
are usually expressed in a variant of temporal logic, which is expressive enough for encoding
regulation rules.
200
Now runtime verification techniques are used for establishing that the
logged event trace indeed conforms with the given requirement. For example, an embodied
actor logs events of its data handling operation, and runtime verification is used to establish
compliance with a given privacy policy or detecting violations thereof. In addition, the root
cause of such a violation may be computed from a log as the basis for calling the responsible
embodied actors to account for the conformance violations they caused.
It is open to discussion if the identified embodied culprits can and should be treated as legal
entities, which can be held responsible for their actions in a traditional brick-and-mortar judicial
system. Nevertheless, a workable judicial governance of one sort or the other is needed for
promoting, by-and-large, compliant behavior - as the basic mechanism for a self-improving
system behavior.
Depending on the regulatory framework, compliance to these requirements by every actor
might lead to functionally inferior behavior or may even be a barrier to innovation. For example,
regulatory frameworks may well be contradictory, there are ongoing considerations of the
relative importance of compliance with a wide range of different requirements, possibly from
198
According to Forbes ,GDPR cost US Fortune 500 companies $7.8 Billion as of 2018. (Forbes, The GDPR racket: Who’s making money
from this $9bn business shakedown, 2020. https://www.forbes.com/sites/oliversmith/2018/05/02/the-gdpr-racket-whos-making-money-from-
this-9bn-business-shakedown/#54c0702034a2)
199
A recent report shows that 74% of small- or mid-sized organizations spent more than 100 k$to prepare for continuous compliance with
GDPR and CCPA. (DataGrail, The age of privacy: The cost of continuous compliance, 2020, https://datagrail.io/downloads/GDPR-CCPA-
cost-report.pdf)
200
DeYoung, Garg, Dia, Kaynar, Datta, Experiences in the logical specification of the HIPAA and GLBA privacy laws. WPES 2010, ACM.
EI Systems Challenges
52
different sources, and, finally, without minor transgressions of, say, traffic rules (such as
crossing solid lines) traffic flow would often be less fluid.
At a high-level, runtime assurance can be approached by compliance checks on recorded logs
when demanded, say, by an audit authority,
201
by online checking of relevant events against
the prevailing policy,
202
203
204
205
206
207
208
209
or by combinations thereof.
210
In federations
of embodied actors, system logs do not only need to contain all the relevant event in the right
order, but they also need to be tamper-proof and they need to comply with applicable privacy
policies. Distributed ledger technology such as various incarnations of blockchains have lately
been tried, for instance, for tamper-proof logging. It is also challenging to monitor information
flow policies such as non-interference or observational determinism, which relate multiple
computation traces with each other.
211
212
Assurance Cases. A fundamentally different approach to assurance is based on constructing
and maintaining safety and assurance cases,
213
which are compelling, comprehensive,
defensible, and valid justification of the compliance of a system. It is based on a structured
argument of assurance considerations, across the system lifecycle, that can assist in convincing
the various stakeholders that the system is acceptably safe. The purpose is, broadly, to
demonstrate that the safety-related risks associated with specific system concerns
214
have been
identified, are well-understood, and have been appropriately mitigated, and that there are
mechanisms in place to monitor the effectiveness of safety-related mitigations. In this sense,
an assurance case is a structured argument for linking safety-related claims through a chain of
arguments to a body of the appropriate evidence.
201
Garg, Jia, Datta, Policy auditing over incomplete logs: Theory, implementation and applications. CCS, 2011.
202
Basin, Klaedtke, Müller. Monitoring security policies with metric first-order temporal logic. SACMAT, 2010, ACM.
203
Basin, Klaedtke, Marinovic, Zalinescu, Monitoring compliance policies over incomplete and disagreeing logs, RV, 2012, Springer.
204
Basin, Klaedtke, Marinovic, Zalinescu, Monitoring of temporal first-order properties with aggregations, RV, 2013, Springer.
205
Chomicki, Efficient checking of temporal integrity constraints using bounded history encoding ACM Trans. Database Syst., 1995.
206
Chomicki, Niwinski, On the feasibility of checking temporal integrity constraints, PODS 1993, ACM.
207
Krukow, Nielsen, Sassone, A logical framework for history-based access control and reputation systems, J. Comput. Secur., 2008.
208
Bauer, Gore, Tiu, A first-order policy language for history-based transaction monitoring, ICTAC 2009.
209
Ozeer, φ comp: An Architecture for Monitoring and Enforcing Security Compliance in Sensitive Health Data Environment, ICSA-C,
2021, ACM.
210
Chowdhury, Jia, Garg, Datta, Temporal mode-checking for runtime monitoring of privacy policies, CAV, 2014, Springer.
211
Finkbeiner, Hahn, Stenger, Tentrup, Monitoring hyperproperties, RV, 2017, Springer.
212
Bonakdarpour, Finkbeiner, The complexity of monitoring hyperproperties, CSF, 2018, IEEE.
213
UK Ministry of Defence, 2007.
214
Including safety and security, but also applies to all the other attributes of trustworthiness.
EI Systems Challenges
53
One of the main benefits for structured arguments in assurance cases is to explicitly capture the
causal dependencies between assurance claims and the substantiating evidence, as obtained, for
instance, by analysis. Assurance cases also determine the level of scrutiny needed for
developing and operating systems which are acceptably safe. This kind of information from
safety cases might in the future also be used by an embodied actor to safely explore and navigate
in largely unknown operating contexts.
On the other hand, assurance cases currently are at most semi-formal and their construction and
maintenance requires significant manual efforts. Adequate notions of assurance case patterns
and modular composition operators for assurance cases, however, are opening up new
possibilities for aligning compliance arguments with system evolvements during runtime (such
as the dynamic formation of federations).
A major challenge is to measure confidence in assurance cases.
215
216
217
218
219
Eliminative
induction, for instance, increases confidence in assurance cases by removing sources of doubt
and using Baconian
220
probability to represent confidence.
221
One systematic approach is
through construction and dialectical consideration of counterclaims and countercases, where
counterclaims are natural in confirmation measures as studied in Bayesian confirmation theory,
and countercases are assurance cases for negated claims.
Assurance cases are successfully applied to traditional safety-critical systems with clearly
defined operating contexts, safety requirements, and fallback strategies to human operators.
Given the increasing complexities and sources of uncertainty, however, the current assurance
approach with prescribed and fixed verification and validation process activities, criteria, and
metrics does not work well for assuring AI-based or even embodied systems (Alves, et al.,
2018). Due to the multitude of sources of uncertainty, assurance arguments for increasingly
autonomous embodied systems, need to (1) stress rigor in the assessment of the evidence and
reasoning employed, and (2) automate the search for defeaters, the construction of cases and
countercases, and the management and representation of dialectical examination.
222
It is of
particular interest to capture how the influence of a learning-enabled component is captured
and reasoned within the control structure of an embodied actor. Rigorous assurance cases can
be adapted, faster and more efficiently, to ever-evolving assurance needs of embodied systems.
215
Grigorova, Maibaum, Taking a page from the law books: Considering evidence weight in evaluating assurance case confidence, Software
Reliability Engineering Workshops. 2013, IEEE.
216
Duan, Rayadurgam, Heimdahl, Sokolsky, Lee, Representing confidence in assurance case evidence, Computer Safety, Reliability, and
Security, 2014, Springer.
217
Bloomfield, Littlewood, Wright, Confidence: its role in dependability cases for risk assessment, Dependable Systems and Networks
(DSN), 2007, IEEE.
218
Rushby, Formalism in safety cases, Making Systems Safer, 2010, Springer.
219
Rushby, Logic and epistemology in safety cases, Computer Safety, Reliability, and Security, 2013, Springer.
220
https://ntrs.nasa.gov/api/citations/20160013333/downloads/20160013333.pdf
221
Goodenough, Weinstock, Toward a theory of assurance case confidence, CMU Technical Report, 2004.
222
Bloomfield, Rushby, Assurance 2.0: A Manifesto, arXiv:2004.10474, 2020.
EI Systems Challenges
54
Rigorous assurance cases open new possibilities for dependable and safe exploration in largely
unknown operating contexts, which have been obtained from relevant information of a safety
case and its certainty assessment. For example, if there is only weak evidence on the fact that
the traffic light in front of the ego cart is green, then the ego car might want to increase her
assurance by strengthen this case, for example, by means of additional sensor activity.
Therefore, rigorous assurance cases can be instrumental in online behavioral self-adaptation
and for determining safe behavior when operating in uncertain contexts.
Evidence-based Assurance is based on explicitly generating verifiable evidence for any
transaction of and between embodied actors. The underlying mindset is “verify, then trust”.
Consider, for example, a responsible visa-granting actor that returns evidence (think of it as a
number) corresponding to the permission for the requester to enter the foreign country for a
specific time frame. The border control, upon entry, verifies the identity of the presenter and
verifies that the presented visa is valid for this person to enter the country now. This basic
mechanism is applicable to all kinds of transactions which involve the exchange of physical
evidence in the form of identity cards, driver’s licenses, money, checks, visas, airline tickets,
traffic tickets, birth certificates, vaccination certificates, and stock certificates, as well as
electronic evidence including PIN numbers, passwords, keys, certificates, digital coins, and
nonces.
It is easy to extrapolate from the above scenario to other uses of digital evidence in electronic
commerce, business and administrative processes, and digital government. where actors
execute smart contracts (as outlined in Subsection Self-Integration) to carry out specific tasks
that require the exchange of authorization and authentication information. In this way, evidence
is produced, say, for asserting the conclusion of contracts. The generated evidence then can be
independently and automatically checked, for example, as the prerequisite for directing
corresponding payments.
Previously we have been defining such a framework for evidential transactions, called
Cyberlogic, which is based on a public key infrastructure.
223
224
The basic ingredients are
extremely simple. First, evidence is encoded by means of numbers using digital certificates
and nonces. Second, predicates are signed by private keys so that a decryption of such a
certificate with the corresponding public key is a proof or evidence for the assertion contained
in the certificate. Third, protocols are distributed logic programs that gather evidence by using
both ordinary predicates, digital certificates, and proof construction based on the Curry-Howard
isomorphism. These simple building blocks are sufficient to encode all-important capabilities
for delegation, retraction of permission, and time-stamped evidence. For example, some
vaccination certificate might be handed out at a certain point of time. Now, new medical
223
Rueß, Shankar, Introducing Cyberlogic, High Confidence Software and Systems Conference, Baltimore, 2003.
224
Nigam, Reis, Rahmouni, S, Rueß, Proof Search and Certificates for Evidential Transactions, Automated Deduction (CADE), 2021.
EI Systems Challenges
55
evidence on the ephemeral efficacy of vaccination results in the need for retracting
corresponding capabilities.
Evidential transaction in a Cyberlogic-like setting have recently been used to demonstrate that
accountability for federated machine learning becomes paramount to fully overcoming
legislative and jurisdictional constraints.
225
In particular, it ensures that all entities' data are
adequately included in the model and that evidence on fairness and reproducibility is curated
towards trustworthiness. or realizing accountability for federated machine learning. Cyberlogic
also forms the foundations for secure and trusted-by-design smart contracts.
226
Even though
Cyberlogic has proven to be instrumental in many cases, it is still unclear if proof-carrying
capabilities can be scaled up for complex real-world systems. Possible directions for addressing
these scalability issues are interactive proof systems and the use of advanced cryptographic
mechanisms.
Altogether, we have described challenges and possible approaches for the