Conference PaperPDF Available

Situation Awareness in Autonomous Service Robots


Abstract and Figures

Despite the increasing demand for service robots, considerable challenges re-main to be solved before these robots can be ubiquitously employed in personal households. To this date, autonomously operating robots lack the flexibility to react appropriately in unforeseen situations. At the Human Factors Institute of the Universität der Bundeswehr München, a multi-disciplinary research group of psychologists, engineers and computer scientists aims to ascertain to which ex-tent cognitive models of situation awareness may enhance a robot's ability to display adaptive behaviour in unforeseen situations.
Content may be subject to copyright.
Situation Awareness in Autonomous Service Robots
Verena Nitsch1
Keywords: Cognitive Modeling, Situation Awareness, Autonomous Robots
Despite the increasing demand for service robots, considerable challenges re-
main to be solved before these robots can be ubiquitously employed in personal
households. To this date, autonomously operating robots lack the flexibility to
react appropriately in unforeseen situations. At the Human Factors Institute of
the Universität der Bundeswehr München, a multi-disciplinary research group of
psychologists, engineers and computer scientists aims to ascertain to which ex-
tent cognitive models of situation awareness may enhance a robot’s ability to
display adaptive behaviour in unforeseen situations.
In 2011, about 2.5 million robots were sold for personal and domestic use
(International Federation of Robotics, 2012). The International Federation of
Robotics projects sales of over 15 million units for the period 2012 to 2015 with
experts indicating that service robots have an innovation and market potential
similar to that currently presented by industrial robots (Decker, et al., 2011). To
this date, autonomously operating robots lack the flexibility to react appropriate-
ly in unforeseen situations as important perceptual and decision-making struc-
tures are still designed by programmers before the robot begins its task. This
leads to fragmented abilities and brittle performance, in particular in unstruc-
tured and dynamic environments, which are prevalent in domestic contexts
(Benjamin, Monaco, Lin, Funk, & Lyons, 2012). Hence, perhaps the greatest
challenge in the engineering of autonomously operating robotic systems remains
the instillation of flexibility in the robot’s decision-making processes that would
allow it to adapt to the demands presented by a dynamically changing environ-
ment at any given moment, whilst still pursuing a particular operative goal (as
opposed to purely reflexive systems). This line of research can be subsumed un-
der the label of cognitive robotics.
In an effort to tackle this problem of failing adaptability in robots, several ap-
proaches have surfaced in the different scientific disciplines that are currently
active in the field of cognitive robotics. One school of thought, which derived
from the computer sciences, focuses primarily on the effective harnessing of
computational processing capabilities and the development of mathematical al-
gorithms to create learning and hence adaptable systems. Within the paradigm of
1 Universität der Bundeswehr München, Institut für Arbeitswissenschaft, Werner-Heisenberg-Weg 39, 85577
Neubiberg; Email:
machine learning, numerous attempts were made to develop heuristics based on
statistical properties of sensor signals and reinforcement learning in an effort to
reduce processing time and computational power requirements that would lead
to slow response times and likely ineffective behaviour in dynamically changing
situations (e.g. Bagnell, Bradley, Silver, Sofman, & Stentz, 2010; Peters,
Morimoto, & Tedrake, 2009; Bratko, 2010). Although considerable progress has
been made in this domain in recent years, real-time learning and in particular
adaptive behavior of robots in unforeseen situations remain a great challenge in
robotics (Benjamin, Monaco, Lin, Funk, & Lyons, 2012). A different approach
may be sought to complement these efforts. Cognitive scientists have tackled the
issue of system adaptability by investigating human cognitive abilities that lead
to adaptive behavior in such situations. Based on decades of experimental psy-
chological work, numerous models have been proposed in this domain that spec-
ify human cognitive processes in dynamic situations (e.g. Baumann & Krems,
2009; Gonzales & Dutt, 2007; Durso & Sethumadhavan, 2008). As of yet, how-
ever, the evaluation of these cognitive models remains chiefly theoretical work;
to our knowledge, a truly cognitive architecture has not yet been implemented in
a robotic system (Benjamin, Monaco, Lin, Funk, & Lyons, 2012). Consequently,
it is not sufficiently clear to which extent relevant models of human cognitive
processes may be applicable to autonomously operating robots and improve per-
formance of such systems. Systematic investigations of the issues which would
surface in the implementation of such models in robots, are hence urgently re-
Situation awareness as a pivotal cognitive process in dynamic en-
Cognitive researchers have suggested that the basis for humans’ decision-
making ability in dynamic situations forms their situation awareness (SA), a
psychological construct which has withstood the scrutiny of several decades of
empirical research. A commonly accepted definition of situation awareness has
been proposed by Endsley (Endsley M. , 1995, p. 36): „Situation awareness is
the perception of the elements in the environment within a volume of time and
space, the comprehension of their meaning and the projection of their status in
the near future.“ Endsley further specifies situation awareness to occur at three
consecutive stages, which are contingent upon the successful interplay of three
key cognitive components: attention, working memory (WM) and long-term
memory (LTM) (s. Figure 1).
The first step in achieving SA is the perception of status, attributes, and dy-
namics of goal-relevant stimuli in the environment. These are only perceived a)
if the physical characteristics of these stimuli capture the human’s attention (bot-
tom-up processing) and/or b) if they are focused upon and determined to be rel-
evant as indicated by operative goals that are stored in the WM and LTM (top-
down processing).
Figure 1. Schematic depiction of the cognitive processes involved in attaining situation awareness.
Adapted from (Endsley M. , 1995).
In a second step towards achieving SA, perceived stimuli are synthesized and
associated with operative goals in the WM, which have been retrieved from a
larger number of goals stored in LTM. Hereby, the WM allows one to modify
attention deployment on the basis of other perceived information or a change in
active goals (Braune & Trollip, 1982). The goal-dependent synthesis of infor-
mation stored in the WM leads to a comprehension of the current situation and
the stimuli’s relative significance in terms of achieving the goals.
In the third and final step towards achieving SA, future states of the elements
in the environment may be projected based on the comprehension of the situa-
tion, leading to a timely and effective decision. For this purpose, a situation
model which was created in the previous steps is stored in LTM and matched to
schemata in memory that depict prototypical situations or states of the system
model, which may be linked with goals that dictate further decision-making and
actions. Hence SA equips the human agent with the ability to react appropriately
in dynamic systems. In fact, numerous studies found a link between SA and ef-
fective performance (e.g. Gaba, Howard, & Small, 1995; Ma & Kaber, 2007;
Gugerty & Tirre, 1996; Golightly, Wilson, Lowe, & Sharples, 2010).
Project SAAROS: Situation Awareness in Autonomous Service
The aim of the proposed research program is to make a substantial theoretical
and practical contribution towards improving the decision-making flexibility and
hence adaptability of autonomously operating service robots using a distinctly
interdisciplinary approach. To this end, a three-pronged approach is taken. First-
ly, empirical data are gathered that reflect human processing of sensory input in
the development of situation awareness, a cognitive process that was identified
as pivotal to achieving adaptive behavior in dynamic situations. Secondly, a
computational cognitive model which reflects these mechanisms is developed,
using ACT-R, a cognitive architecture which was identified as featuring all nec-
essary properties for the simulation of situation awareness (empirically support-
ed cognitive plausibility, appropriate learning mechanisms). Thirdly, the cogni-
tive model is implemented in an autonomously operating NAO Next Gen robot
(by Aldebaran) and evaluated for several scenarios that are relevant to service
robotics and consequently adapted in order to reflect human-like adaptive be-
havior in dynamic environments. In the process of validating the developed
model, systemic as well as structural influences which impact situation aware-
ness in robotic systems are to be empirically uncovered and systematically doc-
umented. The following section details the working program of the Project
Step1: Identification of testing parameters
First, two benchmark scenarios and appropriate experimental setups within the
domain of service robotics are defined, which serve to validate a computational
SA model with experimental data collected from human participants and which
may be juxtaposed with data provided by a cognitively equipped NAO robot in
later stages of the project. Specifically, the first scenario is required for the de-
velopment and validation of the SA model in a particular task domain of service
robots (Step 6). A second scenario is devised at this stage, which will serve to
evaluate the extent to which the developed model generalizes to other task do-
mains within the field of service robotics (Step 7). For both scenarios, perfor-
mance goals are specified that determine the aimed level of functionality for the
robot once it is equipped with a validated cognitive model of SA.
Step2: Formulation of an ACT-R model of SA
A cognitively plausible ACT-R model of SA is formulated for the chosen con-
text. In order to formulate an SA model within a cognitive architecture, the in-
vestigated scenario first needs to be deconstructed to its lowest operative level.
For this purpose, the scenario is analyzed using goal-directed task analysis
(GDTA), as proposed by Endsley (Endsley M. R., 1993). GDTA is a specific
form of cognitive task analysis that focuses on identifying the goals and critical
information needs for a particular task context. In essence, the product of this
analysis is a hierarchy that specifically outlines the processes in which basic data
needed by the user are integrated into higher SA levels of comprehension and
projection. Second, based on the task analysis, an ACT-R model of SA is formu-
Step 3: Acquisition of experimental data & model validation for human data
Following the GDTA (Step2), the model validation will occur in two stages,
which are iteratively repeated until the model is considered to reflect human
cognitive processes involved in SA satisfactorily: In a first step, the model is
tested with regard to achievement and disruption of SA in humans. This includes
experimental analysis with an opportunity sample of naïve participants. To
measure the different levels of SA, a combination of tools will be used. Partici-
pants’ visual focus of attention can be tracked using eye-tracking and motion
tracking. The SA levels of comprehension and projection are assessed with two
measurement techniques that have proven reliable and valid measures in numer-
ous applications: SAGAT (Situation Awareness Global Assessment Technique)
and SPAM (Situation Present Assessment Measure). With SAGAT (Endley,
1995), participants are intermittently queried, in the middle of a dynamic simu-
lation, about the values of various parameters, whilst they are deprived of any
feedback of the scene, so that the participant must rely on working memory to
answer the questions. SPAM (Durso, Rawson, & Girotto, 2007) assesses the
speed of accessing information while the scenario continues, so that the partici-
pant could seek the needed information rather than relying on memory. The SA
measure, in this case is the time to respond. Statistical analyses will be conduct-
ed with quantitative measures in order to ascertain their degree of generalizabil-
The formulated predictions are also tested with the ACT-R model. Functional-
ly equivalent measures to those used with humans (eye-tracking, SPAM,
SAGAT) can be applied to the measurement of SA in the ACT-R model: Visual
focus of attention can be tracked and recorded during the programme’s run-time.
In addition, response times and various task performance indices and buffer con-
tent (i.e. working memory content) can be assessed. In addition, behavioural
trends can be discerned with the ACT-R simulations. Upon conclusion of the
analysis of the model’s results, the ACT-R mechanisms may need to be adjusted
and the hypotheses redefined. Hence, the testing process continues with stage (a)
until the model achieves satisfactory performance (defined as overall model fit
in performance measures of R=.7 or higher and comparable behavioural tenden-
Step 4: Development of NAO/ACT-R Interface
Development work needs to be invested in order to equip an autonomously op-
erating NAO robot, with an ACT-R model of SA. The higher level controls of
NAO are realized by an embedded PC board in the robot’s head, which runs on
a LINUX operating system. Aldebaran provides an SDK-Toolkit called NAOqi
which provides an interface to the hardware on low and high level bases, sup-
porting different programming languages, including C/C++ and Python. ACT-R,
on the other hand, is composed of a set of functions and algorithms implemented
in Common Lisp, which is not supported by NAOqi. Hence, an interface will be
devised for the robot, which enables the communication of the robot’s controls
with the perceptual-motor modules provided by ACT-R, for example, with use
of CL Python, which is an open-source implementation of Python written in
Common Lisp.
Step 5: Operationalisation of symbolic language with operative terms
With respect to the examined scenario and the results of the GDTA, operative
terms need to be defined which translate the symbolic output of ACT-R into ap-
propriate actions of the NAO robot and vice versa. This work package hence
comprises two stages: Based on the GDTA (Step 2), a database is established
which contains detailed instructions for the robot for each symbolism specified
in the ACT-R SA model. For example, if ACT-R determines during its run-time
that the robot should “scan for next visual cue”, a corresponding translation is
defined for the robot’s control and sensory processors, e.g. specifying the de-
grees of rotation of the robot’s head, the area in which it scans and a choice of
target stimuli. The second step involves the implementation of an interface link-
ing the robot’s API with an external PC on which the database is implemented.
The decision to outsource this database was made to maintain fast run-times and
at the same time ensure cognitive plausibility by considering these as subcon-
scious processes, as humans do not, for example, consciously deliberate the de-
gree to which they rotate their heads when they want to focus on an object of
Step 6: Validation of ACT-R model in the NAO robot
Once the ACT-R model of SA developed in Step 2 has been validated with re-
gard to human data and the model’s symbolic language has been translated into
operative terms for the robot, the model can be validated in the NAO robot. For
this purpose, the hypotheses that were established during the model validation
process (Step 3) are investigated with the NAO robot using comparable
measures of SA to those described in Step 3. The model is considered valid for
the presented scenario if it achieves satisfactory performance (defined as overall
model fit in performance measures of R=.7 or higher and the display of compa-
rable behavioural tendencies). If the model does not display satisfactory validity,
possible causes are identified and the model and/or physical setup may be ad-
justed accordingly.
Step 7: Assessment of the model’s ability to generalize to other task domains
In a final step, the extent to which the implemented model is task-specific is
evaluated. For this purpose, the embodied ACT-R model is tested in a different
test setup previously defined in Step 1, that requires the same basic skill set, but
different reactions than the first scenario. From this final evaluation, systemic
parameters may be delineated that can be identified as affecting SA in autono-
mously operating robots.
The multi-disciplinary project SAAROS at the Universität der Bundeswehr
München can make contributions to three different domains: the empirical re-
search on situation awareness in humans, the computational modeling of situa-
tion awareness using a cognitive architecture, and the embodiment of cognitive
architectures in a robotic platform. Further investigations are planned to system-
atically assess the extent to which human cognitive processes may be applied to
autonomously operating robots in order to produce adaptive behaviour in un-
foreseen situations.
International Federation of Robotics. (2012). Retrieved 10 12, 2012, from
Anderson, J. R., & Lebiere, C. (1998). The Atomic Components of Thought.
Lawrence Erlbaum Associates.
Bagnell, J., Bradley, D., Silver, D., Sofman, B., & Stentz, A. (2010). Learning
for autonomous navigation. IEEE Robotics & Automation Magazine,
17(2), pp. 74-84.
Baumann, M. R., & Krems, J. F. (2009). A comprehension based cognitive
model of situation awareness. Digital Human Modeling, (pp. 192-201).
Benjamin, D., Monaco, J. V., Lin, Y., Funk, C., & Lyons, D. (2012). Using a
virtual world for robot planning. SPIE Defense, Security, and Sensing, (p.
doi 10.1117/12.923446 ).
Bratko, I. (2010). Comparison of machine learning for autonomous robot
discovery. Advances in Machine Learning I, 262/2010, pp. 441-456.
Braune, R., & Trollip, S. (1982). Towards an internal model in pilot training.
Aviation, Space and Environmental Medicine(53), pp. 996-999.
Decker, M., Dillmann, R., Dreier, T., Fischer, M., Gutmann, M., Ott, I., &
Döhmann, I. (2011). Service robotics: do you know your new companion?
Framing an interdisciplinary technology assessment. Poiesis & Praxis.
International Journal of Ethics of Science and Technology Assessment, 8,
pp. 25-44.
Durso, F. T., & Sethumadhavan, A. (2008). Situation awareness: understanding
dynamic environments. Human Factors: The Journal of the Human
Factors and Ergonomics Society, 50(3), pp. 442-448.
Durso, F., Rawson, K., & Girotto, S. (2007). Comprehension and situation
awareness. In F. Durso, R. Nickerson, S. Dumais, S. Lewandowsky, & T.
Perfect, Handbook of applied cognition (2. ed., pp. 163-194). New York:
Endley, M. (1995). Measurement of situation awareness in dynamic systems.
Human Factors, 37(1), pp. 65-84.
Endsley, M. (1995). Toward a theory of situation awareness in dynamic
systems. Human Factors, 37(1), pp. 65-84.
Endsley, M. R. (1993). Situation awareness and workload: flip sides of the same
coin. Proceedings of the Seventh International Symposium on Aviation
Psychology, pp. 906-911.
Gaba, D. M., Howard, S. K., & Small, S. D. (1995). Situation awareness in
anesthesiology. Human Factors: The Journal of the Human Factors and
Ergonomics Society, 37(1), pp. 20-31.
Golightly, D., Wilson, J. R., Lowe, E., & Sharples, S. (2010). The role of
situation awareness for understanding signalling and control in rail
operations. Theoretical Issues in Ergonomics Science, 1, pp. 84-98.
Gonzales, C., & Dutt, V. (2007). Learning to control a dynamic task: a system
dynamics cognitive model of the slope effect. Proceedings of the 8th
International Conference on Cognitive Modeling ICCM, (pp. 61-66).
Gonzalez, C., & Dutt, V. (2010). Instance-based learning models of training.
Proceedings of the Human Factors and Ergonomics Society Annual
Meeting, (pp. 2319-2323).
Gonzalez, C., Lerch, J. F., & Lebiere, C. (2003). Instance-based learning in
dynamic decision making. Cognitive Science, pp. 591-635.
Gugerty, L. J., & Tirre, W. C. (1996). Situation awareness: a validation study
and investigation of individual differences. Proceedings of the Human
Factors and Ergonomics Society Annual Meeting, (pp. 564-568).
Ma, R., & Kaber, D. B. (2007). Situation awareness and driving performance in
a simulated navigation task. Ergonomics, 50(8), pp. 1351-1364. Retrieved
November 21st, 2011, from
Peters, J., Morimoto, J., & Tedrake, R. (2009). Robot Learning. IEEE Robotics
& Automation Magazine, 16(3), pp. 19-20.
Sofge, D., Trafton, J. G., Cassimatis, N., Perzanowski, D., Bugajska, M.,
Adams, W., & Schultz, A. (2004). Human-robot collaboration and
cognition with an autonomous mobile robot. Proceedings of the 8th
Conference on Intelligent Autonomous Systems (IAS-8) (pp. 80-87). IOS
... Nitsch [8] stated that SA in context of using service robots in personal households is a relevant construct. Until today robots aren't flexible enough to react appropriately in sudden and non-predictable situation. ...
In-patient and out-patient care processes in the health care system are strongly affected by the effects of demographic change and a tense skill shortage. Digitization - both as a digitizing transformation process and as a process of implementing technical innovations such as robots - is intended to counteract these processes. Not only for safe interaction with a care robot, but also for its acceptance, humans must be able to understand and predict the robot’s movements. The ability to make predictions about the movements of a robot/technology and to be willing to interact with it is linked to the construct of situation awareness. In an experimental setting, a care situation was simulated in the full-scope simulator and the effects of the interaction on the situation awareness and the acceptance of the robot were investigated. The experiment was realized with 33 participants. The methods used were the SAGAT technique and an adapted UTAUT questionnaire. Significant correlations between UTAUT predictors and added measures were found. A stepwise hierarchical regression model found Performance Expectancy and Attitude as significant predictors of acceptance, but none of the other UTAUT and added factors were identified to be significant predictors. There was no significant effect between situation awareness and acceptance. A descriptive analysis of the measures of situation awareness showed an average to good situation awareness across all participants.
Full-text available
The IBL theory (IBLT) was developed to demonstrate the cognitive processes and mechanisms involved in dynamic decision making. IBLT has been implemented in several ACT-R cognitive models that showcase a variety of training effects and generate predictions resulting from several training manipulations. Particular implementations of IBLT-based models of training may depend on the task representation, the cues that are important for that task, and the particular decision actions involved, but the cognitive processes followed are generic. Here, we summarize different cognitive models for a diverse set of tasks relying on the IBLT. We also present an initial implementation of an IBL modeling tool that will help researchers develop their own IBL models without a need to program in ACT-R. Copyright 2010 by Human Factors and Ergonomics Society, Inc. All rights reserved.
Conference Paper
Full-text available
We are building a robot cognitive architecture that constructs a real-time virtual copy of itself and its environment, including people, and uses the model to process perceptual information and to plan its movements. This paper describes the structure of this architecture. The software components of this architecture include PhysX for the virtual world, OpenCV and the Point Cloud Library for visual processing, and the Soar cognitive architecture that controls the perceptual processing and task planning. The RS (Robot Schemas) language is implemented in Soar, providing the ability to reason about concurrency and time. This Soar/RS component controls visual processing, deciding which objects and dynamics to render into PhysX, and the degree of detail required for the task. As the robot runs, its virtual model diverges from physical reality, and errors grow. The Match-Mediated Difference component monitors these errors by comparing the visual data with corresponding data from virtual cameras, and notifies Soar/RS of significant differences, e.g. a new object that appears, or an object that changes direction unexpectedly.
Full-text available
Effective collaboration between robots and humans requires the use of an efficient interface whereby a human can communicate and interact with a robot almost as efficiently as he/she would with another human. In this interaction the human may act as a supervisor and/or collaborator with the robot. Human-robot collaboration is facilitated by a number of capabilities built into the robot and robot interface, including voice recognition, natural language and gesture understanding, and behaviors supporting dynamic autonomy. The inclusion of cognitively plausible representations and processes aboard the robot provides a further basis for facilitating collaboration between humans and robots, thereby reducing the human effort required to adapt to limitations of the robot as a non-human collaborator.
Full-text available
Rail signalling is typical of a complex control task where situation awareness (SA) has been thought to play a significant role. Taking into account the main theoretical positions, and through reviewing cognitive accounts of signalling, it is apparent that SA is a useful construct in describing and understanding rail signalling at both a cognitive and a system level. Development and maintenance of SA is complex, requiring much expertise and knowledge on the part of the signaller to extract and synthesise relevant information from the environment. The display and other artefacts and actors in the system also play a major role in the construction and maintenance of SA. This suggests that it is not sufficient to consider SA purely as an individual's product of knowledge. Instead, the active use of SA, such as in design, must reflect the distributed nature of the signalling environment, as found in other command and control-type domains. Rail signalling has not yet been subjected to in-depth analysis in terms of SA and has characteristics that make it distinct from other areas where SA has been analysed (e.g. air traffic control). This paper reviews the validity of SA and associated concepts to this safety-critical function, as well as highlighting some of the characteristics of SA (multiple tasks, display-based and distributed SA, role of local knowledge and expertise) that must be considered to use the construct in an operational context.
Full-text available
This chapter contains section titled:
Written by a team of leading international researchers under the guidance of Frank Durso, the second edition of the Handbook of Applied Cognition brings together the latest research into this challenging and important field, and is presented across thirty stimulating and accessible chapters. Stewarded by experiences editors from around the globe, the handbook has been fully updated with eleven new chapters covering materials that focus on the topics critical to understanding human mental functions in complex environments. It is an essential single-source reference for researchers, cognitive engineers and applied cognitive psychologists, as well as advanced students in the flourishing field of applied cognition.
The first experiment found that varying the rate of road hazards in a personal-computer-based driving simulator had no effect on subjects' situation awareness, as measured in the simulator. Thus, setting a high rate of hazards does not distort subjects' situation awareness. In the second experiment, the situation awareness test was found to predict driving performance in a realistic simulator. Individual differences in situation awareness were correlated with working memory and psychomotor abilities.