Conference PaperPDF Available

Cognitive Architectural Control for Free-Flying Robots on the Lunar Orbital Platform-Gateway


Abstract and Figures

In this paper we describe a proposed integration between the DIARC robot cognitive architecture and the NASA Astrobee robot to enable goal-directed cognition and natural language control over this platform. After describing the capabilities of both architectures, we describe how the architectures can be integrated and provide examples of capabilities that would be enabled through this integration.
Content may be subject to copyright.
Cognitive Architectural Control for Free-Flying
Robots on the Lunar Orbital Platform-Gateway
Jon Emmanuel Serrano
Colorado School of Mines
Golden, CO USA
Shania Jo Runningrabbit
Colorado School of Mines
Golden, CO USA
Sayanti Roy
Colorado School of Mines
Golden, CO USA
Alexandra Bejarano
Colorado School of Mines
Golden, CO USA
Tom Williams
Colorado School of Mines
Golden, CO USA
Abstract—In this paper we describe a proposed integration
between the DIARC robot cognitive architecture and the NASA
Astrobee robot to enable goal-directed cognition and natural lan-
guage control over this platform. After describing the capabilities
of both architectures, we describe how the architectures can be
integrated and provide examples of capabilities that would be
enabled through this integration.
Index Terms—Cognitive Architectures, Space Robotics, As-
The Lunar Orbital Platform-Gateway will serve as a staging
point for crewed and uncrewed missions to the Moon, Mars,
and beyond [4]. While the Gateway will sustain human crews
for small periods of time, it will be primarily staffed by
autonomous caretaker robots like the free-flying Astrobee
platform [11]: the Gateway’s sole residents during quiescent
(uncrewed) periods [3]. This creates a unique human-technical
system comprised of two categories of human teammates:
ground control workers permanently stationed on earth and
astronauts that may transition over time between work on
Earth, the Gateway, the Moon, and Mars; and three types of
machine teammates: robot workers stationed on the Gateway;
robot workers stationed on the Moon and Mars; and the
Gateway itself.
This distributed multi-robot system comprised of both em-
bodied actors (e.g., robots) and minimally embodied actors
(e.g., the Gateway) must be capable of (1) autonomously allo-
cating tasks to and coordinating computation between multiple
robot bodies, and (2) accepting directives and communicating
feedback to human teammates, both through standard control
interfaces and through natural language. Interaction of the dis-
tributed multi-robot system with human teammates can occur
in a variety of ways: each individual robot and the Gateway
may have individual minds and one-to-one communication
with human teammates, robots and the Gateway may all share
the same mind (i.e. a hive mind) and line of communication, or
Fig. 1. Astrobee robot in the simulator.
only a single actor has one-to-one communication with human
teammates as it serves to collect feedback from all other robots
and distribute commands from human teammates to them (e.g.,
a ground control worker on Earth communicating with a local
social interface that then transmits commands to non-social
robot workers stationed on the Moon.)
In this paper, we argue that these capabilities are best
enabled through Robot Cognitive Architectures such as the
Distributed Integrated Affect Reflection Cognition (DIARC)
architecture [10], which not only provides capabilities for
goal-directed cognition and natural language interaction, but
does so in a way that is naturally suited to enable interaction
between multiple humans and multi-robot distributed systems.
To explain how cognitive architectures could be used for
control of and action with robots like the Astrobee, in this
paper we will describe (1) the unique capabilities of those
robots and of the DIARC architecture; (2) how the Astrobee
robots can be integrated into the DIARC architecture; and (3)
an example dyadic interaction that can be enabled through this
Fig. 2. Network graph of all the available Astrobee ROS nodes.
Astrobees are free-flying robots designed to work closely
with astronauts and helping them with routine tasks aboard
the International Space Station (ISS). The Astrobee Robot
Software uses the open-source Robot Operating System (ROS)
as message-passing middleware for performing vision-based
localization, autonomous navigation, docking and perching,
managing various sensors and actuators, and supporting user
interaction via screen-based displays, light signaling, and
sound. ROS enables Astrobee robots to be operated au-
tonomously or via teleoperation, monitors timeouts and ac-
tion execution progress, identifies system faults, and controls
nodelet lifecycle [5]. While ROS affords Astrobee robots a
wide range of capabilities, however, they currently lack natural
language understanding and generation capabilities. In our
research, we are working to provide such capabilities through
the DIARC architecture, in order to allow astronauts to task
Astrobees to execute complex tasks such as Spot Checks
without requiring the use of a graphical interface.
The Distributed, Integrated, Affect, Reflection, Cognition
(DIARC) architecture is a component-based robot architecture
that focuses on enabling robust goal-driven cognition and
open-world spoken language interaction [10]. DIARC has
previously been used for a variety of projects of relevance to
or in collaboration with NASA, e.g., work on shared mental
model development from [6]. Central to DIARC are a key set
of language-, memory-, and action-oriented components.
Language-oriented components allow for robots to pro-
cess and understand referring expressions such as definite,
anaphoric and deictic expressions [13], as well as under-
standing and generating a variety of speech acts (including
clarification requests), through DIARC’s Dialogue Manager.
This includes speech acts phrased as indirect speech acts [14].
For language-oriented components to allow the processing of
any kind of referring expression (i.e. reference resolution),
they must leverage memory-oriented components.
Memory-oriented components allow for robots to access and
extract information from knowledge bases distributed across
the robot architecture. This can be accomplished, for example,
using either a centralized Prolog knowledge base such as
DIARC’s Belief Manager, or through POWER Consultants,
which serve as interfaces to information stored across multiple
heterogeneous knowledge bases [17], [18]. These components
are in turn leveraged by action-oriented components which
manage high-level goals and actions.
At the core of DIARC is its Goal Manager, which allows
for goal-directed cognition, i.e., creation, satisfaction, and
monitoring of logically specified one-time and maintenance
goals. DIARC also provides components for physical manip-
ulation and navigation in order to provide primitive actions
for satisfying theses sorts of goals [15], but these action-
oriented components are de-emphasized relative to cognition
and dialogue oriented capabilities.
DIARC is implemented in the Agent Development Envi-
ronment (ADE). ADE is a software Middleware that supports
the development and implementation of agent architectures [8]
using a distributed multi-agent system computing infrastruc-
ture (cp. [2], [12]). This framework provides dynamic, reliable,
fault-recovering, remotely accessible, distributed computing
and autonomic computing by treating architectural compo-
nents as autonomous agents [1], [7], [9].
By integrating the Astrobee system and DIARC robot
architecture (through specific integration of the Astrobee’s
ROS and ADE MAS middlewares), we have produced a new
robot that can both leverage the unique physical capabilities
of the Astrobee Robot and the state-of-the-art linguistic capa-
bilities of the DIARC architecture, as well as new synergistic
capabilities made possible only through this integration (e.g.,
natural language tasking to perform unique tasks aboard the
The integration between the Astrobee software and DIARC
are comprised of three types of components.
DIARC Components: ADE components that only exist
within the DIARC architecture, and are only aware of
components implemented in the ADE middleware.
Astrobee Components: Astrobee components made up
of ROS nodes that sends and gets data from other nodes
using the publish/subscribe model, either publishing top-
ics or subscribing to them.
Dual-Citizen Components: Components that exist
within both architectures, and can communicate both with
ADE components and Astrobee nodes. In our current
implementation a single Dual-Citizen Component is used,
but the architecture is sufficiently flexible to allow for an
arbitrary number of Dual-Citizen Components to effect
different points of interface between the architectures.
As shown in Figure 3 the Astrobee operates using a large
number of ROS nodes. To effect our integration, a DIARC
utility is used to automatically generate wrappers to connect
between DIARC and ROS. For example, a wrapper component
Fig. 3. Diagram of our proposed integrated Architecture with relevant
components and their information flow.
is generated for the Astrobee’s MLP Mobility node so that
movement goals can be sent from DIARC through publishing
to the topics used by that node. This also provides DIARC with
access to pose representations that can be used to determine
where the robot is on the station. In addition, the Astrobee has
a number of unique sensors and effectors used for functionality
beyond point-to-point motion, which are exposed through its
ROS nodes and can be wrapped in this way. These include
the Astrobee’s cameras, laser pointer, touch screen, arm,
speaker/microphone, and flashlights.
The Dual-Citizen component used in our integration is
implemented as a Java class that extends the ADE Component
interface and uses service calls to communicate with other
ADE nodes. The component then imports these generated
wrappers described above in order to participate in pub-
lish/subscribe communication with the wrapped ROS nodes.
By integrating Astrobee and DIARC, each can leverage the
other’s capabilties, resulting in new synergistic capabilities
and behaviors. DIARC can leverage Astrobee’s Free-flying
navigational capabilities and situated spatial knowledge in
order to discuss, reason about, and travel through the ISS,
and the Astrobee can leverage DIARC’s linguistic capabilities,
to enable the Astrobee robots to travel to locations that are
loosely specified; an instruction such as ”I need you to survey
the ship for leaks,” for example, is not literally a direct
command, and does not clearly specify a location, yet implies
an intended command to travel around the ship, and, when
paired with Astrobee’s knowledge of its environment, may
imply locations around the ship to which the robot should
In this section we present an example of how we envision
our integrated approach playing out. While the implementation
has not yet been fully verified end-to-end, each constituent
piece has been verified. For an example of how a simi-
lar demonstration was implemented on the Vulcan robotic
wheelchair, we direct the reader to Williams et al. [16]. In this
example, the Astrobee is told “Astrobee, go to Control Panel
C”. After recognition, DIARC’s ASR component passes this
utterance to its NLP component, which performs parsing and
reference resolution. This utterance is parsed into the utterance
form Statement(moveto(X)) with supplemental semantics
At the start of this interaction, the robot’s
Shor tT ermM emory and F ocusof Attention are both
empty, and thus the robot’s LongT ermM emory is searched
for a suitable referent to bind to the variable X. The
property controlpanelc(X)can be advertised by the
Astrobee component if that component is implemented as
a POWER consultant and populated with knowledge of
relevant objects, including their properties and locations. If
Control Panel C is represented in the Astrobee component
with memory trace astrobee5, this trace will be bound
to X, producing Statement(moveto(astrobee5)), which
is passed to DIARC’s pragmatic reasoning component.
This component has a rule with implicative content:
Statement(moveto(X)) goal(at(self , X)), resulting
in the goal at(astrobee5)being adopted. If the Astrobee
Component exposes a method with the effect at(self, X )
using Action Effect Annotations, then the robot will identify
that this method can be used to achieve at(astrobee5). If
this method has access to the metric location of astrobee5,
then the Astrobee Component can broadcast a MLP Mobility
goal to travel to that location. ROS’s Mobility node can then
receive this goal and initiate a new motion planning task to
go to the specified coordinates.
In this brief workshop paper we have demonstrated how
the DIARC architecture and the ROS-based Astrobee robot
can be integrated in order to enable goal-directed natural
language control over the Astrobee robot. While we have not
yet evaluated the demonstration described in an end-to-end
fashion, each constituent piece of the demonstration has been
shown to work either in our recent work or through our past ten
years of work on DIARC. In future work, we aim to complete
this demonstration and produce a configuration that will allow
natural language tasking of spot checks, as described by Bualat
et al. [3].
[1] Virgil Andronache and Matthias Scheutz. Ade—an architecture develop-
ment environment for virtual and robotic agents. International Journal
on Artificial Intelligence Tools, 15(02):251–285, 2006.
[2] Fabio Bellifemine, Agostino Poggi, and Giovanni Rimassa. Jade–a
fipa-compliant agent framework. In Proceedings of PAAM, volume 99,
page 33. London, 1999.
[3] Maria G Bualat, Trey Smith, Ernest E Smith, Terrence Fong, and
DW Wheeler. Astrobee: A new tool for iss operations. In 2018 SpaceOps
Conference, 2018.
[4] Jason C Crusan, R Marshall Smith, Douglas A Craig, Jose M Caram,
John Guidi, Michele Gates, Jonathan M Krezel, and Nicole B Herrmann.
Deep space gateway concept: Extending human presence into cislunar
space. In 2018 IEEE Aerospace Conference, pages 1–10. IEEE, 2018.
[5] Lorenzo Fluckiger and Brian Coltin. Astrobee robot software: Enabling
mobile autonomy on the iss. 2019.
[6] Felix Gervits, Terry W Fong, and Matthias Scheutz. Shared mental
models to support distributed human-robot teaming in space. In 2018
AIAA SPACE and Astronautics Forum and Exposition, page 5340, 2018.
[7] James Kramer and Matthias Scheutz. Ade: A framework for robust com-
plex robotic architectures. In 2006 IEEE/RSJ International Conference
on Intelligent Robots and Systems, pages 4576–4581. IEEE, 2006.
[8] James Kramer and Matthias Scheutz. Development environments for
autonomous mobile robots: A survey. Autonomous Robots, 22(2):101–
132, 2007.
[9] Matthias Scheutz. Ade: Steps toward a distributed development and
runtime environment for complex robotic agent architectures. Applied
Artificial Intelligence, 20(2-4):275–304, 2006.
[10] Matthias Scheutz, Thomas Williams, Evan Krause, Bradley Oosterveld,
Vasanth Sarathy, and Tyler Frasca. An overview of the distributed
integrated cognition affect and reflection diarc architecture. In Maria
Isabel Aldinhas Ferreira, Joao S.Sequeira, and Rodrigo Ventura, editors,
Cognitive Architectures. 2019.
[11] Trey Smith, Jonathan Barlow, Maria Bualat, Terrence Fong, Christopher
Provencher, Hugo Sanchez, and Ernest Smith. Astrobee: A new platform
for free-flying robotics on the international space station. In Proceedings
of International Symposium on Artificial Intelligence, Robotics and
Automation in Space, 2016.
[12] Katia Sycara, Massimo Paolucci, Martin Van Velsen, and Joseph Gi-
ampapa. The retsina mas infrastructure. Autonomous agents and multi-
agent systems, 7(1-2):29–48, 2003.
[13] Tom Williams, Saurav Acharya, Stephanie Schreitter, and Matthias
Scheutz. Situated open world reference resolution for human-robot
dialogue. In 2016 11th ACM/IEEE International Conference on Human-
Robot Interaction (HRI), pages 311–318. IEEE, 2016.
[14] Tom Williams, Gordon Briggs, Bradley Oosterveld, and Matthias
Scheutz. Going beyond literal command-based instructions: Extending
robotic natural language interaction capabilities. In AAAI, pages 1387–
1393, 2015.
[15] Tom Williams, Rehj Cantrell, Gordon Briggs, Paul Schermerhorn, and
Matthias Scheutz. Grounding natural language references to unvisited
and hypothetical locations. In Twenty-Seventh AAAI Conference on
Artificial Intelligence. Citeseer, 2013.
[16] Tom Williams, Collin Johnson, Matthias Scheutz, and Benjamin Kuipers.
A tale of two architectures: A dual-citizenship integration of natural
language and the cognitive map. In Proceedings of the 16th International
Conference on Autonomous Agents and Multi-Agent Systems, 2017.
[17] Tom Williams and Matthias Scheutz. Power: A domain-independent
algorithm for probabilistic, open-world entity resolution. In 2015
IEEE/RSJ International Conference on Intelligent Robots and Systems
(IROS), pages 1230–1235. IEEE, 2015.
[18] Tom Williams and Matthias Scheutz. A framework for resolving open-
world referential expressions in distributed heterogeneous knowledge
bases. In AAAI, pages 3958–3965, 2016.
ResearchGate has not been able to resolve any citations for this publication.
Full-text available
DIARC has been under development for over 15 years. Different from other cognitive architectures like SOAR or ACT-R, DIARC is an intrinsically component-based distributed architecture scheme that can be instantiated in many different ways. Moreover, DIARC has several distinguishing features, such as affect processing and deep natural language integration, is open-world and multi-agent enabled, and allows for “one-shot instruction-based learning” of new percepts, actions, concepts, rules, and norms. In this chapter, we will present an overview of the DIARC architecture and compare it to classical cognitive architectures. After laying out the theoretical foundations, we specifically focus on the action, vision, and natural language subsystems. We then give two examples of DIARC configurations for “one-shot learning” and “component-sharing”. We also briefly mention different use cases of DIARC, in particular, for autonomous robots in human-robot interaction experiments and for building cognitive models.
Conference Paper
Full-text available
Vulcan and DIARC are two robot architectures with very different capabilities: Vulcan uses rich spatial representations to facilitate navigation capabilities in real-world, campus-like environments, while DIARC uses high-level cognitive representations to facilitate human-like tasking through natural language. In this work, we show how the integration of Vulcan and DIARC enables not only the capabilities of the two individual architectures, but new syner-gistic capabilities as well, as each architecture leverages the strengths of the other. This integration presents interesting challenges, as DIARC and Vulcan are implemented in distinct multi-agent system middlewares. Accordingly, a second major contribution of this paper is the Vulcan-ADE Development Environment (VADE): a novel multi-agent system framework comprised of both (1) software agents belonging to a single robot architecture and implemented in a single multi-agent system middleware, and (2) " Dual-Citizen " agents that belong to both robot archi-tectures and that use elements of both multi-agent system middlewares. As one example application, we demonstrate the implementation of the new joint architecture and novel multi-agent system framework on a robotic wheelchair, and show how this integration advances the state-of-the-art for NL-enabled wheelchairs.
Conference Paper
Full-text available
A robot participating in natural dialogue with a human interlocutor may need to discuss, reason about, or initiate actions concerning dialogue-referenced entities. To do so, the robot must first identify or create new representations for those entities, a capability known as reference resolution. We previously presented algorithms for resolving references occurring in definite noun phrases. In this paper we present an algorithm for resolving references occurring in a wider array of linguistic forms, by making novel extensions to the Givenness Hierarchy, and evaluate the proposed algorithm on natural task-based human-human and human-robot dialogues.
Conference Paper
Full-text available
We present a domain-independent approach to reference resolution that allows a robotic or virtual agent to resolve references to entities (e.g., objects and locations) found in open worlds when the information needed to resolve such references is distributed among multiple heterogeneous knowledge bases in its architecture. An agent using this approach can combine information from multiple sources without the computational bottleneck associated with centralized knowledge bases. The proposed approach also facilitates " lazy constraint evaluation " , i.e., verifying properties of the refer-ent through different modalities only when the information is needed. After specifying the interfaces by which a reference resolution algorithm can request information from distributed knowledge bases, we present an algorithm for performing open-world reference resolution within that framework , analyze the algorithm's performance, and demonstrate its behavior on a simulated robot.
Conference Paper
Full-text available
The problem of uniquely identifying an entity described in natural language, known as reference resolution, has become recognized as a critical problem for the field of robotics, as it is necessary in order for robots to be able to discuss, reason about, or perform actions involving any people, locations, or objects in their environments. However, most existing algorithms for reference resolution are domain-specific and limited to environments assumed to be known a priori. In this paper we present an algorithm for reference resolution which is both domain independent and designed to operate in an open world. We call this algorithm POWER: Probabilistic Open-World Entity Resolution. We then present the results of an empirical study demonstrating the success of POWER both in properly identifying the referents of referential expressions and in properly modifying the world model based on such expressions.
Conference Paper
Full-text available
The ultimate goal of human natural language interaction is to communicate intentions. However, these intentions are often not directly derivable from the semantics of an utterance (e.g., when linguistic modulations are employed to convey politeness , respect, and social standing). Robotic architectures with simple command-based natural language capabilities are thus not equipped to handle more liberal, yet natural uses of linguistic communicative exchanges. In this paper, we propose novel mechanisms for inferring intentions from utterances and generating clarification requests that will allow robots to cope with a much wider range of task-based natural language interactions. We demonstrate the potential of these inference algorithms for natural human-robot interactions by running them as part of an integrated cognitive robotic architecture on a mobile robot in a dialogue-based instruction task.
Conference Paper
Full-text available
While much research exists on resolving spatial natural language references to known locations, little work deals with handling references to unknown locations. In this paper we introduce and evaluate algorithms integrated into a cognitive architecture which allow an agent to learn about its environment while resolving references to both known and unknown locations. We also describe how multiple components in the architecture jointly facilitate these capabilities. Copyright © 2013, Association for the Advancement of Artificial Intelligence ( All rights reserved.