Content uploaded by Melodie Yashar
Author content
All content in this area was uploaded by Melodie Yashar on Apr 29, 2022
Content may be subject to copyright.
Integrating Mission Timelines and Procedures to Enhance
Situational Awareness in Human Spaceflight Operations
John A. Karasinski
NASA Ames Research Center
Mountain View, California, USA
Jimin Zheng
Melodie Yashar
San José State University Research
Foundation
Mountain View, California, USA
Jessica J. Marquez
NASA Ames Research Center
Mountain View, California, USA
ABSTRACT
Future human spaceight missions to the Moon, Mars, and beyond
will have signicant transmission delays and reduced communi-
cations bandwidth compared to operations in Low Earth Orbit,
requiring increased crew autonomy due to the limited availability
of real-time support from Mission Control Center (MCC). In this
paper, we argue for the development of technology to enhance
situational awareness (SA) for a remote MCC and astronauts on-
board the vehicle, describe a usability study conducted towards
this aim, and identify key usability features for the design of future
integrated timeline tools. By tightly integrating timelines and proce-
dures, future crew systems can predict future constraint violations,
suggest resolutions to these violations, and ultimately support more
successful missions.
CCS CONCEPTS
•Human-centered computing
→
Human computer interac-
tion (HCI);Visualization;•Information systems
→
Decision
support systems.
KEYWORDS
timeline, procedure execution, situational awareness, space explo-
ration
ACM Reference Format:
John A. Karasinski, Jimin Zheng, Melodie Yashar, and Jessica J. Marquez.
2022. Integrating Mission Timelines and Procedures to Enhance Situational
Awareness in Human Spaceight Operations. In Proceedings of SpaceCHI
2.0 workshop at ACM SIGCHI 2022 (CHI 2022). ACM, New York, NY, USA,
4pages.
1 INTRODUCTION
Future human spaceight missions are characterized as requiring
increased crew autonomy due to communication delays and re-
duced support from Mission Control Center (MCC). While this
level of crew autonomy is desirable, it produces new challenges
as these shifting responsibilities will result in reduced situational
awareness for both support personnel in MCC and crew members
onboard. The distances involved with future missions will cause
delayed communications, prevent MCC from following crews in
This paper is authored by an employee(s) of the United States Government and is in
the public domain. Non-exclusive copying or redistribution is allowed, provided that
the article citation is given and the authors and agency are clearly identied as its
source.
CHI 2022, April 30–May 5, 2022, New Orleans, LA
2022.
real-time as they execute procedures, and reduce their ability to col-
laboratively resolve issues. Limited bandwidth will further reduce
situational awareness, as MCC will no longer be able to downlink
thousands of streams of sensor data and multiple high-denition
video feeds from the vehicle. We envision a future where human
spaceight operations are better integrated, where crew timelines
(collections of activities) and task procedures (collections of indi-
vidual actions) are tightly coupled to support and assist astronauts.
Such an integration would enable tools to have shared situation
awareness of spaceight activities.
Towards this aim, we conducted a remote usability study to
evaluate a new software prototype being developed to assist with
procedure execution. The purpose of the study was to investigate
the utility of countermeasures designed to assist with procedure
execution and determine how the various types of data generated
during International Space Station (ISS) operations can be inte-
grated into a mission timeline tool to improve overall situational
awareness of mission personnel. Remote usability tests were com-
pleted with participants from the Human Computer Interaction
Group at NASA Ames Research Center and those with experience
in ISS operations as payload controllers to identify key usability
issues with the interface and user experience. In this paper, we ar-
gue for the development of tools to enhance situational awareness
for MCC and astronauts onboard, and we identify key usability
features for the design of future integrated timeline tools.
2
TIMELINE AND PROCEDURE INTEGRATION
Enabling future crew autonomy — the ability for astronauts to
execute tasks more independently from MCC — requires a substan-
tial change in NASA’s current concept of operations and must be
supported with new technologies (e.g., advanced automated sys-
tems, improved just-in-time training). We propose that one way of
improving on the state-of-the-art for crew autonomy is to better
integrate two of the software systems that currently support task
execution: timeline and procedures management. At the moment,
timelines are pre-determined by a large team of Ops Planners, who
coordinate with the rest of MCC to create a packed schedule for
each of the seven crewmembers onboard the ISS. Each scheduled ac-
tivity has one or more associated procedures that astronauts need to
follow to complete each task. Astronauts onboard the ISS currently
use Operations Planning Timeline Integration System (OPTIMIS)
to view their timelines and International Procedures Viewer (IPV)
to view their procedures [
3
,
7
]. However, there is currently no rela-
tionship between these two fundamental software systems for task
execution aside from including a link to procedures embedded in
the scheduled activity description. In current operations the only
CHI 2022, April 30–May 5, 2022, New Orleans, LA Karasinski, et al.
exception to this are extravehicular activities (EVAs), which require
months of dedicated planning and the addition of dozens of addi-
tional personnel in MCC during real-time execution. This current
paradigm, while successful for ISS operations, is only possible when
supported by near real-time communications and high bandwidth
availability in Low Earth Orbit. Long duration exploration missions
will benet from neither of these capabilities, requiring a shift in
architecture for how we support human spaceight. A modern
human-systems integration architecture (HSIA) should include a
“one-stop shop” crew interface that reduces cognitive workload,
facilitates operations, and ultimately enables crew autonomy [
4
,
8
].
This system should be ubiquitous and accessible across the space-
craft, providing access to key aspects of the system, such as the
vehicle status, training protocols, procedures, and mission timelines.
Consistent designs across the system will enhance usability across
the HSIA and can help to provide enhanced situational awareness
to an increasingly remote MCC.
3 EXPLORATORY RESEARCH
A series of remote interviews were conducted as a means of con-
sidering new capabilities within timeline tools for enhancing sit-
uational awareness (SA) during task execution. The purpose of
the interviews was to determine what kinds of information are
most useful to track task execution progress and success to better
determine ways to present this information. The utility of certain
types of data, metrics, and task progress information were evalu-
ated within a usability study which followed. The interviews were
conducted with 1) researchers focusing on human capabilities for
autonomous missions and 2) ISS operations personnel, and were
aimed at identifying pain points in current work processes. We
also asked participants about the potential benets of incorporat-
ing multimodal interactions with timeline information and data
relevant to procedure management.
Based on the interviews, the following objectives were identied
as research questions for a usability study:
•
Possible interface enhancements to enhance situational aware-
ness for the crewmember rather than MCC personnel could
include progress or task bar indicating task progress, aug-
mented feedback for steps that are known to be “harder” to
assist or enhance performance;
•
Validating graphical elements for: warnings relevant to com-
plete or incomplete task execution to enhance SA, or whether
time on task information is benecial to SA;
•
Validating whether boolean data types (e.g., on/o) for sen-
sors or other telemetry enhance situational awareness.
Based on the above objectives, we derived several concepts for
how timeline and procedures might be more tightly integrated, that
were later tested within a usability study. By following a procedures’
individual actions or steps, future systems could automatically sta-
tus the activity as started/in progress/complete. Task execution time
estimates could then be used to predict whether the activity is run-
ning early, on-time, or late. This alone would provide better shared
situation awareness for both crew and MCC and would provide a
low bandwidth way to continuously observe astronauts working in
space. Many of the systems that astronauts work with are highly in-
strumented and produce a variety of data from sensors (e.g., power
generation, system states, life support sensors). Visualization of this
data, which is already being collected onboard the ISS, often allows
procedural tracking at the step level, without requiring any extra
work by the astronauts. Tightly integrating the data from these sys-
tems into a common interface and allowing the data to be visualized
alongside the procedures and timelines would allow MCC to follow
task execution without requiring high bandwidth video camera
feeds. In addition to the two separate software systems currently
used to track procedures and timelines, many activities require
astronauts to work with yet another system to complete their tasks.
An application that enables procedure execution workows inte-
grated with timeline tools may improve situational awareness and
reduce cognitive load. Finally, most spaceight procedures require
the retrieval and/or stowage of tools and hardware from several
locations across the spacecraft. This information could be trans-
ferred from the procedures and used as constraints in the activities,
reducing disruptions among the crew when locations or tools are
not available due to unforeseen conicts.
4 USABILITY STUDY
To evaluate some of these ideas for procedure and timeline in-
tegration, we conducted a series of usability studies with space
operations-naive participants, as well as individuals who have ex-
perience with ISS ight operations. The goal of the usability stud-
ies was to investigate dierent techniques for improving general
situational awareness among ight controllers during spaceight
missions by providing enhanced electronic procedures and systems
data alongside mission timelines
The timeline application used as the foundation for the usability
studies was Playbook, a mission planning and execution software
used on a variety of NASA analog missions as the primary mission
planning tool [
1
,
2
]. Playbook visualizes a crew’s activities, day by
day, across an entire mission. Alongside these existing Playbook
capabilities, we created a low-delity prototype of new features
that allowed Playbook to integrate with external data systems and
Figure 1: The user interface in our usability study provides
immediate access to crew activities along with various sys-
tems data, such as signal acquisition, payload equipment
on/o states, and equipment power output.
Integrating Mission Timelines and Procedures to Enhance Situational Awareness in Human Spaceflight Operations CHI 2022, April 30–May 5, 2022, New Orleans, LA
visualize that information in the user interface as well. The proto-
typed features included real-time graphs, numeric displays, imagery,
progress meters, and status indicators (Figure 1).
In total, we had 9 participants in the usability study, includ-
ing two pilot studies, conducted remotely. Sessions lasted approxi-
mately 45 minutes per participant and consisted of the participant
taking a role as an MCC ight controller following the events of a
task execution (“ight-follow”). The scenario prompt read to the
user at the beginning of the usability session was:
Imagine you are an operations planner in the mission control
center for ISS. You are on-console monitoring the overall mis-
sion timeline in Playbook as the crew conduct their science
experiments. As an operations planner, your goal is to make
sure that crew members stay on task and on schedule, and
have resolutions prepared if there are ever deviations from the
planned timeline. In the following scenario, you will be moni-
toring the crew member’s progress as they start an experiment
called “Carburetor Float Valve Inspection,” which is related to
maintenance on an on-board generator.
As part of their roles as simulated ight controllers, participants
were asked to complete a series of tasks of various complexity as
instructed by the test conductor within a prototype of Playbook
created in the design tool Figma (Figure 2). One instruction was
given at a time over Microsoft Teams, and after the completion
of each task, participants were asked several situational questions
before moving onto the next.
The Playbook interface was augmented with features and data
relevant to procedure documentation workows. The Playbook
interface was augmented with a vertical timeline conguration
which guided participants through execution of the “Carburetor
Float Valve Inspection” task; while integrating data on step and
task completion, time on task, tools relevant for task completion,
multimedia training documents relevant to task completion, and
task accuracy. Participants were asked to identify how incomplete
or delayed procedure execution activities might impact scheduling
within the mission timeline.
Participants shared their screens so that we could see their in-
teractions with the prototype. Interaction with the prototype was
done through a “Wizard of Oz” methodology (Figure 3). Participants
were also asked to speak aloud as they went about accomplishing
the task. Think-aloud testing is a usability engineering method
championed by Jakob Nielsen and is a direct observation method of
user testing that asks users to report on their cognitive process as
the demo occurs in real-time [
5
]. Users are asked to share thoughts
with each interaction in the prototype as a means of assessing user
focus, how the user brings prior knowledge to bear, and what the
predominant usability issues may be based on the user’s reasoning.
Usability issues have been categorized as either: a usability catas-
trophe, a major severity, or a minor severity. Severity ratings, ac-
cording to Nielsen, are intended to reveal the most serious usability
problems in advance of releasing a system to market [
6
]. The sever-
ity of a usability problem is a combination of the frequency with
which the problem occurs, the impact of the problem (whether
it might be easy or dicult for the users to overcome), and the
persistence of the problem. In this context, a usability catastrophe
Figure 2: Participants were asked to identify sources of exe-
cution anomalies as part of the usability study. When crew is
running behind, the estimated time remaining in the activity
can be used to identify potential problems downstream in
the schedule.
Figure 3: Progress into the simulated execution ight-follow
task, as depicted in Playbook via Wizard of Oz. The view
allows ight controllers to visualize the status of individual
activities within the task execution schedule. Prototyped
features add additional situational data for MCC support
personnel to draw from to assess crew performance.
is classied as a problem imperative to x, a major usability prob-
lem is important to x and thus should be given high priority, a
minor usability problem should be given low priority, and cosmetic
problems need not be xed unless all other higher priority usability
issues are addressed [6].
From the usability studies, the following major or higher usability
takeaways emerged:
•Verbiage and nomenclature — Terminology and verbiage is
not universally understood. Without training, dierent users
can interpret the same piece of text to mean signicantly
dierent things. Therefore, it is important that verbiage is
clear and that the use of terminology across the tool remains
consistent. For example, when referring to an equipment
CHI 2022, April 30–May 5, 2022, New Orleans, LA Karasinski, et al.
hatch, it is important to decide to use a denitive term, such
as “closed,” instead of a potentially ambiguous one, like “on.”
•
Visual scale and salience of information — Participants com-
mented that the representation of systems and procedure
data in the prototype could be improved. Multiple partici-
pants commented that the placement of relevant pieces of
information in small sections of the user interface was prob-
lematic and made them dicult to nd; while conversely,
seemingly insignicant pieces of information were given
much more interface real estate. Whether it be a timer, a
graph, a status indicator, etc., the information that is most
important to the current circumstance of the task execution
should be the most easily accessible.
•
Meaning of color — Like verbiage and nomenclature, with-
out training, colors in the user interface can mean dierent
things to dierent people, so communication through color
must be clear. Typically, green is generally used in various
software tools to indicate something positive, while red is
used to indicate something that is negative. In the context of
the usability study task to monitor an execution task, green
can be used to communicate the desired state as it relates
to the goal of the task, not for the state of the system on its
own. For example, using the color green to indicate that the
equipment hatch from above is “closed” is counterproductive
if the intention of the task is for the equipment hatch to be
“open.”
•
Expected state vs current state — What is the current state
of the system, and what should the state of the system be?
The integration of systems data within a timeline tool like
Playbook needs to be shown not just as it is, but also as it
needs to be. Because EVA activities often require the crew
to perform an action until an expected state is achieved, that
end state should be explicitly depicted in the user interface,
so that users have a constant reference. The use of clear
and concise verbiage and color, as mentioned earlier, is one
method of conveying this information to the user.
Each of these four major usability takeaways introduce opportu-
nities for future research and usability studies towards enhancing
situational awareness. All of the participants noted that the ad-
ditional information provided by the graph and the progress and
status indicators was helpful to provide insight into task execution
progress, but participants interpreted the new features in incon-
sistent ways. While the benets of tying together timelines and
procedures are clear, improper integration may instead lead to
greater confusion by the crew or those monitoring remotely in
MCC.
5 FUTURE WORK
The development of future crew systems and human systems inte-
gration architectures must focus on enabling crew autonomy in the
absence of real-time mission support. These systems will provide
enhanced situational awareness to both a remote MCC and to crew
within the spacecraft and will help to reduce crew idle time. By pro-
viding tighter integration between timelines and procedures, future
interfaces can support crew by suggesting dynamic, constraint-
free plans that allow for successful missions in the presence of
anomalies. Additional research in the development of multimodal
countermeasures for enhanced crew autonomy will benet from
closer integration between tools for managing scheduling and time-
lines with data and workows relevant to procedure execution.
Applications for this research could include analog experiments
where crewmembers are required to execute complex procedures
over the course of the mission, yet nonetheless are given some
exibility with self-scheduling.
ACKNOWLEDGMENTS
This work was performed under a US Govt. Contract in the Human-
Systems Integration Division at NASA Ames Research Center. This
research was funded in part by the NASA Human Research Pro-
gram’s Human Factors and Behavior Performance Element (NASA
Program Announcement number 80JSC017N0001-BPBA) Human
Capabilities Assessment for Autonomous Missions (HCAAM) Vir-
tual NASA Specialized Center of Research (VNSCOR) eort (NASA
grant number 80NSSC19K0657).
REFERENCES
[1]
Jessica J. Marquez, Steven Hillenius, Bob Kanefsky, Jimin Zheng, Ivonne Deliz, and
Marcum Reagan. 2017. Increasing crew autonomy for long duration exploration
missions: Self-scheduling. In 2017 IEEE Aerospace Conference. IEEE, Big Sky, MT,
USA, 1–10. https://doi.org/10.1109/AERO.2017.7943838
[2]
Jessica J. Marquez, Guy Pyrzak, Sam Hashemi, Kevin McMillin, and Joseph Med-
wid. 2013. Supporting Real-Time Operations and Execution through Timeline
and Scheduling Aids. In 43rd International Conference on Environmental Sys-
tems. American Institute of Aeronautics and Astronautics, Vail, CO. https:
//doi.org/10.2514/6.2013-3519
[3]
Maurizio Martignano, Mikael Wol, Uwe Brauer, and Paul Kiernan. 2007. Software
assisted authoring and viewing of ISS crew procedures. Acta Astronautica 61,
11-12 (Dec. 2007), 1053–1060. https://doi.org/10.1016/j.actaastro.2006.12.028
[4]
Kaitlin McTigue, Megan Parisi, Tina Panontin, Shu-Chieh Wu, and Alonso Vera.
2021. Extreme Problem Solving: The New Challenges of Deep Space Exploration.
Association for Computing Machinery, Virtual. https://human- factors.arc.nasa.
gov/publications/McTigue_Parisi_Panontin_Wu_Vera_SpaceCHI.pdf
[5] Jakob Nielsen. 1993. Usability Engineering. Elsevier Science.
[6]
Jakob Nielsen. 1994. Severity Ratings for Usability Problems. https://www.
nngroup.com/articles/how-to-rate-the-severity-of-usability- problems/ publisher:
Nielsen Norman Group.
[7]
Ernest E. Smith, David J. Korsmeyer,and Vern Hall. 2014. Exploration Technologies
for Operations. In SpaceOps 2014 Conference. American Institute of Aeronautics
and Astronautics, Pasadena, CA. https://doi.org/10.2514/6.2014-1619
[8]
Azita Valinia, John R. Allen, David R. Francisco, Joseph I. Minow, Jonathan A.
Pellish, and Alonso H. Vera. 2022. Safe Human Expeditions Beyond Low Earth
Orbit (LEO). Technical Report NASA/TM-20220002905 NESC-RP-20-01589. https:
//ntrs.nasa.gov/citations/20220002905
