Robot Teleoperation Interfaces for Customized Therapy for Autistic Children

Abstract

Socially Assistive Robots are effective at supporting autistic children in a variety of different therapies. Therapists can control the robots' motions and verbalizations to engage children and deliver therapeutic interventions based on their needs. We present teleoperation capabilities to support therapists in customizing therapy to their clients' needs. Specifically, we introduce a documentation sidebar that aims to prime therapists using their clients' documented needs, and a session summary report that helps therapists reflect on the session with the child. We present preliminary designs for these capabilities and describe future work to build upon them.

Full text

Robot Teleoperation Interfaces for
Customized Therapy for Autistic Children
Saad Elbeleidy
MIRRORLab
Colorado School of Mines
Golden, CO, USA
selbeleidy@mines.edu
Aryaman Jadhav
MIRRORLab
Colorado School of Mines
Golden, CO, USA
aryamanjadhav@mines.edu
Dan Liu
ATLAS Institute
University of Colorado Boulder
Boulder, CO, USA
dali2731@colorado.edu
Tom Williams
MIRRORLab
Colorado School of Mines
Golden, CO, USA
twilliams@mines.edu
Abstract—Socially Assistive Robots are effective at supporting
autistic children in a variety of different therapies. Therapists
can control the robots’ motions and verbalizations to engage
children and deliver therapeutic interventions based on their
needs. We present teleoperation capabilities to support therapists
in customizing therapy to their clients’ needs. Specifically, we
introduce a documentation sidebar that aims to prime therapists
using their clients’ documented needs, and a session summary
report that helps therapists reflect on the session with the child.
We present preliminary designs for these capabilities and describe
future work to build upon them.
Index Terms—socially assistive robots, teleoperation, teleoper-
ation interface, autism
I. INTRODUCTION
Autistic individuals1may have a variety of different needs
for which they receive therapy or support services. Therapy
must be tailored to the individual’s needs and interests. More-
over, the success of a therapeutic intervention is dependent on
how engaging therapy is to the client [2]. This has resulted
in therapists sometimes adopting unconventional approaches
to therapy such as art therapy [3], [4], music and dance
therapy [5], and robot assisted therapy [6]. Children show
increased interest and engagement when interacting with a
robot [7], [8]. Autistic children, especially, are open to inter-
acting with robots [9]. This has made socially assistive robots
(SARs) a great fit for therapy with autistic children.
When these robots are used in practice, they are often tele-
operated by a therapist [10]. This is likely the same therapist
who would have provided therapy directly to the client were
a robot not used; and the robot is often teleoperated to deliver
the same therapeutic interventions they would normally have
delivered without the robot. While the introduction of robots
has documented benefits as described above, it also comes at
a cost. Therapists report that while therapy is already difficult,
doing so with a robot is even more time consuming.
There is therefore an opportunity to improve teleoperation
interfaces to support therapists in ways that will limit these
costs. Specifically, since therapists need to customize content
for their clients, teleoperation interfaces can present informa-
tion about clients to therapists as a reminder about the goals
1Following guidelines provided by autistic self advocates, we will use
identity-first language when referring to autistic individuals [1]
of a session. At the end of a session, the interface can present
a session report that summarizes the content covered during
a session including any metadata the therapist had entered
previously. In this paper, we present two preliminary designs
for these capabilities, and describe our plans to evaluate them
through human subject experiments.
II. MOT IVATIO N
A. Therapy for Autistic Individuals
Autism is a developmental disability that spans a wide range
of experiences and behaviors [11]. Autistic individuals make
up about 2% of the population [12] and may differ from neu-
rotypical people in the way they communicate, socialize, and
go about their daily life [11]. Autistic traits include, and are not
limited to, exhibiting repetitive (and sometimes self-injurous)
behaviors, preferring to avoid eye contact, and showing interest
in few topics [11]. As such, autistic individuals will often
receive therapeutic services at a young age as arranged by
their parents.
Since autistic individuals do not all have the same thera-
peutic needs, therapy for autistic individuals can vary greatly.
Depending on their specific disabilities, different therapies
may be appropriate. If an autistic individual has a physical dis-
ability, physical or occupational therapy may be appropriate. In
contrast, if they have a speech or language impairment, speech
and language pathology may be appropriate. If the autistic
individual has difficulty communicating and socializing, then
applied behavior analysis may be suggested. Autism does not
have a direct therapy that maps to fit all autistic individuals
since each individual can have varying needs [11]. Addition-
ally, each of these therapies must be carefully customized to
the needs of individual clients [13], [14].
To customize content, therapists can spend a large amount
of time preparing for sessions. They do so by examining
their client’s goals and interests. Using that information they
can prepare content that engages the client and meets their
therapeutic needs [13], [14]. Additionally, therapists must doc-
ument sessions for a variety of reasons. Throughout therapy, a
client’s needs constantly change as they improve their skills or
overcome challenges they are facing. This requires therapists
to keep track of how the client is doing over time and update
their goals. Doing so results in more preparation since therapy

must now change to accommodate the new needs of the client.
Additionally, therapy must stay customized to the client’s
interests to remain engaging.
Research has shown that for therapy to be effective, it must
be engaging [2]. As such, therapists follow guidelines on how
to ensure that engagement is a key part of therapy [15]; and
the use of non-conventional methods such as art therapy [3],
[4] or music and dance therapy [5] to appeal to children
has shown much success, in part due to their ability to
encourage engagement. Robot-assisted therapy has also shown
much success, especially with autistic children, for similar
reasons [6].
B. Socially Assistive Robots
Socially assistive robots (SARs) are robots that provide
assistive services through social interaction [16]. An example
of this is when robots are used to interact with autistic children
in therapy [7], [8], [9], [17]. When used in therapy, these robots
have resulted in increased eye contact by autistic children [18],
[19] likely due to an increase in interest and engagement.
This increased engagement can also lead to more collaboration
between autistic children [20]. When interacting with robots,
autistic children have also increased their verbalizations [21],
[22], [19]. These examples show how SARs can support autis-
tic children in a variety of therapies and increase children’s
engagement with therapeutic content. When these therapies
are delivered without a robot, they are facilitated by a human
therapist.
When SARs are used in practice, they are often teleoperated
by a therapist [10]. Therapists control the robot’s motion
and verbalization to use the robot as the session facilitator.
Children may be more receptive to the robot in that way
since the robot does not present the same power dynamic
that an adult would [23]. Therapists are often in the same
location as their client and the robot while controlling the
robot through the teleoperation interface. However, therapists
are tasked with conducting similar therapies to what they
would conduct without a robot. This requires fairly complex
teleoperation capabilities and preparation. While therapists
already may spend a large amount of time preparing for
therapy and customizing therapeutic content, doing so with
a robot takes significantly more time. This is due to the
fact that therapists need to predict their clients’ responses
in therapy, and prepare and customize content responding to
those predictions in advance of each session.
Research on SARs often focuses on the assistive capabil-
ities of the robot and evaluates the resulting impact on the
assisted individual. However, when SARs are teleoperated,
the individual experiencing the burden of the system is the
teleoperator. The therapist teleoperating the robot is the user of
the teleoperation interface and should therefore be the focus of
attention for teleoperation interface developers. Teleoperation
interfaces should be designed with the operating therapist’s
needs in mind.
There is an opportunity to support therapists in teleoperating
robots by improving their documentation capabilities so that
Fig. 1. An example of the Peerbots teleoperation interface. The center portion
of the screen includes the buttons that, when selected, result in the connected
robot verbalizing the contents of the button. The left sidebar presents a
list of collections of buttons. This section allows a teleoperator to organize
their content and easily navigate between grouped content. The right sidebar
presents additional details about the last selected button and allows the user to
edit its attributes. The bottom section of the screen presents robot connection
and motion control capabilities. ©Peerbots
therapists can more easily customize therapy to their clients.
In this paper, we present preliminary designs for two such
capabilities: (1) incorporating client documentation in the tele-
operation interface and (2) presenting documentation reports at
the end of sessions to summarize a session. We also outline the
research questions we hope to answer through human subjects
experiments to evaluate these designs.
III. TECHNICAL APP ROAC H
A. Robot Teleoperation Interface
For this work we have chosen to build off of existing
teleoperation interfaces. Specifically, we use the Peerbots [24]
application as the teleoperation interface to improve upon since
it is an open source application. The Peerbots application
provides a comparatively low cost solution to SAR teleop-
eration and has also been used in practice in social skills
programs for autistic children [10]. An example of the Peerbots
teleoperation interface is shown in Figure 1.
Peerbots allows a teleoperator to control a robot’s motion
and verbalization in real-time. Ahead of time, a therapist can
author and organize content they plan to have the robot verbal-
ize during a session. Therapists can include useful metadata
for each item verbalized. Importantly, therapists can specify a
goal for the content verbalized as well as the proficiency level
needed. This creates an opportunity for therapists to use this
information after a session to evaluate a client’s performance.
B. Client Documentation Sidebar
As a therapist controls the robot, they are actively selecting
content that is specific to their client’s needs. To support
therapists with the recollection of their clients’ needs, we
propose a documentation sidebar that presents information
about the child that the therapist or their supervisor have
previously entered. By introducing the documentation sidebar,
we aim to answer the following questions:
•Does having built-in documentation capabilities lead to
more documentation by the therapist?

Fig. 2. A preliminary design of the documentation sidebar to include in
teleoperation interfaces.
•Would a therapist check documentation about their client
during a session if it was built in?
•Would a therapist update documentation about their client
during a session if it was built in?
•How does a built-in system compare to current (poten-
tially non-technological) systems?
Our preliminary design of the client documentation sidebar
is shown in Figure 2. This sidebar allows therapists to select
a particular child that they are interacting with during teleop-
eration. This can be done using the dropdown at the top that
has selected John Doe in Figure 2. Upon selecting a specific
client, the documentation sidebar shows the documentation
about that client’s sessions. It allows the teleoperator to enter
content documenting the current session and also includes the
ability to navigate between past sessions to preview earlier
documentation that might be relevant.
C. Documentation Report
As a therapist controls a robot throughout a session, the ther-
apist is able to save session logs locally through the Peerbots
application. These session logs contain information about the
content verbalized by the robot and its metadata; including
timestamp, goal of content, and proficiency of content. The
button information is primarily entered by the therapist using
the robot or a supervising therapist. The goal behind the
documentation report is to present the information provided
by the therapist in a useful way at the end of sessions to
guide them in reflecting about the session and evaluating their
client. We aim to answer the following questions:
•Do reports help teleoperators develop an accurate mental
model of what happened in the session?
•Do therapists think these reports help them at evaluating
their clients?
•Does report accuracy affect teleoperator perception?
•Are visualized reports better at helping therapists notice
inaccuracies in the metadata of the content?
•Does teleoperator perception of the reports and their effi-
cacy differ based on teleoperator’s therapeutic expertise?
Our preliminary design is implemented as a web application
that allows a user to upload their saved logs and view a
summary report of their session. The user can upload their log
file and receive a report with the relevant information as shown
in Figure 3. This web application can also integrate with the
core Peerbots application to display the report directly after
a session is complete. This approach gives users flexibility in
reviewing reports of past sessions as well as the ability to view
session reports upon session completion.
Fig. 3. An example session report.
Importantly, the session report contains information that is
provided by the content author, and the teleoperator of the
robot, both usually therapists when this tool is used with
autistic children. The report aims to share a summary of that
content so that therapists can reflect on the session. The report
begins with some session identification information so the
therapist is clear on which session the report is referencing.
The report includes charts about the proficiency, goals, and
emotions of the content selected. Each button containing con-
tent is labeled by a particular proficiency level and goal. These
graphs present the frequency of each proficiency level and goal
based on what was used in the session. Additionally, each
button results in an emotional expression by the robot. The
emotion chart presents the emotions that the robot expressed
throughout the session and their frequencies.

Below the charts, the report includes several tables. The
first table presents the content collections (palettes) used in the
session and for how long each was used. The second and third
table show the verbalizations that were followed by the longest
robot pauses. These are split into buttons the teleoperator
pressed and speech they typed (”Quick Speech”). This section
may signal to therapists that there are particular verbalizations
that result in a long pause until the next interaction. A box
plot is also shown to visualize this information, depicting
the distribution of intra-verbalization pauses. The outliers
in this plot (i.e., the items shown in the preceding tables)
can be clicked for more information. Finally, at the bottom
of the report, the therapist can see every verbalization in
chronological order with all the verbalizations’ attributes such
as proficiency, goal, and emotion.
IV. CONCLUSION
In this paper, we proposed preliminary designs of two
teleoperation interface capabilities to support therapists in
customizing therapy for their clients when using a robot
during therapy. We also presented specific research questions
to answer regarding each of these designs.
In future work, we plan on running several experiments with
human subjects to answer our research questions. We plan on
running different versions of these experiments with therapists
and non-therapists to account for and understand the effect of
therapeutic expertise on the usage of the new features. These
experiments will be used to determine whether the newly
designed product is in fact helpful at supporting therapists,
and may allow us to make generalizable recommendations to
the wider audience of SAR teleoperation interface developers.
REFERENCES
[1] K. Bottema-Beutel, S. K. Kapp, J. N. Lester, N. J. Sasson, and B. N.
Hand, “Avoiding ableist language: Suggestions for autism researchers,”
Autism in Adulthood, vol. 3, no. 1, pp. 18–29, 2021.
[2] K. Barish, “What is therapeutic in child therapy? i. therapeutic engage-
ment.” Psychoanalytic psychology, vol. 21, no. 3, p. 385, 2004.
[3] M. J. Emery, “Art therapy as an intervention for autism,” Art therapy,
vol. 21, no. 3, pp. 143–147, 2004.
[4] N. Martin, “Art therapy and autism: Overview and recommendations,”
Art Therapy, vol. 26, no. 4, pp. 187–190, 2009.
[5] H. Takahashi, K. Matsushima, and T. Kato, “The effectiveness of
dance/movement therapy interventions for autism spectrum disorder: a
systematic review,” American Journal of Dance Therapy, vol. 41, no. 1,
pp. 55–74, 2019.
[6] B. Scassellati, H. Admoni, and M. Matari´
c, “Robots for use in autism
research,” Annual review of biomedical engineering, vol. 14, 2012.
[7] K. Kabaci´
nska, T. J. Prescott, and J. M. Robillard, “Socially assistive
robots as mental health interventions for children: A scoping review,”
International Journal of Social Robotics, pp. 1–17, 2020.
[8] J. Dawe, C. Sutherland, A. Barco, and E. Broadbent, “Can social robots
help children in healthcare contexts? a scoping review,” BMJ paediatrics
open, vol. 3, no. 1, 2019.
[9] B. Robins, K. Dautenhahn, R. Te Boekhorst, and A. Billard, “Effects
of repeated exposure to a humanoid robot on children with autism,” in
Designing a more inclusive world. Springer, 2004, pp. 225–236.
[10] S. Elbeleidy, D. Rosen, D. Liu, A. Shick, and T. Williams, “Analyzing
teleoperation interface usage of robots in therapy for children with
autism,” in Proceedings of the ACM Interaction Design and Children
Conference, 2021.
[11] A. S. A. Network and L. Berry, Welcome to the
Autistic Community. Autistic Press, 2020. [Online]. Available:
https://books.google.com/books?id=bzRszQEACAAJ
[12] M. J. Maenner, K. A. Shaw, J. Baio et al., “Prevalence of autism spec-
trum disorder among children aged 8 years—autism and developmental
disabilities monitoring network, 11 sites, united states, 2016,” MMWR
Surveillance Summaries, vol. 69, no. 4, p. 1, 2020.
[13] D. M. Petrescu, “One size doesn’t fit all: An inclusive art therapy
approach for communication augmentation and emotion control in
children with autism,” 2013.
[14] S. Elbeleidy, “Towards effective robot-teleoperation in therapy for chil-
dren with autism,” in Interaction Design and Children, 2021, pp. 633–
636.
[15] R. Chasin and T. B. White, “The child in family therapy: Guidelines for
active engagement across the age span.” 1989.
[16] D. Feil-Seifer and M. J. Mataric, “Defining socially assistive robotics,” in
9th International Conference on Rehabilitation Robotics, 2005. ICORR
2005. IEEE, 2005, pp. 465–468.
[17] T. Chaminade, D. Da Fonseca, D. Rosset, E. Lutcher, G. Cheng, and
C. Deruelle, “Fmri study of young adults with autism interacting with a
humanoid robot,” in 2012 IEEE RO-MAN: The 21st IEEE International
Symposium on Robot and Human Interactive Communication. IEEE,
2012, pp. 380–385.
[18] O. Damm, K. Malchus, P. Jaecks, S. Krach, F. Paulus, M. Naber,
A. Jansen, I. Kamp-Becker, W. Einhaeuser-Treyer, P. Stenneken et al.,
“Different gaze behavior in human-robot interaction in asperger’s syn-
drome: An eye-tracking study,” in 2013 IEEE RO-MAN. IEEE, 2013,
pp. 368–369.
[19] E. Y.-h. Chung, “Robotic intervention program for enhancement of social
engagement among children with autism spectrum disorder,” Journal of
Developmental and Physical Disabilities, vol. 31, no. 4, pp. 419–434,
2019.
[20] J. Wainer, E. Ferrari, K. Dautenhahn, and B. Robins, “The effectiveness
of using a robotics class to foster collaboration among groups of
children with autism in an exploratory study,” Personal and Ubiquitous
Computing, vol. 14, no. 5, pp. 445–455, 2010.
[21] I. Giannopulu, “Multimodal cognitive nonverbal and verbal interactions:
the neurorehabilitation of autistic children via mobile toy robots,” IARIA
International Journal of Advances in Life Sciences, vol. 5, 2013.
[22] E. S. Kim, L. D. Berkovits, E. P. Bernier, D. Leyzberg, F. Shic, R. Paul,
and B. Scassellati, “Social robots as embedded reinforcers of social
behavior in children with autism,” Journal of autism and developmental
disorders, vol. 43, no. 5, pp. 1038–1049, 2013.
[23] J.-J. Cabibihan, H. Javed, M. Ang, and S. M. Aljunied, “Why robots?
a survey on the roles and benefits of social robots in the therapy of
children with autism,” International journal of social robotics, vol. 5,
no. 4, pp. 593–618, 2013.
[24] PEERbots. (2021) Peerbots. [Online]. Available: https://peerbots.org