Content uploaded by Douglas Brooks
Author content
All content in this area was uploaded by Douglas Brooks on May 21, 2015
Content may be subject to copyright.
Engaging Children with Severe Physical Disabilities via
Teleoperated Control of a Robot Piano Player
Miyako Jones, Terrence Trapp, Naquasia Jones, Douglas Brooks∗, and Ayanna M.
Howard
Georgia Institute of Technology
School of Electrical and Computer Engineering
777 Atlantic Drive
Atlanta, Georgia
miyako@ece.gatech.edu, terrence@ece.gatech.edu, naqusia@ece.gatech.edu, douglas.brooks@gatech.edu∗,
ayanna.howard@ece.gatech.edu
ABSTRACT
This research focuses on investigating communication tech-
nologies that can enable interaction between robot play-
mates and children with severe physical disabilities. Many
children with physical disabilites such as cerebral palsy have
difficulty using their hands for tasks. However, toys have a
significant impact on a child’s cognitive, social, and physical
development. This paper explores the possibility of using
the Boardmaker R
Plus! special needs educational software
to develop an interface to provide alternative access to toys.
As such, a piano playing robotic playmate is used as a device
that engages children with physical disabilites into learning
and communicating their emotions.
Categories and Subject Descriptors
J.2 [Physical Sciences and Engineering]: Engineering
General Terms
Developmental Delay, Robotic Platform
Keywords
Cerebral Palsy, Autism, Boardmaker, Lynxmotion
1. INTRODUCTION
Cerebral Palsy is a growing concern in the U.S. The Center
for Disease Control reports that cerebral palsy is prevalent
1 in 303 children in the U.S. and that an estimated 800,000
people in the U.S. have cerebral palsy [8]. Of these children,
70 to 80 percent are affected by spasticity [11], an invol-
untary muscle tightness that occurs due to discoordination
of their neural motor control and stiffness of their joints.
The resulting outcome is that children with cerebral palsy
∗Corresponding Author
12th International ACM SIGACCESS Conference on Com-
puters and Accessibility October 25-27, 2010 Orlando, FL, USA
have difficulty in performing traditional manipulation tasks,
such as those required to interact with toys. Yet, interactive
play with physical toys has an important role in the de-
velopment of cognitive, physical, and social development in
children. For children with severe physical disabilities, tele-
operated robots have been shown to enable achievement of
play-related tasks that go beyond their manipulation capa-
bilities [2,9, 15]. Robotic toys have also been shown to aid in
early intervention for children with development delays and
to engage children in imitation base play [7, 16]. The pri-
mary limitation to date is that interaction between children
and robot is limited both by the capability of the robot as
well as the interfaces used to communicate.
Since the effect of play has shown to have a lasting effect due
to the dynamic nature of interacting with the world [12],
our goal is to enable shared play that continuously enter-
tains, thus engaging the mind, and creating opportunities
for extended play over longer durations. As such, our pri-
mary focus is on designing robotic playmates capable of en-
gaging children in therapeutic and educational play. Our
first efforts in this area have focused on developing methods
to learn acceptable play behavior by observing others play.
These interpretations of basic play movements are then used
to command a robot to interact, in the same way as the ob-
served child, with common toy items. Currently, this is a
one-way interaction – i.e. the human plays and the robot re-
peats. If dealing with able-bodied children, the next logical
step is to develop methods to allow robot and human play
together by interleaving play actions. However, regarding
children with severe physical disabilities, we must still bridge
the gap between tele-operated and autonomous robots in or-
der to enable interaction with the robot playmate.
For children with severe physical disabilities, augmentative
communication devices enable children to interact through
their environment. Currently adopted devices such as single
switch machines allow children to select pictures, letter, or
words on the interface through a row-column scanning pro-
cess that is initiated by a single click of the switch. These de-
vices, although limited in their communication speed, have
been extensively used to enable children with cerebral palsy
to communicate with their caregivers, family, and friends.
The goal of the project is therefore to investigate communi-
cation technologies (both existing and new) that can enable
(a) Cover page. (b) Introduction page.
Figure 1: Illustrations of Boardmaker R
Plus!
interaction between robot playmates and children with se-
vere physical disabilities.
Section 2 discusses the background and motivation for this
work; here, we specifically discuss the importance of play
in child cognitive development coupled with information re-
garding early intervention programs to aid with develop-
ment. Section 3 discusses our approach to designing a novel
toy to aid in the cognitive and physical development of chil-
dren with cerebral palsy. Section 4 gives the results of this
work, while Sections 5 and 6 discuss the future work and
concludes the article, respectively.
2. BACKGROUND AND MOTIVATION
2.1 The Importance of Play
The role of play in the development of children is an im-
portant aspect, which has been explored in a large body of
work. Piaget’s book “Play, dreams, and imitation in child-
hood” is one of the earlier references showing the importance
of play [12]. This work suggests that play is useful for a vari-
ety of reasons, including helping to develop motor skills and
spatial abilities.
Some anecdotes on the toys that are appropriate for various
ages of children are presented in [5]. For infants, toys such
as brightly colored blocks, rattles, and baby puzzles are sug-
gested. Toys that require pushing, pulling, rolling, lifting,
and other physical activities are recommended for children
old enough to walk. According to [6], preschool aged chil-
dren spend about 20% of their time playing with toys and
13-16% doing non-play activities such as watching tv, read-
ing, listening to music, and helping with chores.
2.2 Early Intervention Programs
Early intervention programs are government sponsored pro-
grams that help ensure that children have the opportunities
they need to have successful development and education. An
overview of early intervention programs is given in [13]. The
rationale for these programs is that children’s early experi-
ences have been shown to be important to their cognitive
development. There is also evidence that children can make
progress on developing skills in which they are lacking if
given the proper intervention at an early stage. Studies of
the effectiveness of early intervention programs are also sum-
marized in [13], which shows that these programs seem to
have positive effects on children’s cognitive development.
2.3 Children with Developmental Delays
Studies involving therapeutic play between robots and chil-
dren with pervasive developmental disorders such as autism
have been of particular interest for several reasons. Autism
can refer to a wide range of disorders. The main feature
of autism is difficulty with social and communication skills,
such as relating to others and/or having appropriate social
reactions. The use of robots for play in children with autism
has been of special interest for several reasons. First, the
current accepted way to teach autistic children involves use
of repetition [3]. Robots are well suited to perform consis-
tent, repetitive actions. Also, it has been shown that chil-
dren with autism find robots quite engaging and respond fa-
vorably to social interactions with them, even when the chil-
dren typically do not respond socially with humans [4,10,14].
Another developmental disorder that is of particular interest
is cerebral palsy. In much of the same way that robotics are
capable of engaging children diagnosed with autism, robots
can be used to engage children with cerebral palsy in phys-
ical activity that will aid in increasing motor functionality.
Unfortunately, access to necessary assistive technology re-
mains unequal and persons with severe or multiple physical
disabilities are largely overlooked [1]. Hence the prevelance
for continued research efforts towards benefiting individuals
with developmental disorders.
3. APPROACH
3.1 BoardmakerR
Plus!
Boardmaker R
is a computer program for educators intro-
duced in 1990 that creates printed educational and AAC
materials. Boardmaker R
Plus!, shown in Fig.1, takes the
original idea and adapts it for onscreen use. All versions
of the software use Mayer-Johnson’s Picture Communica-
tion Symbols (PCS), which were designed for students with
(a) Piano display board with key scanning group
circled. (b) A portion of the batch file used to send
commands to the robot.
Figure 2: Illustrations of the piano display and batch file.
special needs in 1981.
Within BoardmakerR
Plus!, buttons are created to allow
user interaction with a board after it is put in “use mode”.
A user can choose the size, color, and border of a button;
and add text in various font faces, colors, sizes, and styles.
Images or PCS can also be added. Buttons can also have
action lists, which greatly increase the quality of the user
experience. A button can change a setting, load another
board, activate a video or play a sound, type messages, and
launch external applications. For example, a board could
be used as an alternative desktop with buttons acting as
shortcuts to frequently-used programs. The buttons can also
be configured to store variables, launch macros, or activate
only if certain conditions have been met. The text features
use buttons to act like an onscreen keyboard for AAC users.
It can also predict the words a user intends to type to aid
the process.
Boardmaker R
Plus! offers methods of access for a variety of
users. There is touchscreen access; head mouse access; auto-
, step-, and inverse scanning access for use with switches (all
of which use the row/column method); and joystick access.
Standard mouse buttons and the keyboard arrow keys are
supported as well. All options can be configured for individ-
ual needs. As there are so many options, Boardmaker can
be used to create an interface for just about anything. The
built-in control options for assistive devices allow physically
disabled users to interact with it independently. A button
can launch any program by loading it directly or by using
a document to load it indirectly. The interface created for
the project demonstrates Boardmaker’s ability to facilitate
access to non-AT devices.
3.2 The Interface
The interface has three main boards. The first board, shown
in Fig.2a.,contains a graphical representation of a piano with
four primary “keys” (blue, green, yellow, and orange) and
a button for each double-key note that is feasible by the
robotic manipulator (blue-green, green-yellow, and yellow-
orange). These keys are in a scanning group. The scan will
loop indefinitely through a group in a set pattern as long
as the user makes at least one selection. If no selection has
been made, the scan will loop as many times as indicated
in the scanning options before exiting the group. Upon se-
lecting one of the keys, Boardmaker will load a specific DOS
batch file, shown in Fig.2b., and pass an argument to it. The
argument will then be used to select a particular sequence
of commands to send to a robot, discussed in Section 3.3,
connected to the computer that will play a physical toy pi-
ano. For example, selecting the blue key will open the batch
file and pass the argument “blue-key” to it. The “blue-key”
commands will then be located and “echoed” to the robot,
which will then hit the blue key on the piano. On the bot-
tom of the board are two buttons that belong to a “board
navigation” scanning group: one to advance to the first song
board and one to quit the program. By default, the program
loops indefinitely through a top-level group that contains the
piano key group and the board navigation group.
The third and fourth boards, shown in Fig.3, contain songs
that can be played using the seven notes of the piano. Three
of them are classic children’s songs (Mary had a Little Lamb;
Twinkle, Twinkle Little Star ; and Row, Row, Row Your
Boat), while the fourth is an original composition. The name
of each song is placed on an invisible button so that the user
can select the song by selecting its name. Doing this will
also instruct Boardmaker to load the batch file and pass an
argument to it. The robot will be instructed to hit the se-
ries of keys that make up the tune as illustrated on the two
song boards. Beneath each song title is the sequence of keys
(a) First song board with song button circled. (b) Second song board with navigation scanning group cir-
cled.
Figure 3: Illustrations of the third and fourth boards, which contain songs.
that make up the tune and the lyrics (if applicable). The
navigation scanning group at the bottom of each song board
includes a button to change to the second song board, a but-
ton to change to the piano key board, and a quit program
button.
It was important to consider the layout of the board while
placing items. Similar buttons should be grouped together.
If using multiple boards, buttons should be consistently placed
in order to assist the user with locating selections. Stan-
dard visual design rules also apply. Colors should have high
contrast and low intensity. There should also be adequate
“negative space” between each element.
Using a switch for access is often as easy as plugging it into
the computer through a switch interface. Switches use a
1/8 inch plug like the one used to plug a pair of headphones
into an MP3 player or speakers into a computer. However,
a switch will not work if plugged directly into a computer.
A switch interface is needed in order to connect the two
together by providing a port for the switch and an input for a
USB, serial, or PS/2 port on the computer. Switch interfaces
can connect one switch to the computer or multiple switches
simultaneously.
The QuizWorks USB switch interface used for the project
allows up to four switches to be connected at once and has
templates to enable the emulation of different keyboard keys
or mouse buttons. Two different switches were tested – a
traditional wired one by QuizWorks and a wireless one by
AbleNet that used a wired receiver. The wired one was
more accurate and more responsive. This does not prove
that wired switches are always better as outside radio inter-
ference or a malfunction with the switch itself could have
affected its performance. The wired switch was rectangu-
lar, and only the end opposite the connection cable could
be pressed. The base of the switch was also only about a
quarter inch in height. The wireless switch was circular and
the entire switch could be pressed. The base was about five
inches high. Both the switch and receiver required two AA
batteries for operation, and each had an on/off switch.
3.3 Robotic Platform
The robotic platform that was chosen for this project is a
Lynxmotion AL5D robotic arm. The robotic arm has 4-
DOF, which are controlled by five different servos numbered
zero through four. Servos 0-4, control the base, shoulder,
elbow, wrist, and gripper, respectively. To program the
robotic arm, we first used the Lynxmotion RIOS SSC-32
software. The initial step was to configure the robotic arm’s
range of motion and home position. In the configuration
window, we were able to set the maximum and minimum
positions, limiting the arm’s movement. This made pro-
gramming the arm to play the piano simpler, as the required
range of motion was defined. Once the range of motion was
set, the configurations could be used at any time.
After the robotic arm was configured, we proceeded with
programming it to play the piano. By pressing the ‘moves’
button on the start up menu, we were able to track and store
arm positions in either cartesian coordinates, distance to a
goal location, or joint angles. We found it simpliest to move
and store the arm positions by joint angles.
We created seven projects, each named according to what
keys on the piano were pressed (blue, yellow, green, orange,
blue-green, green-yellow, and yellow-orange). After all keys
were successfully programmed, we connected the projects
to the Boardmaker program via the batch files that were
mentioned in Section 3.2. The experimental setup is shown
in Fig.4.
(a) Top view. (b) Side view.
Figure 4: Top and side views of the robot piano-playing playmate used for this research.
4. RESULTS
Compensating for potentially large and variable time delays
in robotic devices needs further work [2]. Specifically when
working with children during play activites (both neurotypi-
cal and non-neurotypical), it is vital to ensure a continuance
of interaction with minimal delay. Therefore, speed for song
completion was considered to be of the uttermost impor-
tance.
To send commands to the robotic arm using the batch file,
the command begins with “echo”and is followed by the servo
number and position. The initial programming method was
to send one command at a time. This worked well when
playing the keys, but did not work well when programming
the arm to play songs. Since there was a pause after each
command was sent, the songs were played very slowly re-
sulting unrecognizable songs. To speed up the flow of the
songs, the second attempt consisted of sending multiple com-
mands simultaneously. As a result, the pauses in between
each movement was removed, and the keystrokes were much
quicker.
The robotic platform received data in eight bit segments,
with an added bit for termination, and a baud rate of 115200
Bd. On average, it took approximately 0.727s for the robotic
platform to receive a signal from the computer and strike the
appropriate key using the scanning protocol. Also, on av-
erage, it took approximately 22s to execute each song men-
tioned in Section 3.2. Given the short amount of time that
it takes for the robotic arm to strike a key and execute a
song, short sessions (<5 minutes) of child-robot interaction
are feasible. As a comparison, the designers of Tito, a tele-
operated robot mediator designed to interact with autistic
children, conducted successful studies that consisted of 22
exposures, 5 minute cases, 3 times/week over 7 weeks [10].
The success of the work with Tito is one of many illustra-
tions that short segments of robotic interaction with children
with developmental delays can prove to be beneficial.
We are optimistic that with further research, patients with
cerebral palsy, and other developmental delays, will be able
to improve their communication skills and physical stability.
Specifically, we are confident that when the Boardmaker, the
Lynxmotion AL5D robotic arm, and the interface are used
together, children with cerebral palsy will be more respon-
sive and show a greater improvement in their communica-
tion skills while in the classroom and when engaged in other
activities such as physical therapy.
5. FUTURE WORK
Batch file programming was required to make the robot re-
spond to the commands given to it by the Boardmaker in-
terface. It is not possible to command it to do something
not defined on one of the boards without editing the batch
file. A future version of assistive control software should al-
low device customization within the program itself so that
it will be easier to make behavioral modifications. In the
meantime, programming a robot to respond to general com-
mands such as “left” and “right” when using the Boardmaker
interface would give a child with cerebral palsy significant
control over their play time.
In the immediate future, we intend to program the software
so that children can record their own songs. We are cur-
rently investigating various ways to interact with the robot
using Matlab and serial data. A new board has to be cre-
ated in which a “Record” button will initiate a batch file.
Although, the new batch file created can be called using the
new board in BoardMaker, the position recorded by Mat-
lab might not be the same as the one BoardMaker typically
reads. Therefore, there will be some decoding required to
make this interaction feasible. One option is to let Mat-
lab write the information from the previous section in the
text file and then transfer the details to a batch file. Using
‘textread’ and ‘fprintf’ we can read the data and store it in
terms that BoardMaker will understand.
6. CONCLUSIONS
Computer software is usually not adapted for alternative ac-
cess methods and those that are often restrict the user to
one type of activity. BoardmakerR
Plus! is a different type
of special needs software. It allows the user to launch appli-
cations externally, giving a switch user access to programs
that were previously unavailable. Through Boardmaker, it
is also possible for the user to access a nonadaptive device
when it is connected to the computer.
Boardmaker’s ability to process variables and conditional
statements make it a viable platform for assistive software
development. In addition, alternative access methods are
built in. One significant disadvantage to this method is be-
ing required to install Boardmaker and have the CD in the
drive in order to use a board. For developers who need a
stand-alone program, Boardmaker could be used as a start-
ing point.
7. REFERENCES
[1] D. Brooks and A. Howard. Quantifying Upper-Arm
Rehabilitation Metrics for Children through
Interaction with a Humanoid Robot. Applied Bionics
and Biomechanics, 2010 (submitted).
[2] C. Carignan and H. Krebs. Telerehabilitation robotics:
Bright lights, big future? Journal of Rehabilitation
Research and Development, 43(5):695, 2006.
[3] K. Dautenhahn. Robots as social actors: Aurora and
the case of autism. In Proc. CT99, The Third
International Cognitive Technology Conference,
August, San Francisco, pages 359–374, 1999.
[4] K. Dautenhahn and A. Billard. Games children with
autism can play with robota, a humanoid robotic doll.
Universal Access and Assistive Technology, pages
179–190, 2002.
[5] M. Erxleben. As they play. The American Journal of
Nursing, 34(12):1144–1146, December 1934.
[6] M. Giddings and C. Halverson. Young children’s use of
toys in home environments. Family Relations,
30(1):69–74, January 1981.
[7] A. Howard, S. Remy, C. Park, H. Park, and
D. Brooks. The Path to Autonomous Robots,
chapter 5. Springer, November 2008.
[8] Institution. Center for Disease Control and Prevention.
[9] R. Mahoney, H. Van der Loos, P. Lum, and C. Burgar.
Robotic stroke therapy assistant. Robotica,
21(01):33–44, 2003.
[10] F. Michaud, T. Salter, A. Duquette, H. Mercier,
M. Lauria, H. Larouche, and F. Larose. Assistive
technologies and child-robot interaction. In AAAI
Spring Symposium on Multidisciplinary Collaboration
for Socially Assistive Robotics, 2007.
[11] N. I. of Neurological Disorders and Stroke. Cerebral
Palsy: Hope Through Research. NIH Publication
Number 06-159, updated 7/13/07.
[12] J. Piaget. Play, dreams and imitation in childhood.
Routledge, 1999.
[13] C. Ramey and S. Ramey. Early intervention and early
experience. American psychologist, 53(2):109–120,
1998.
[14] B. Scassellati. How social robots will help us to
diagnose, treat, and understand autism. Robotics
Research, pages 552–563, 2007.
[15] J. Speich and J. Rosen. Medical robotics. Encyclopedia
of Biomaterials and Biomedical Engineering: LZ, page
983, 2004.
[16] A. Trevor, H. Park, and C. Kemp. Playing with Toys:
Towards Autonomous Robot Manipulation for
Therapeutic Play. In IEEE Int. Conf. on Robotics and
Automation, May 2009.
