A Mobile Game System for Improving
the Speech Therapy Experience
Speech Processing; Speech Therapy; Human Computer
Interaction; Games; Mobile Applications
ACM Classification Keywords
H. Information Systems; H.5. Information interfaces
and presentation (e.g., HCI); H.5.2. Voice I/O; I/O J.3.
Life and Medical Sciences: Health
A lack of intrinsic motivation to practice speech is
attributed to tedious and repetitive speech curriculums,
but mobile games have been widely recognized as a
valid motivator for jaded individuals. SpokeIt is an
interactive storybook style speech therapy game that
intends to turn practicing speech into a motivating and
productive experience for individuals with speech
impairments as well as provide speech therapists an
important diagnostic tool. In this paper, I discuss the
novel intellectual contributions SpokeIt can provide
such as an offline critical conversational speech
recognition system, and the application of therapy
curriculums to mobile platforms, I present conducted
research, and consider exciting future work and
SpokeIt, shown in Figure 1, is a large collaborative
interdisciplinary project that incorporates the medical
expertise from UC Davis’s medical team, developmental
psychologists from UCSC, and my team of researchers
in the Interactive Systems for Individuals with Special
Needs Lab at UCSC . We hope to create a solution
that speech language pathologists (SLPs) can use with
their patients and offer the following functionality:
A diagnostic tool that assists SLPs in listening to
the patient's speech
Permission to make digital or hard copies of part or all of this work for
personal or classroom use is granted without fee provided that copies are
not made or distributed for profit or commercial advantage and that
copies bear this notice and the full citation on the first page. Copyrights
or third-party components of this work must be honored. For all other
uses, contact the Owner/Author.
Copyright is held by the owner/author(s).
MobileHCI '17, September 04-07, 2017, Vienna, Austria
University of California Santa Cruz
Santa Cruz, CA 95060, USA
A critical offline speech recognition system that
accurately scores speech, responds accordingly,
and records progress data made accessible to the
A dynamic curriculum that adjusts practice session
difficulty according to performance
A patient profiling system that assigns words and
phrases to target each individual’s unique speech
Research Goals, Questions, and Approach
The repetitive nature of speech therapy curriculums
often leads to unmotivated individuals, but games have
been shown to make repetition a fun and necessary
component required for progress in the game . To
make a speech therapy game we first were interested
in validating that we could employ an offline accurate
real-time speech recognition system that was capable
of being used as game input . Now that we have
one, we have many research questions:
What style of game is most motivating for
individuals with speech impairments?
Can we verify that our solution scores speech
accurately enough for use as a diagnostic tool?
How do we incorporate therapeutic input
mechanics that induce flow and immersion rather
than distract from gameplay?
Can we verify that our solution leads to improved
Can we improve the efficiency of time spent in-
office with SLPs?
Can SpokeIt provide insight into phonological
versus phonetic errors?
Can procedural content generation effectively
produce content for as long as therapy is required?
Can our games be abstracted in a way that allows
for other sensory inputs and mechanics to be
employed for other types of therapy?
The number of novel contributions SpokeIt is capable of
producing is exciting. As development continues and
high fidelity prototypes are iterated, we actively employ
user-centered design by including individuals with
speech impairments as well as SLPs and our
interdisciplinary team in the design of SpokeIt. We
incorporate feedback from experts and employ
practices from credible contributions to the field. We
test our speech therapy solutions on a diverse range of
individuals with speech impairments in both short and
long term studies. We regularly meet within our
interdisciplinary team to collaborate and ensure that
everyone’s research questions are being appropriately
Studies, Conducted Research, and Results
Our lab has produced multiple iterations of speech
therapy games, shown in Figure 2, 3, and 4, that have
resulted in five publications, and three games
[1,2,3,4,5]. SpokeIt is our newest iteration and has
recently been highlighted in an accepted publication to
IDC . SpokeIt was chosen to be presented in the
Three-Minute Thesis Challenge, Grad Slam, at UCSC.
In the past year, SpokeIt has been tested in user
studies on four children with cleft speech at the UC
Davis medical center, eight individuals with varying
physical and mental disabilities at Hope Services, and
three Stoke Survivors at Cabrillo College Stroke Center.
We will soon be submitting to the TAACCESS journal.
The studies have yielded extremely valuable insights
into making games usable and accessible for individuals
with different physical and mental disabilities. We have
Figure 1: SpokeIt
Figure 2: Speech with Sam
Figure 3: Speech Adventure
learned that storybook style games are the preferred
style for practicing speech and have validated that our
games motivate practice more than existing
curriculums . We have designed our characters,
shown in Figure 5, and plot to be as engaging as
possible, which has been validated by users in our
Remaining Research Efforts
There is massive potential for research opportunities as
we continue to develop SpokeIt. We are currently
building a sandbox level creator so researchers with no
coding experience can build levels for SpokeIt. We are
iterating on our procedural art generator, shown in
Figure 6, that utilizes evolutionary algorithms to learn
how to make game art. We are developing a better
data visualization and scoring system for reporting
speech data. We hope to design a narrative generator
that will allow SpokeIt to be available indefinitely to
accommodate different speech therapy lengths. We
hope to learn what other therapy applications SpokeIt
can be used for.
We plan to continue to use user-centered design
throughout our development, but once development
has ended, we hope to run long term studies that
answer the questions listed above. We believe that
SpokeIt will motivate individuals to practice speech
more than conventional methods, will make in-office
visits with SLPs more efficient, can be applied to other
forms of therapy, and produce guidelines for designing
therapeutic games that employ procedural content
1. Duval, J. in press. Designing Towards Maximum
Motivation and Engagement in an Interactive
Speech Therapy Game. ACM.
2. Rubin, Z. 2017. Development and evaluation of
software tools for speech therapy (Doctoral
dissertation, University of California, Santa Cruz).
3. Rubin, Z., & Kurniawan, S. 2013. Speech
adventure: using speech recognition for cleft
speech therapy. In Proceedings of the 6th
International Conference on PErvasive Technologies
Related to Assistive Environments (p. 35). ACM.
4. Rubin, Z., Kurniawan, S., & Tollefson, T. 2014.
Results from Using Automatic Speech Recognition
in Cleft Speech Therapy with Children. In
International Conference on Computers for
Handicapped Persons (pp. 283-286). Springer
5. Rubin, Z., Kurniawan, S., Gotfrid, T., & Pugliese, A.
2016. Motivating Individuals with Spastic Cerebral
Palsy to Speak Using Mobile Speech Recognition. In
Proceedings of the 18th International ACM
SIGACCESS Conference on Computers and
Accessibility (pp. 325-326). ACM.
6. Sauvé, L. 2010. Effective educational games.
Educational gameplay and simulation
environments: Case studies and lessons learned,
7. Sri Kurniawan, Su-hua Wang, Christina Roth, Travis
Tollefson. 2016. CHS: Small: Game for Cleft
Speech Therapy (No. 1617253). National Science
Figure 4: SpokeIt Prototype
Figure 5: SpokeIt race of
characters known as Migs
Figure 6: Procedural Art