Content uploaded by Jared Duval
Author content
All content in this area was uploaded by Jared Duval on Dec 02, 2018
Content may be subject to copyright.
SpokeIt: A Co-Created Speech Therapy
Experience
Abstract
We present SpokeIt, a novel speech therapy game co-
created with users, medical experts, speech
pathologists, developmental psychologists, and game
designers. Our serious game for health aims is to
augment and support in-office speech therapy sessions
with an at-home supported speech therapy experience.
SpokeIt was co-created using participatory methods
involving multiple stakeholders and target users. We
describe the technical details of SpokeIt, the process of
working with multiple stakeholders, and the methods
that allowed us to create medium fidelity prototypes in
real-time with players. We share our emerging designs,
tools, insights for the broader CHI audience, and plans
for future work.
Author Keywords
SpokeIt; Speech Therapy; Serious Game for Health;
Participatory Design
ACM Classification Keywords
H. Information Systems; H.5. Information interfaces
and presentation (e.g., HCI); H.5.2. Voice I/O; I/O J.3.
Life and Medical Sciences: Health
Introduction
Without a trained speech therapist at home, there is an
opportunity to support and augment the speech
Permission to make digital or hard copies of part or all of this work for
personal or classroom use is granted without fee provided that copies are
not made or distributed for profit or commercial advantage and that
copies bear this notice and the full citation on the first page. Copyrights
for third-party components of this work must be honored. For all other
uses, contact the Owner/Author.
CHI'18 Extended Abstracts, April 21–26, 2018, Montreal, QC, Canada
© 2018 Copyright is held by the owner/author(s).
ACM ISBN 978-1-4503-5621-3/18/04.
https://doi.org/10.1145/3170427.3186494
Jared Scott Duval
University of California Santa Cruz
Santa Cruz, CA 95060, USA
JDuval@ucsc.edu
Elena Márquez Segura
University of California Santa Cruz
Santa Cruz, CA 95060, USA
Elena.Marquez@ucsc.edu
Sri Kurniawan
University of California Santa Cruz
Santa Cruz, CA 95060, USA
skurnia@ucsc.edu
CHI 2018 Demonstration
CHI 2018, April 21–26, 2018, Montréal, QC, Canada
D501, Page 1
therapy experience outside of the office. Our interviews
with Speech Language Pathologists (SLPs) show that
at-home curriculums are tedious and often forgotten.
SLPs recommend patients practice speech for 10
minutes per day, but suspect little to no time is spent
practicing out of the office, and as a result, much of in-
office time is spent reestablishing patient’s abilities.
SLPs have no record of time spent practicing at home
or the progress that is being made outside of the office.
A serious game for health has the potential to support
speech therapy at home, provide feedback to the
player, and supply SLPs with new information, making
in-office sessions more efficient.
In this document, we first provide the technical details
that power our critical speech recognition system. Next,
we show how multiple stakeholders and users co-
created and designed SpokeIt. We share our emerging
designs and tools used, and conclude with a brief
discussion and future work.
Related Work
According to SLPs we interviewed, the extrinsic
motivation to improve speech and the encouragement
of traditional therapy supporters is not always enough
to keep patients practicing. Literature has shown that
games are an effective educational intervention and
medium to convey and support feelings of self-efficacy
due to the immersive and pervasive virtual
environment, and that it had been shown to work
effectively as educational interventions [4]. Games
have demonstrated the ability to teach while providing
a motivating and interactive environment [10] are
effective with children [7] and can be as effective as
face-to-face instruction [6]. PD can empower people
with sensory, cognitive, or social impairments to
actively take part in the design process of a system [1].
Technical Details
The offline real-time critical speech recognition system
is built using RapidEars, an extension of Carnegie
Mellon’s Pocketsphinx [5]. We have modified their
system to listen for both correct and incorrect
utterances of speech. SpokeIt uses speech input as the
principal mechanic to make progress in the game,
which includes single-word targets and repetition of
longer phrases. Balancing offline speech recognition
and a game engine is challenging considering the
limited processing power of mobile devices.
Predecessors of SpokeIt [2,3] that use similar
technology include Rubin’s Speech Adventure, and
Speech with Sam [8,9]. We use Apple’s SpriteKit and
Swift 3, native to IOS devices, and RapidEars, written
in Objective C, for the development of our game.
The Co-Creation of SpokeIt
Medical experts and SLPs provided the initial functional
requirements for the project. Namely, the game must
run on iPads, must not require an internet connection,
and critically listen to speech. Our players are often in
lower socioeconomic homes and do not have Wi-Fi or
regular access to a speech therapist. The clinic we are
working with provides patients iPads with the software
installed. We worked with developmental psychologists
and game designers to create an overarching plot and
characters that are developmentally appropriate for
children and adults with developmental disabilities.
Plot and Characters
Our story follows Nova, a star powered by speech
energy, who has fallen from the sky and is tasked with
Figure 1 Adobe Character
Animator. To the top, Nova
demonstrating /P and /L
phenomes. To the bottom, Migs
expressing exaggerated
emotions.
CHI 2018 Demonstration
CHI 2018, April 21–26, 2018, Montréal, QC, Canada
D501, Page 2
helping the Migs, a race of speechless characters,
return color to their world. The player must use the
power of their voice to charge Nova and empower the
Migs, shown in Figure 3.
Each of our characters are designed as puppets that
can follow an actor’s movements, facial expressions,
and speech using Adobe’s Character Animator, shown
in Figure 1. In addition, we have created mouth
transitions that demonstrate correct mouth shapes for
the players to follow and exaggerated expressions for
the Migs to communicate without speech.
Participatory Design and Play Sessions
We believe users are best suited to create the therapy
experience they want to play. We have developed a
process using the latest tech and tools to develop
medium fidelity prototypes on scene with players that
can be played and iterated on within one design
session. Our full design process:
1) Depending on the target audience, we use
tangible representations of our characters,
cards, props, paper, and writing utensils,
shown in Figure 2, to brainstorm, bodystorm,
play, and ideate new subplots, mechanics, and
challenges for SpokeIt.
2) Users then work with designers to develop
these ideas into medium fidelity prototypes
using Adobe XD, shown in Figure 4.
Participants choose where elements are placed
on the screen, what those elements should
look like, prompts and scripts for the
characters, and the possible interactions
between scenes.
3) We use Wizard of Oz techniques to test the
prototypes with players. The facilitators do the
voice acting and act as the speech recognition
system as players navigate their designs on
the iPad using Adobe XD.
4) Based on observed reactions and player
feedback, the medium fidelity prototypes are
iterated until they are ready for full
development. We often have participants
present their designs to gain an understanding
of what aspects of the design they find
important.
5) We conduct user studies at medical clinics and
centers for people with developmental
disabilities to evaluate new materials.
Discussion
We believe the tools and tactics used for our design
process could benefit HCI practitioners who are
interested in creating medium fidelity prototypes in
real-time with target users. The tools allow for
drastically reduced time spent creating prototypes,
which affords more time co-creating with users. Within
one design session, we are able to ideate new games,
rapid prototype in medium fidelity, and iterate on
designs that users can play.
Adobe Character Animator saves time on creating rich
animations because actors’ faces are mapped directly
to our characters’ puppets. These animations support
physics and realistic mouth transitions for speech. HCI
practitioners who create games may be interested in
using these tools to boost productivity and minimize
development costs.
Figure 2 Tangible character
representations and design tools
Figure 3 SpokeIt—A co-created
speech therapy game
CHI 2018 Demonstration
CHI 2018, April 21–26, 2018, Montréal, QC, Canada
D501, Page 3
Future Work
We continue turning medium fidelity prototypes into
fully developed content for SpokeIt. After SpokeIt is
released to the App Store, we will be able to collect
larger amounts of speech and usage data.
Acknowledgements
This material is based in part upon work supported by
the National Science Foundation under Grant number
#1617253. We also thank Doctor Travis Tollefson and
SLP Christina Roth for their aid in conducting user
evaluations. Any opinions, findings, and conclusions or
recommendations expressed in this material are those
of the authors and do not necessarily reflect the views
of the National Science Foundation.
References
1) Jelle Van Dijk, Niels Hendriks, Frauenberger
Christopher, Fenne Verhoeven, Karin Slegers,
Eva Brandt, and Rita Maldonado Branco. 2016.
Empowering people with impairments: How
participatory methods can inform the design of
empowering artifacts. In Proceedings of the
14
th
Participatory Design Conference: Short
Papers, Interactive Exhibitions, Workshops,
121–122.
2) Jared Duval. 2017. A mobile game system for
improving the speech therapy experience. In
Proceedings of the 19th International
Conference on Human-Computer Interaction
with Mobile Devices and Services, 72.
3) Jared Duval, Zachary Rubin, Elizabeth
Goldman, Nick Antrilli, Yu Zhang, Su-Hua
Wang, and Sri Kurniawan. 2017. Designing
Towards Maximum Motivation and Engagement
in an Interactive Speech Therapy Game. In
Proceedings of the 2017 Conference on
Interaction Design and Children, 589–594.
4) James Paul Gee. 2005. Learning by design:
Good video games as learning machines. E-
learning and Digital Media 2, 1: 5–16.
5) David Huggins-Daines, Mohit Kumar, Arthur
Chan, Alan W Black, Mosur Ravishankar, and
Alexander I Rudnicky. 2006. Pocketsphinx: A
free, real-time continuous speech recognition
system for hand-held devices. In Acoustics,
Speech and Signal Processing, 2006. ICASSP
2006 Proceedings. 2006 IEEE International
Conference on, I--I.
6) Josephine M Randel, Barbara A Morris, C
Douglas Wetzel, and Betty V Whitehill. 1992.
The effectiveness of games for educational
purposes: A review of recent research.
Simulation & gaming 23, 261–276.
7) Ricardo Rosas, Miguel Nussbaum, Patricio
Cumsille, Vladimir Marianov, Mónica Correa,
Patricia Flores, Valeska Grau, Francisca Lagos,
Ximena López, Verónica López, and others.
2003. Beyond Nintendo: design and
assessment of educational video games for first
and second grade students. Computers &
Education 40, 1: 71–94.
8) Zachary Rubin. 2017. Development and
evaluation of software tools for speech
therapy. University of California, Santa Cruz.
9) Zachary Rubin, Sri Kurniawan, and Travis
Tollefson. 2014. Results from using automatic
speech recognition in cleft speech therapy with
children. In International Conference on
Computers for Handicapped Persons, 283–286.
10) Maria Virvou, George Katsionis, and
Konstantinos Manos. 2005. Combining software
games with education: Evaluation of its
educational effectiveness. Educational
Technology & Society 8, 2: 54–65.
Figure 4 Medium fidelity prototypes
developed on-site with co-creators
CHI 2018 Demonstration
CHI 2018, April 21–26, 2018, Montréal, QC, Canada
D501, Page 4