Conference PaperPDF Available

SpokeIt: A Co-Created Speech Therapy Experience

Authors:

Abstract and Figures

We present SpokeIt, a novel speech therapy game co-created with users, medical experts, speech pathologists, developmental psychologists, and game designers. Our serious game for health aims is to augment and support in-office speech therapy sessions with an at-home supported speech therapy experience. SpokeIt was co-created using participatory methods involving multiple stakeholders and target users. We describe the technical details of SpokeIt, the process of working with multiple stakeholders, and the methods that allowed us to create medium fidelity prototypes in real-time with players. We share our emerging designs, tools, insights for the broader CHI audience, and plans for future work.
Content may be subject to copyright.
SpokeIt: A Co-Created Speech Therapy
Experience
Abstract
We present SpokeIt, a novel speech therapy game co-
created with users, medical experts, speech
pathologists, developmental psychologists, and game
designers. Our serious game for health aims is to
augment and support in-office speech therapy sessions
with an at-home supported speech therapy experience.
SpokeIt was co-created using participatory methods
involving multiple stakeholders and target users. We
describe the technical details of SpokeIt, the process of
working with multiple stakeholders, and the methods
that allowed us to create medium fidelity prototypes in
real-time with players. We share our emerging designs,
tools, insights for the broader CHI audience, and plans
for future work.
Author Keywords
SpokeIt; Speech Therapy; Serious Game for Health;
Participatory Design
ACM Classification Keywords
H. Information Systems; H.5. Information interfaces
and presentation (e.g., HCI); H.5.2. Voice I/O; I/O J.3.
Life and Medical Sciences: Health
Introduction
Without a trained speech therapist at home, there is an
opportunity to support and augment the speech
Permission to make digital or hard copies of part or all of this work for
personal or classroom use is granted without fee provided that copies are
not made or distributed for profit or commercial advantage and that
copies bear this notice and the full citation on the first page. Copyrights
for third-party components of this work must be honored. For all other
uses, contact the Owner/Author.
CHI'18 Extended Abstracts, April 21–26, 2018, Montreal, QC, Canada
© 2018 Copyright is held by the owner/author(s).
ACM ISBN 978-1-4503-5621-3/18/04.
https://doi.org/10.1145/3170427.3186494
Jared Scott Duval
University of California Santa Cruz
Santa Cruz, CA 95060, USA
JDuval@ucsc.edu
Elena Márquez Segura
University of California Santa Cruz
Santa Cruz, CA 95060, USA
Elena.Marquez@ucsc.edu
Sri Kurniawan
University of California Santa Cruz
Santa Cruz, CA 95060, USA
skurnia@ucsc.edu
CHI 2018 Demonstration
CHI 2018, April 21–26, 2018, Montréal, QC, Canada
D501, Page 1
therapy experience outside of the office. Our interviews
with Speech Language Pathologists (SLPs) show that
at-home curriculums are tedious and often forgotten.
SLPs recommend patients practice speech for 10
minutes per day, but suspect little to no time is spent
practicing out of the office, and as a result, much of in-
office time is spent reestablishing patient’s abilities.
SLPs have no record of time spent practicing at home
or the progress that is being made outside of the office.
A serious game for health has the potential to support
speech therapy at home, provide feedback to the
player, and supply SLPs with new information, making
in-office sessions more efficient.
In this document, we first provide the technical details
that power our critical speech recognition system. Next,
we show how multiple stakeholders and users co-
created and designed SpokeIt. We share our emerging
designs and tools used, and conclude with a brief
discussion and future work.
Related Work
According to SLPs we interviewed, the extrinsic
motivation to improve speech and the encouragement
of traditional therapy supporters is not always enough
to keep patients practicing. Literature has shown that
games are an effective educational intervention and
medium to convey and support feelings of self-efficacy
due to the immersive and pervasive virtual
environment, and that it had been shown to work
effectively as educational interventions [4]. Games
have demonstrated the ability to teach while providing
a motivating and interactive environment [10] are
effective with children [7] and can be as effective as
face-to-face instruction [6]. PD can empower people
with sensory, cognitive, or social impairments to
actively take part in the design process of a system [1].
Technical Details
The offline real-time critical speech recognition system
is built using RapidEars, an extension of Carnegie
Mellon’s Pocketsphinx [5]. We have modified their
system to listen for both correct and incorrect
utterances of speech. SpokeIt uses speech input as the
principal mechanic to make progress in the game,
which includes single-word targets and repetition of
longer phrases. Balancing offline speech recognition
and a game engine is challenging considering the
limited processing power of mobile devices.
Predecessors of SpokeIt [2,3] that use similar
technology include Rubin’s Speech Adventure, and
Speech with Sam [8,9]. We use Apple’s SpriteKit and
Swift 3, native to IOS devices, and RapidEars, written
in Objective C, for the development of our game.
The Co-Creation of SpokeIt
Medical experts and SLPs provided the initial functional
requirements for the project. Namely, the game must
run on iPads, must not require an internet connection,
and critically listen to speech. Our players are often in
lower socioeconomic homes and do not have Wi-Fi or
regular access to a speech therapist. The clinic we are
working with provides patients iPads with the software
installed. We worked with developmental psychologists
and game designers to create an overarching plot and
characters that are developmentally appropriate for
children and adults with developmental disabilities.
Plot and Characters
Our story follows Nova, a star powered by speech
energy, who has fallen from the sky and is tasked with
Figure 1 Adobe Character
Animator. To the top, Nova
demonstrating /P and /L
phenomes. To the bottom, Migs
expressing exaggerated
emotions.
CHI 2018 Demonstration
CHI 2018, April 21–26, 2018, Montréal, QC, Canada
D501, Page 2
helping the Migs, a race of speechless characters,
return color to their world. The player must use the
power of their voice to charge Nova and empower the
Migs, shown in Figure 3.
Each of our characters are designed as puppets that
can follow an actor’s movements, facial expressions,
and speech using Adobe’s Character Animator, shown
in Figure 1. In addition, we have created mouth
transitions that demonstrate correct mouth shapes for
the players to follow and exaggerated expressions for
the Migs to communicate without speech.
Participatory Design and Play Sessions
We believe users are best suited to create the therapy
experience they want to play. We have developed a
process using the latest tech and tools to develop
medium fidelity prototypes on scene with players that
can be played and iterated on within one design
session. Our full design process:
1) Depending on the target audience, we use
tangible representations of our characters,
cards, props, paper, and writing utensils,
shown in Figure 2, to brainstorm, bodystorm,
play, and ideate new subplots, mechanics, and
challenges for SpokeIt.
2) Users then work with designers to develop
these ideas into medium fidelity prototypes
using Adobe XD, shown in Figure 4.
Participants choose where elements are placed
on the screen, what those elements should
look like, prompts and scripts for the
characters, and the possible interactions
between scenes.
3) We use Wizard of Oz techniques to test the
prototypes with players. The facilitators do the
voice acting and act as the speech recognition
system as players navigate their designs on
the iPad using Adobe XD.
4) Based on observed reactions and player
feedback, the medium fidelity prototypes are
iterated until they are ready for full
development. We often have participants
present their designs to gain an understanding
of what aspects of the design they find
important.
5) We conduct user studies at medical clinics and
centers for people with developmental
disabilities to evaluate new materials.
Discussion
We believe the tools and tactics used for our design
process could benefit HCI practitioners who are
interested in creating medium fidelity prototypes in
real-time with target users. The tools allow for
drastically reduced time spent creating prototypes,
which affords more time co-creating with users. Within
one design session, we are able to ideate new games,
rapid prototype in medium fidelity, and iterate on
designs that users can play.
Adobe Character Animator saves time on creating rich
animations because actors’ faces are mapped directly
to our characters’ puppets. These animations support
physics and realistic mouth transitions for speech. HCI
practitioners who create games may be interested in
using these tools to boost productivity and minimize
development costs.
Figure 2 Tangible character
representations and design tools
Figure 3 SpokeIt—A co-created
speech therapy game
CHI 2018 Demonstration
CHI 2018, April 21–26, 2018, Montréal, QC, Canada
D501, Page 3
Future Work
We continue turning medium fidelity prototypes into
fully developed content for SpokeIt. After SpokeIt is
released to the App Store, we will be able to collect
larger amounts of speech and usage data.
Acknowledgements
This material is based in part upon work supported by
the National Science Foundation under Grant number
#1617253. We also thank Doctor Travis Tollefson and
SLP Christina Roth for their aid in conducting user
evaluations. Any opinions, findings, and conclusions or
recommendations expressed in this material are those
of the authors and do not necessarily reflect the views
of the National Science Foundation.
References
1) Jelle Van Dijk, Niels Hendriks, Frauenberger
Christopher, Fenne Verhoeven, Karin Slegers,
Eva Brandt, and Rita Maldonado Branco. 2016.
Empowering people with impairments: How
participatory methods can inform the design of
empowering artifacts. In Proceedings of the
14
th
Participatory Design Conference: Short
Papers, Interactive Exhibitions, Workshops,
121–122.
2) Jared Duval. 2017. A mobile game system for
improving the speech therapy experience. In
Proceedings of the 19th International
Conference on Human-Computer Interaction
with Mobile Devices and Services, 72.
3) Jared Duval, Zachary Rubin, Elizabeth
Goldman, Nick Antrilli, Yu Zhang, Su-Hua
Wang, and Sri Kurniawan. 2017. Designing
Towards Maximum Motivation and Engagement
in an Interactive Speech Therapy Game. In
Proceedings of the 2017 Conference on
Interaction Design and Children, 589–594.
4) James Paul Gee. 2005. Learning by design:
Good video games as learning machines. E-
learning and Digital Media 2, 1: 5–16.
5) David Huggins-Daines, Mohit Kumar, Arthur
Chan, Alan W Black, Mosur Ravishankar, and
Alexander I Rudnicky. 2006. Pocketsphinx: A
free, real-time continuous speech recognition
system for hand-held devices. In Acoustics,
Speech and Signal Processing, 2006. ICASSP
2006 Proceedings. 2006 IEEE International
Conference on, I--I.
6) Josephine M Randel, Barbara A Morris, C
Douglas Wetzel, and Betty V Whitehill. 1992.
The effectiveness of games for educational
purposes: A review of recent research.
Simulation & gaming 23, 261–276.
7) Ricardo Rosas, Miguel Nussbaum, Patricio
Cumsille, Vladimir Marianov, Mónica Correa,
Patricia Flores, Valeska Grau, Francisca Lagos,
Ximena López, Verónica López, and others.
2003. Beyond Nintendo: design and
assessment of educational video games for first
and second grade students. Computers &
Education 40, 1: 71–94.
8) Zachary Rubin. 2017. Development and
evaluation of software tools for speech
therapy. University of California, Santa Cruz.
9) Zachary Rubin, Sri Kurniawan, and Travis
Tollefson. 2014. Results from using automatic
speech recognition in cleft speech therapy with
children. In International Conference on
Computers for Handicapped Persons, 283–286.
10) Maria Virvou, George Katsionis, and
Konstantinos Manos. 2005. Combining software
games with education: Evaluation of its
educational effectiveness. Educational
Technology & Society 8, 2: 54–65.
Figure 4 Medium fidelity prototypes
developed on-site with co-creators
CHI 2018 Demonstration
CHI 2018, April 21–26, 2018, Montréal, QC, Canada
D501, Page 4
... The same applies for co-creation processes aiming at the development of technologies that facilitate personal health care. Co-creating for personal health care aims at either supporting the personal treatment of patients with a particular disease (Duval et al., 2018;Stütz et al., 2017), the reporting and/or visualization of patient data (Ivatury et al., 2017;Sauvé et al., 2017) or to support active and healthy aging (Foverskov & Binder, 2011). In all these cases, co-creation is understood as a collaborative design practice where the technological outcome facilitates more independent forms of living, therapy, or preventive health care of a (more or less) specific target audience. ...
Preprint
Full-text available
New methods and technologies for engaging future users and other stakeholders in participatory (design) processes are being developed and proposed in computing. Researchers increasingly refer to co-creation in order to capture these approaches. However, how co-creation is being framed and understood across domains differs substantially. To better understand co-creation in computing, we conducted a literature review of all papers in the ACM Digital Library with co-creation or co-create in their abstracts. After an initial screening, we retained 62 for further analysis. We introduce a framework to analyse different notions of co-creation, distinguishing between co-creation target audiences, the roles of co-creators, the role of technology (as means or objective) and the results of co-creation. We discuss the adoption of co-creation in domains such as learning, business, arts and culture, health, and the public sector. This paper contributes to the understanding of different approaches and conceptualisations of co-creation in computing and puts forward an agenda for future research.
... For example, in previous work [8], we found that an MPD algorithm overwhelmingly outperformed caregivers at word-level MPD in-the-field. Although some digital child therapy projects have provided word-level feedback [12,13], systems like these eventually need phonemelevel feedback so that speech therapy practice can target specific problematic sounds [14]. This is a substantially more challenging task because the system needs to model individual errors, rather than matching whole utterances to a certain word label. ...
... Shaping the learning health system has been identified as one of the grand challenges for HCI researchers and one which will require a bottom up approach to engaging patients and clinicians [44]. Previous HCI studies exploring SLT have largely focused on designing technology applications for specific use cases within SLT treatment [13,26,29], however, we are not aware of any studies with a focus on designing digital tools that support the enhancement of SLT education and training. That said, there has been work exploring the role of digital technology in enhancing clinical skills training within other medical fields. ...
... ognition system capable of "hearing" many common speech errors. SpokeIt contains numerous mini-games that target specific types of speech therapy, such as rhythm and articulation, that fit within an overarching narrative. SpokeIt's narrative, mechanics, and characters were iteratively co-designed and presented in our previous works (J. Duval, 2017;J. S. Duval, Márquez Segura, & Kurniawan, 2018). ...
Conference Paper
Full-text available
Therapy can be costly, time-consuming, repetitive, and difficult. Games have the power to teach transferable skills, can turn repetitive tasks into engaging mechanics, have been proven to be effective at delivering various forms of therapy, and can be deployed at large scales. Therapy games represent fertile ground for connected learning. In this work, we collaborate with 7 children with corrected cleft palate aged 2-10 and their parents during their yearly visit to UC Davis Medical center to co-create and evaluate SpokeIt, a speech therapy game. Each of these children come from low socioeconomic statuses with limited access to speech therapy and would benefit from the amplified opportunities of new media in connected learning. Throughout the study, we ran multiple cascading participatory design sessions using design principles of connected learning, which culminated in the design of two new medium-fidelity prototypes presented in this paper.
... [16]), health (e.g. [9]), or workplace (e.g. [14]). ...
Conference Paper
We propose a Situated Play Design (SPD) workshop aimed at exploring how culture and traditions can guide playful design. Using food as an accessible starting point, we invite scholars from diverse communities to share, analyze, and make creative use of playful traditions, and prototype new and interesting eating experiences. Through hands-on engagement with traditions, play and technology, we will discuss strategies to make designerly use of forms of play that are embedded in culture. The outcomes of the workshop will be twofold: First, in response to recent calls for increasingly situated and emergent play design methods, we explore strategies to chase culturally-grounded play. Second, we produce an annotated portfolio of "play potentials" to inspire the design of future food-related technologies. The workshop will contribute to enriching the set of tools available for designers interested in play and technologies for everyday-use, in and beyond the food domain.
Article
Digital games can make speech therapy exercises more enjoyable for children and increase their motivation during therapy. However, many such games developed to date have not been designed for long-term use. To address this issue, we developed Apraxia World, a speech therapy game specifically intended to be played over extended periods. In this study, we examined pronunciation improvements, child engagement over time, and caregiver and automated pronunciation evaluation accuracy while using our game over a multi-month period. Ten children played Apraxia World at home during two counterbalanced 4-week treatment blocks separated by a 2-week break. In one treatment phase, children received pronunciation feedback from caregivers and in the other treatment phase, utterances were evaluated with an automated framework built into the game. We found that children made therapeutically significant speech improvements while using Apraxia World, and that the game successfully increased engagement during speech therapy practice. Additionally, in offline mispronunciation detection tests, our automated pronunciation evaluation framework outperformed a traditional method based on goodness of pronunciation scoring. Our results suggest that this type of speech therapy game is a valid complement to traditional home practice.
Conference Paper
Full-text available
A lack of intrinsic motivation to practice speech is attributed to tedious and repetitive speech curriculums, but mobile games have been widely recognized as a valid motivator for jaded individuals. SpokeIt is an interactive storybook style speech therapy game that intends to turn practicing speech into a motivating and productive experience for individuals with speech impairments as well as provide speech therapists an important diagnostic tool. In this paper, I discuss the novel intellectual contributions SpokeIt can provide such as an offline critical conversational speech recognition system, and the application of therapy curriculums to mobile platforms, I present conducted research, and consider exciting future work and research directions.
Conference Paper
Full-text available
Children with speech impairments often find speech curriculums tedious, limiting how often children are motivated to practice. A speech therapy game has the potential to make practice fun, may help facilitate increased time and quality of at-home speech therapy, and lead to improved speech. We explore using conversational real-time speech recognition, game methodologies theorized to improve immersion and flow, and user centered approaches to design an immersive interactive speech therapy solution. Our preliminary user evaluation showed that compared to traditional methods, children were more motivated to practice speech using our system.
Article
Full-text available
This article asks how good video and computer game designers manage to get new players to learn long, complex and difficult games. The short answer is that designers of good games have hit on excellent methods for getting people to learn and to enjoy learning. The longer answer is more complex. Integral to this answer are the good principles of learning built into successful games. The author discusses 13 such principles under the headings of ‘Empowered Learners’, ‘Problem Solving’ and ‘Understanding’ and concludes that the main impediment to implementing these principles in formal education is cost. This, however, is not only (or even so much) monetary cost. It is, importantly, the cost of changing minds about how and where learning is done and of changing one of our most profoundly change-resistant institutions: the school.
Conference Paper
Full-text available
The availability of real-time continuous speech recognition on mobile and embedded devices has opened up a wide range of research opportunities in human-computer interactive applications. Unfortunately, most of the work in this area to date has been confined to proprietary software, or has focused on limited domains with constrained grammars. In this paper, we present a preliminary case study on the porting and optimization of CMU Sphinx-11, a popular open source large vocabulary continuous speech recognition (LVCSR) system, to hand-held devices. The resulting system operates in an average 0.87 times real-time on a 206 MHz device, 8.03 times faster than the baseline system. To our knowledge, this is the first hand-held LVCSR system available under an open-source license
Article
Full-text available
The main objective of this study was to evaluate the effects of the introduction of educational videogames into the classroom, on learning, motivation, and classroom dynamics. These effects were studied using a sample of 1274 students from economically disadvantaged schools in Chile. The videogames were specifically designed to address the educational goals of the first and second years of school, for basic mathematics and reading comprehension. The sample was divided into experimental groups (EG), internal control groups (IC) and external control groups (EC). Students in the EG groups, used the experimental video games during an average of 30 h over a 3-month period. They were evaluated on their acquisition of reading comprehension, spelling, and mathematical skills, and on their motivation to use video games. Teachers’ expectations of change due to the use of video games, their technological transfer, and handling of classroom dynamics, were assessed through ad hoc tests and classroom observations. The results show significant differences between the EG and IC groups in relation to the EC group in Math, Reading Comprehension and Spelling, but no significant differences in these aspects were found between the EG and the IC groups. Teacher reports and classroom observations confirm an improvement in motivation to learn, and a positive technological transfer of the experimental tool. Although further studies regarding the effects of learning through videogame use are imperative, positive effects on motivation and classroom dynamics, indicate that the introduction of educational video games can be a useful tool in promoting learning within the classroom.
Article
Full-text available
Computer games are very popular among children and adolescents. In this respect, they could be exploited by educational software designers to render educational software more attractive and motivating. However, it remains to be explored what the educational scope of educational software games is. In this paper, we explore several issues concerning the educational effectiveness, appeal and scope of educational software games through an evaluation study of an Intelligent Tutoring System (ITS) that operates as a virtual reality educational game. The results of the evaluation show that educational virtual reality games can be very motivating while retaining or even improving the educational effects on students. Moreover, one important finding of the study was that the educational effectiveness of the game was particularly high for students who used to have poor performance in the domain taught prior to their learning experience with the game.
Conference Paper
Most children with cleft are required to undertake speech therapy after undergoing surgery to repair their craniofacial defect. However, the untrained ear of a parent can lead to incorrect practice resulting in the development of compensatory structures. Even worse, the boring nature of the cleft speech therapy often causes children to abandon home exercises and therapy altogether. We have developed a simple recognition system capable of detecting impairments on the phoneme level with high accuracy. We embed this into a game environment and provide it to a cleft palate specialist team for pilot testing with children 2 to 5 years of age being evaluated for speech therapy. The system consistently detected cleft speech in high-pressure consonants in 3 out of our 5 sentences. Doctors agreed that this would improve the quality of therapy outside of the office. Children enjoyed the game overall, but grew bored due to the delays of phrase-based speech recognition.
Conference Paper
Participatory Design has developed methods that empower people with impairments to actively take part in the design process. Many designed artifacts for this target group likewise aim to empower their users in daily life. In this workshop, we share and relate best practices of both empowering methods and empowering designs. Participants are therefore invited to bring along cases of designing for- and with people with sensory-, cognitive- or social impairments. Our workshop consists of three parts: (1) Foregrounding empowering elements in PD methods using method stories, containing the backstory of a method put into practice; (2) Reflecting on technological artifacts, exploring the empowering qualities of person-artifact-context interaction; (3) constructing a critical synopsis of the various relationships between empowering products and - methods.
Article
This article reviews the literature that compares the instructional effectiveness of games to conventional classroom instruction. Studies dealing with empirical research rather than teachers'judgments are reviewed. Published reviews of research in English dating from 1963 to 1984 were examined and the literature was searched for studies from 1984 to 1991. Of the 67 studies considered over a period of 28 years, 38 show no difference between games and conventional instruction; 22 favor games; 5 favor games, but their controls are questionable; and 3 favor conventional instruction. Results for social sciences, math, language arts, logic physics, biology, retention over time, and interest are examined. Math is the subject area with the greatest percentage of results favoring games, but only eight studies have adequate controls. Thirty-three out of 46 social science games/simulations show no difference between games/simulations and classroom instruction. The authors conclude that subject matter areas where very specific content can be targeted are more likely to show beneficial effects for gaming.
Pocketsphinx: A free, real-time continuous speech recognition system for hand-held devices
  • David Huggins-Daines
  • Mohit Kumar
  • Arthur Chan
  • Alan W Black
  • Mosur Ravishankar
  • Alexander I Rudnicky
David Huggins-Daines, Mohit Kumar, Arthur Chan, Alan W Black, Mosur Ravishankar, and Alexander I Rudnicky. 2006. Pocketsphinx: A free, real-time continuous speech recognition system for hand-held devices. In Acoustics, Speech and Signal Processing, 2006. ICASSP 2006 Proceedings. 2006 IEEE International Conference on, I-I.