Conference PaperPDF Available

Designing Towards Maximum Motivation and Engagement in an Interactive Speech Therapy Game


Abstract and Figures

Children with speech impairments often find speech curriculums tedious, limiting how often children are motivated to practice. A speech therapy game has the potential to make practice fun, may help facilitate increased time and quality of at-home speech therapy, and lead to improved speech. We explore using conversational real-time speech recognition, game methodologies theorized to improve immersion and flow, and user centered approaches to design an immersive interactive speech therapy solution. Our preliminary user evaluation showed that compared to traditional methods, children were more motivated to practice speech using our system.
Content may be subject to copyright.
Designing Towards Maximum
Motivation and Engagement in an
Interactive Speech Therapy Game
Children with speech impairments often find speech
curriculums tedious, limiting how often children are
motivated to practice. A speech therapy game has the
potential to make practice fun, may help facilitate
increased time and quality of at-home speech therapy,
and lead to improved speech. We explore using
conversational real-time speech recognition, game
methodologies theorized to improve immersion and
flow, and user centered approaches to design an
immersive interactive speech therapy solution. Our
preliminary user evaluation showed that compared to
traditional methods, children were more motivated to
practice speech using our system.
Author Keywords
Speech Processing; Speech Therapy; Human Computer
Interaction; Games; Motivation
ACM Classification Keywords
H. Information Systems; H.5. Information interfaces
and presentation (e.g., HCI); H.5.2. Voice I/O; I/O J.3.
Life and Medical Sciences: Health
Permission to make digital or hard copies of part or all of
this work for personal or classroom use is granted without
fee provided that copies are not made or distributed for
profit or commercial advantage and that copies bear this
notice and the full citation on the first page. Copyrights for
third-party components of this work must be honored. For
all other uses, contact the Owner/Author.
IDC '17, June 27-30, 2017, Stanford, CA, USA
© 2017 Copyright is held by the owner/author(s).
ACM ISBN 978-1-4503-4921-5/17/06.
Jared Duval
Zachary Rubin
Elizabeth Goldman
Nick Antrilli
Yu Zhang
Su-Hua Wang
Sri Kurniawan
University of California Santa Cruz
Santa Cruz, CA 95064 USA
Work in Progress/Late Breaking
IDC 2017, June 27–30, 2017, Stanford, CA, USA
The prevalence of speech sound phonological and
articulation disorders in young children is 8 to 9 percent
[13]. Most speech impairments can be corrected with
practice and speech therapy. Speech therapy consists
of two components: in-office sessions with a speech
language pathologist (SLP) and at-home practice
curriculums [6, 11]. Orofacial cleft is one of the most
common causes of speech impairments.
Orofacial cleft is a common birth defect that results in a
gap in the lip or mouth, shown in Figures 1 and 2.
Children born with orofacial cleft undergo multiple
surgical procedures and often require long-term speech
therapy. Speech impairment is extremely common with
this birth defect because it allows air to escape through
the nasal cavity, which results in the use of
compensatory glottal attacks to mimic sounds, lisps,
and difficulty forming plosives.
During in-office therapy sessions, SLPs evaluate the
child’s speech to find specific targets that the child has
difficulty speaking. Once the SLP knows the child’s
unique impairments, they work with child on techniques
to improve the detected targets. The SLP gives children
and parents resources to take home to continue
practicing until the next session, where the child is
SLPs instruct parents to have their child practice for ten
minutes per day, but according to our interview with a
SLP, it is suspected that children practice little to none
outside of speech therapy sessions. There are many
contributing factors as to why children do not practice
enough at home: children find the curriculums tedious
and repetitive, parents are not qualified to assess the
small nuances in leading a speech curriculum, and
children may not be intrinsically motivated to improve
their speech.
Many children do not have enough access to SLPs and
could benefit from a supplemental speech therapy
game. The challenges a speech therapy game may
address are numerous [12]:
Moves responsibility to lead practice from the
parent to the game
Can assign appropriate words and phrases
from an expansive knowledge base for each
child’s unique speech goals
Adjusts difficulty as the child plays using a
dynamic curriculum to balance challenge and
boredom [3]
Adds intrinsic motivation to practicing speech
through gameplay and mechanics
Allows SLP to track practice and performance
in-between in-office therapy sessions to make
better use of their time
Allows children with less access to an SLP to
continue improving speech
A high-fidelity prototype of our speech therapy game
named SpokeIt has been developed and tested. Many
of the design considerations mentioned below are the
result of feedback we received from the original high-
fidelity prototype of SpokeIt. We first discuss the new
design motivations and then the results of our user
study that lead to these decisions.
Design of the game content, mechanics, and
interactions are central to the success of improving
Figure 1: An image that shows a
baby with cleft lip. Image CC-BY
Center for Disease Control and
Figure 2: An image that shows a
baby with cleft palate. Image CC-
BY Center for Disease Control and
Work in Progress/Late Breaking
IDC 2017, June 27–30, 2017, Stanford, CA, USA
speech and motivating practice. “Placing high hopes on
games designed for the public good- as many
nonprofits, health organizations, and social enterprises
are doing-without realizing the bad game design can
undermine the most noble of ambitions. It's quite
possible to make terrible, dull, and unappealing games
for learning or training or health” [9].
To improve speech, one must practice speaking, but
saying a word or phrase repeatedly for the sake of
practice is often found to be tedious and boring. During
our observations of interactions between SLPs and their
patient’s we noticed the jaded children were not
interested in practicing speech.
Games turn repetition from something that stifles
motivation and induces boredom into an element that is
recognized as useful for progress in the game [10].
Hiding information about gameplay and maintaining a
sense of uncertainty about the outcome can make
games fun [10]. Repeating target words and phrases
for the advancement in a game is much more
motivating than for the completion of a speech
Educational software has traditionally attempted to
harness games as extrinsic motivation by using them
as sugar coating for learning content, but children learn
more from intrinsic approaches [8]. The learning
objectives should be part of the gameplay itself [8]
Practicing speech should be a game mechanic that
induces immersion and flow rather than a disruption to
play. The narrative and plot should align with the
learning content [5].
If done correctly, games can lead to improved learning,
enhanced motivation, greater attention, and increased
retention [2]. The end state we desire is the motivated
learner, which is someone who is enthusiastic, focused,
and engaged [7].
In order to understand the needs of the relevant
stakeholders we held a focus group with cleft lip
specialists, researchers, and game developers. Cleft lip
specialists emphasized the need for virtual incentives
that would remain effective for months or even years.
Researchers and game developers agreed that
maximizing immersion through emotionally motivating
elements would best hide the repetitive speech
exercises. We design and validate the following aspects
of our game are emotionally motivating:
Characters that are relatable and have the
capacity to create empathy
An overarching plot for our characters that is
interesting and defines the player’s goals
Narrators that engage and encourage the
player to continue helping the game
A Procedural Content Background Art
Generator that creates rich environments to
Mechanics that seamlessly incorporate
conversational speech recognition into
completing game objectives
The characters in a game are often the single most
motivating factors to continue play [9]. We become
invested in their wellbeing, life, and goals [9]. The
Figure 3: Blue game character
Figure 4: Yellow game character
Work in Progress/Late Breaking
IDC 2017, June 27–30, 2017, Stanford, CA, USA
children who play our speech therapy game should be
able to relate to the characters and be motivated to
help them.
Our speech therapy game follows a race known as the
Migs, shown if Figures 3 and 4. The Migs do not speak.
The player is their voice. The game includes Migs of six
different colors, six different personalities, and unique
relationships with the other Migs.
We animate the Migs facial expressions to display the
primary emotions as a starting point for creating
natural and realistic experiences within the game [1].
Using Adobe’s Character Animator, we are able to
employ actors to capture and record animations in real
time that are mapped directly to the Migs. By
combining the animations created by graphics artists,
performances by actors in Character Animator, and
physics, we can make believable and natural facial
expressions so that children can easily understand our
game character’s emotions and hopefully empathize
with them.
The story of the Migs is a hopeful one. Before the game
is played for the first time, the player is provided an
introductory cinematic to the game that describes the
Migs history and story. The story begins in the Mig’s
beautiful world, where the narrator describes their
utopian lives. As the background music gets more
dramatic, the Migs are whisked up by a storm into a
new unfamiliar two-dimensional world.
Procedural Content Generation
Speech therapy is an ongoing process that takes each
child a different amount of time to complete. Therefore,
our speech therapy game should continue to be
available. Previous iterations of our work employed
mini-games that would play for an allotted amount of
time before switching to the next game. Games could
be randomly ordered and could be played multiple
times. We observed that the mini-games lacked
replayability and would not sustain long-term use.
Fresh procedurally generated content should solve
replayability issues.
Our PCG employs an evolutionary algorithm that learns
how to make art with our collection of over 750 image
assets. The evolutionary algorithm borrows from
biology to develop art that is believable and
aesthetically pleasing, shown in Figure 6. It actively
learns to match selected themes, materials, terrain
settings, lighting, and time of day to produce visually
striking worlds, shown in Figure 5.
As development continues, the plot and game
objectives will also employ PCG so that our speech
therapy game can always deliver fresh new content
that will keep the children motivated and engaged.
The elements of engaged learning are focused goals,
challenging tasks, clear and compelling standards,
protection from adverse consequences for initial
failures, affirmation of performance, affiliation with
others, novelty and variety, choice, and authenticity
Games are proven to have perceptual and cognitive
impacts as well as the ability for the player to acquire
new skills [2]. We can adjust difficulty by assigning
target words that are more difficult to the child,
Figure 5: Settings available to
system to procedurally generate
background art using
evolutionary algorithm
Work in Progress/Late Breaking
IDC 2017, June 27–30, 2017, Stanford, CA, USA
changing the number of repetitions needed to move
past a challenge, adjusting the rhythm and timing
required to say the targets, and by tweaking the
number of targets required per interaction.
Volume is a challenge for many children. To
successfully complete a challenge the child must say
the correct target at an appropriate volume. There is an
indicator that displays the loudness of the child’s
speech and it changes colors when volume moves in
and out of the threshold. It should be clear to the
player when the game is listening and when they
should be speaking. The child should only be able to
speak when the narrators are not. To make this clear,
we include an ear in the top-right corner of the game to
indicate when the child should speak and is toggled off
when the game is not listening, shown in Figure 7.
SpokeIt (1
iteration shown in Figure 8 and 2
iteration shown in Figure 9) has been tested on children
with cleft speech as well as with expert medical
professionals that work with children with cleft speech.
Four participants with cleft speech played SpokeIt: 3
children under 10 (Mean: 5.33, σ: 2.3) and 1 teenager.
One child in particular engaged much more with the
game than the shyer children. The children under 10
had speech barely above the detection threshold,
resulting in play difficulties. We addressed this with a
volume indicator and adding narrator voice prompts
instructing the player to speak louder.
The child life specialist said she loved watching the
eight-year old play the game. She noticed that he
immediately put much more effort into his speech. She
demonstrated how he overemphasized each spoken
syllable in the target sentence to make the game
advance. The SLP had trouble keeping his attention,
but when he was given the opportunity to try SpokeIt,
he completed the entire prototype and asked for more
The SLP who watched her patients play SpokeIt loved
the storybook style of the game. She is skeptical that
the game will ever be able to make the distinction
between correct and incorrect speech, but looks
forward to motivating practice nonetheless.
The plastic surgeon who debriefed with us wants to add
a feature to the game that would display a mouth and
demonstrate correct speech when the child was
The eldest spoke primarily using sign language. It was
very difficult for the SLP to convince her to use her
voice. At first, she was also too shy to try SpokeIt, but
once we started the game, she started speaking
without the SLP’s encouragement.
Discussion and Future Work
The speech recognition system accuracy can be
improved greatly. We are curious about how effectively
it can differentiate between correct and incorrect
speech. If the system can accurately diagnose speech
impairment details, it would save SLPs a lot of time that
could be used for helping the child improve.
A speech therapy game has the potential to track
progress for each child. It may be able to grade
children’s speech and report progress to SLPs. The data
can be studied to look for demographic patterns and
Figure 6: Sample output of PCG
while it continues learning how to
create background art
Figure 7: Indicator that displays
when the game is listening for
input. Indicator changes colors
when loudness thresholds are
met by the player
Work in Progress/Late Breaking
IDC 2017, June 27–30, 2017, Stanford, CA, USA
correlations to other aspects of the child’s
SpokeIt may make it easier for SLPs to assign unique
curriculums for each child’s unique speech goals. It can
assign target words and phrases that will have the
most impact on the child.
This material is based in part upon work supported by
the National Science Foundation under Grant number
#1617253. We also thank Doctor Travis Tollefson and
SLP Christina Roth for their aid in conducting user
evaluations. Any opinions, findings, and conclusions or
recommendations expressed in this material are those
of the authors and do not necessarily reflect the views
of the National Science Foundation.
1. Becker-Asano, C., & Wachsmuth, I. (2010).
Affective computing with primary and
secondary emotions in a virtual
human. Autonomous Agents and Multi-Agent
Systems, 20(1), 32.
2. Boyle, E., Connolly, T. M., & Hainey, T. (2011).
The role of psychology in understanding the
impact of computer games. Entertainment
Computing, 2(2), 69-74.
3. Conati, C., & Zhou, X. (2002, June). Modeling
students’ emotions from cognitive appraisal in
educational games. In International Conference
on Intelligent Tutoring Systems (pp. 944-954).
Springer Berlin Heidelberg.
4. Dickey, M. D. (2005). Engaging by design: How
engagement strategies in popular computer
and video games can inform instructional
design. Educational Technology Research and
Development, 53(2), 67-83.
5. Dondlinger, M. J. (2007). Educational video
game design: A review of the literature.
Journal of applied educational technology,
4(1), 21-31.
6. Facts about Cleft Lip and Cleft Palate. (2015,
November 12). Retrieved March 25, 2017, from
7. Garris, R., Ahlers, R., & Driskell, J. E. (2002).
Games, motivation, and learning: A research
and practice model. Simulation & gaming,
33(4), 441-467.
8. Habgood, M. J., & Ainsworth, S. E. (2011).
Motivating children to learn effectively:
Exploring the value of intrinsic integration in
educational games. The Journal of the Learning
Sciences, 20(2), 169-206.
9. Isbister, K. (2016). How Games Move Us:
Emotion by Design. MIT Press.
10. Sauvé, L. (2010). Effective educational games.
Educational gameplay and simulation
environments: Case studies and lessons
learned, 27-50.
11. Speech Development – Cleft Palate Foundation.
(n.d.). Retrieved March 25, 2017, from
12. Sri Kurniawan, Su-hua Wang, Christina Roth,
Travis Tollefson. (2016). CHS: Small: Game for
Cleft Speech Therapy (No. 1617253). National
Science Foundation.
13. Quick Statistics About Voice, Speech,
Language. (2016, May 19). Retrieved April 01,
2017, from
Figure 8: Original high fidelity
prototype of SpokeIt actively
using conversational speech
Figure 9: Example challenge
where child must say “Fire”
Work in Progress/Late Breaking
IDC 2017, June 27–30, 2017, Stanford, CA, USA
... Our lab has produced multiple iterations of speech therapy games, shown in Figure 2, 3, and 4, that have resulted in five publications, and three games [1,2,3,4,5]. SpokeIt is our newest iteration and has recently been highlighted in an accepted publication to IDC [1]. ...
... Our lab has produced multiple iterations of speech therapy games, shown in Figure 2, 3, and 4, that have resulted in five publications, and three games [1,2,3,4,5]. SpokeIt is our newest iteration and has recently been highlighted in an accepted publication to IDC [1]. SpokeIt was chosen to be presented in the Three-Minute Thesis Challenge, Grad Slam, at UCSC. ...
... The studies have yielded extremely valuable insights into making games usable and accessible for individuals with different physical and mental disabilities. We have learned that storybook style games are the preferred style for practicing speech and have validated that our games motivate practice more than existing curriculums [1]. We have designed our characters, shown in Figure 5, and plot to be as engaging as possible, which has been validated by users in our studies. ...
Conference Paper
Full-text available
A lack of intrinsic motivation to practice speech is attributed to tedious and repetitive speech curriculums, but mobile games have been widely recognized as a valid motivator for jaded individuals. SpokeIt is an interactive storybook style speech therapy game that intends to turn practicing speech into a motivating and productive experience for individuals with speech impairments as well as provide speech therapists an important diagnostic tool. In this paper, I discuss the novel intellectual contributions SpokeIt can provide such as an offline critical conversational speech recognition system, and the application of therapy curriculums to mobile platforms, I present conducted research, and consider exciting future work and research directions.
... PCG have has applied in EG approaching a varied set of subjects, such as math [Rodrigues et al. 2017, Smith et al. 2012, Butler et al. 2015, Lara et al. 2018, computing [Horn et al. 2016, Dong and Barnes 2017, Valls-Vargas et al. 2017, Dezani et al. 2017], resource management [Aslam et al. 2017, Luo et al. 2017, Grappiolo et al. 2011, evolution [Soule et al. 2017], therapy [Duval et al. 2017], and reading [Hooshyar et al. 2018]. In those studies, PCG was mostly used to generate levels or its components (e.g. ...
... The exceptions are the cases where the algorithms were analyzed [Soule et al. 2017, Smith et al. 2012, Valls-Vargas et al. 2017, Dong and Barnes 2017 or no user evaluation was performed [Dezani et al. 2017]. In contrast, works reporting user-based studies mainly evaluated users' behavior [Butler et al. 2015], affective states' changes [Lara et al. 2018], learning gains [Hooshyar et al. 2018, Rodrigues et al. 2017, Horn et al. 2016, in-game performance [Aslam et al. 2017], opinions [Duval et al. 2017], and the proposed approach's performance [Luo et al. 2017, Grappiolo et al. 2011]. ...
Full-text available
Understanding which and how attributes impact player experience can contribute to designing more tailored tools, providing concerns on how to improve these. However, there is a gap in the understanding of what impacts learners' experience when interacting with Educational Games (EG) featuring Procedural Content generation (PCG) as these have been scantly used together. This article presents an empirical study on which attributes impact both math's and game's curiosity of players when interacting with an EG that uses PCG. The results show the attributes that led to higher or lower curiosity, as well as which of them are associated with it. Hence, advancing the understanding of what drives players' curiosity, contributing to the design of EG that feature PCG.
... Este artículo presenta un juego de video que propone actividades lúdicas que, siguiendo una orientación por procesos, se enfoca a que los niños puedan practicar habilidades de observación que les permitan tener las bases necesarias para llegar a realizar posteriormente dichas descripciones estáticas, diferenciando esta propuesta de otras propuestas de herramienta encontradas en la literatura consultada en la formulación del proyecto (O'Bryan, et al., 2012;Tan, Johnston, Ballard, Ferguson, & Perera-Schulz, 2013;Navarro-Newball, et al., 2014;Cano, Muñoz Arteaga, Collazos, & Bustos Amador, 2015;Duval, 2017a;Duval, et al., 2017b;Wauck, 2017). ...
... En la propuesta de Tan et al. (2014), el sistema realimentaba al paciente utilizando juegos de arcada (es decir, juegos clásicos similares a los disponibles en máquinas recreativas de centros comerciales, como por ejemplo Pac Man), pero se limitaba al reconocimiento de vocales. Duval et al. (2017b) y Duval (2017a) presentan un sistema móvil que considera reconocimiento en tiempo real de producciones orales conversacionales. Aquí, se evidencian nuevamente los beneficios del uso de los videojuegos y retos como el soportar la responsabilidad de la práctica en el videojuego y no en los padres del paciente y; cómo adicionar palabras y frases que se ajusten al creciente conocimiento de cada paciente. ...
Conference Paper
Full-text available
Al realizar procesos de (re)habilitación de niños con pérdida auditiva oralizados, es importante ofrecer una intervención terapéutica que favorezca tanto al desarrollo del lenguaje como los procesos cognitivos, y a la vez, ofrecer a las familias las herramientas necesarias para llevar a buen término dichos procesos. Aplicar técnicas de ludificación en actividades del proceso de terapia, puede motivar a los niños y permitir que se desempeñen de una forma más adecuada. Estas actividades podrían plantearse de forma que cumplan con las dinámicas utilizadas en los videojuegos y permitan cumplir los objetivos planteados para cada actividad dentro del proceso de intervención terapéutica. El proceso de selección de las actividades adecuadas para trabajar la habilidad de observación requerida para desarrollar la descripción estática, y su transformación en actividades lúdicas en las que es posible controlar sus objetivos, interacción con los usuarios y dificultad, permite crear una herramienta lúdica que apoya el proceso de terapia del lenguaje realizada por los fonoaudiólogos. Este proyecto es una propuesta de trabajo interdisciplinar de diseño y desarrollo de un videojuego con actividades para ejercitar habilidades de observación requeridas en la descripción estática, en el proceso de desarrollo de la narrativa en el niño con pérdida auditiva oralizado.
... While games are popular in education because of their motivational and simulation potential (see for example Costanza et al., 2014;Duval et al., 2017;Garris et al., 2002;Hertzog et al., 2014;The Thiagi Group, 2019), their specific structure is also exploited for data collection. ...
The authors developed and used a game ( Tigo-Tigo) to collect data on people’s emotional experience in their environment in an area hit by a typhoon (Philippines). With the aim of encouraging the use of games for data collection in the field, they provide an in-depth analysis of all phases of the process, from the game development to the experience of the game sessions and the quality of the data produced. Designing a data collection game is creating an immersive experience that get people to share information with the researcher. However challenging to develop as it has to meet both data gathering and game requirements, Tigo-Tigo successfully produced complex data and a positive experience. By following its simple rules, the respondents were led to formulate and share both quantitative (emotion levels) and qualitative (explanations for emotion-environment associations) data. Moreover, the game was motivating and changed the status of participation, as the researchers played with the respondents in an inversed power setting. Finally, its particular interactional structure also improved the quality of the data produced by reducing expectation as well as cultural and translation barriers encountered in the field.
... Some major barriers to develop games that facilitate communication for this population include technology limitations and challenges in knowledge representation. Current technology for automatic speech recognition lacks sufficient accuracy to assess impairments in children's speech production [29,30] and requires prolonged training to understand individual voices [31]. Furthermore, existing research in computational linguistics have not developed robust natural language processing methods to analyze speech production from games. ...
Speech language pathologists (SLPs) work with children with diverse types of disabilities. Their clinical knowledge and experience using mobile technology can inform child–computer researchers novel design techniques to facilitate communication and learning for children with disabilities. This paper presents perspectives from an SLP through an overview of design challenges and opportunities for children with communication impairments across disciplines such as instructional design, game design, and therapy design. By illustrating two clinical decision-making models and implementing a design-based research framework, designers and researchers may benefit from collaborating with SLPs for clinically-informed participatory design practices. Preliminary results from an ongoing case study and future research agenda is also described.
... With the widespread use of mobile technology among modern youth, concerns about excessive "screen time" spent for entertainment purposes and its potential detrimental effects on children's communication development, have drawn public debates and research interests from different stakeholders such as health professionals, parents, and technology inventors. Many researchers have attempted to leverage the ubiquitous mobile technology through designing mobile games using automatic speech recognition to increase children's motivation to participate in speech therapy tasks 7,9 , however, existing progress on these applications are still under development and have not yet reached clinical success. Commercialized mHealth products such as Cognoa 5 have succeeded in using machine learning to detect early signs of autism through parental questionnaires and home video screening 1 . ...
Full-text available
Early identification and intervention of speech and language delays in children contribute to better communication and literacy skills for school readiness and are protective against behavioral and mental health problems. Through collaboration between the data science and clinical teams at Cognoa, we designed Storytime, an interactive storytelling experience on a mobile device using a virtual avatar to mediate speech and language screening for children ages 4 to 6 years old. Our proof-of-concept study collects Storytime session footage from 71 pairs of parents and children including 57 typically developing children and 14 children with a current or prior history of communication impairments. Initial findings suggest that participating children verbally engaged with the video avatar without significant differences in performance across age, gender, and experimental location, leading to promising implications for using Storytime as a future tracking tool with automated feature analyses to detect speech and language delays.
... Adults with developmental disabilities co-occurring with speech impairment would benefit from speech therapy [38]. There is a trend towards helping children with speech impairments [6,8,12,41,43,49,51], but adults with speech impairments need support as well [44,45]. This support can come in the form of a speech recognition system, discussed below. ...
Conference Paper
Full-text available
SpokeIt is a mobile serious game for health designed to support speech articulation therapy. Here, we present SpokeIt as well as 2 preceding speech therapy prototypes we built, all of which use a novel offline critical speech recognition system capable of providing feedback in real-time. We detail key design motivations behind each of them and report on their potential to help adults with speech impairment co-occurring with developmental disabilities. We conducted a qualitative within-subject comparative study on 5 adults within this target group, who played all 3 prototypes. This study yielded refined functional requirements based on user feedback, relevant reward systems to implement based on user interest, and insights on the preferred hybrid game structure, which can be useful to others designing mobile games for speech articulation therapy for a similar target group.
... Balancing offline speech recognition and a game engine is challenging considering the limited processing power of mobile devices. Predecessors of SpokeIt [2,3] that use similar technology include Rubin's Speech Adventure, and Speech with Sam [8,9]. We use Apple's SpriteKit and Swift 3, native to IOS devices, and RapidEars, written in Objective C, for the development of our game. ...
Conference Paper
Full-text available
We present SpokeIt, a novel speech therapy game co-created with users, medical experts, speech pathologists, developmental psychologists, and game designers. Our serious game for health aims is to augment and support in-office speech therapy sessions with an at-home supported speech therapy experience. SpokeIt was co-created using participatory methods involving multiple stakeholders and target users. We describe the technical details of SpokeIt, the process of working with multiple stakeholders, and the methods that allowed us to create medium fidelity prototypes in real-time with players. We share our emerging designs, tools, insights for the broader CHI audience, and plans for future work.
Digital games can make speech therapy exercises more enjoyable for children and increase their motivation during therapy. However, many such games developed to date have not been designed for long-term use. To address this issue, we developed Apraxia World, a speech therapy game specifically intended to be played over extended periods. In this study, we examined pronunciation improvements, child engagement over time, and caregiver and automated pronunciation evaluation accuracy while using our game over a multi-month period. Ten children played Apraxia World at home during two counterbalanced 4-week treatment blocks separated by a 2-week break. In one treatment phase, children received pronunciation feedback from caregivers and in the other treatment phase, utterances were evaluated with an automated framework built into the game. We found that children made therapeutically significant speech improvements while using Apraxia World, and that the game successfully increased engagement during speech therapy practice. Additionally, in offline mispronunciation detection tests, our automated pronunciation evaluation framework outperformed a traditional method based on goodness of pronunciation scoring. Our results suggest that this type of speech therapy game is a valid complement to traditional home practice.
Full-text available
Therapy games have the potential to offer people with disabilities a cost-effective, personalized, data-driven, connected, and motivating context for otherwise tedious and repetitive therapy. The paramount challenge in creating therapy games is creating a motivating experience with mechanics that translate into improved health outcomes---a wicked problem. To this end, I use research through design to explore multiple approaches to the co-creation of therapy games for various populations, including children with speech impairments, adults with developmental disabilities, children with Sensory-Based Motor Disorder (SBMD), and stroke survivors. I have collaborated on 3 therapy games, which serve as case studies where I explore identifying best practices, unique insights, and suggestions for future therapy game creators. Specifically, I discuss game-first versus therapy-first approaches, closed-game systems versus more open-ended playful systems, and potential future research directions.
Full-text available
Much attention has been directed to the use of video games for learning in recent years, in part due to the staggering amounts of capital spent on games in the entertainment industry, but also because of their ability to captivate player attention and hold it for lengthy periods of time as players learn to master game complexities and accomplish objectives. This review of the literature on video game research focuses on publications analyzing educational game design, namely those that present design elements conducive to learning, the theoretical underpinnings of game design, and learning outcomes from video game play.
Full-text available
The concept of intrinsic motivation lies at the heart of the user engagement created by digital games. Yet despite this, educational software has traditionally attempted to harness games as extrinsic motivation by using them as a sugar coating for learning content. This article tests the concept of intrinsic integration as a way of creating a more productive relationship between educational games and their learning content. Two studies assessed this approach by designing and evaluating an educational game called Zombie Division to teach mathematics to 7- to 11-year-olds. Study 1 examined the learning gains of 58 children who played either the intrinsic, extrinsic, or control variants of Zombie Division for 2 hr, supported by their classroom teacher. Study 2 compared time on task for the intrinsic and extrinsic variants of the game when 16 children had free choice of which game to play. The results showed that children learned more from the intrinsic version of the game under fixed time limits and spent 7 times longer playing it in free-time situations. Together, these studies offer evidence for the genuine value of an intrinsic approach for creating effective educational games. The theoretical and commercial implications of these findings are discussed.
Full-text available
Computer and video games are a prevalent form of entertainment in which the purpose of the design is to engage players. Game designers incorporate a number of strategies and tactics for engaging players in “gameplay.” These strategies and tactics may provide instructional designers with new methods for engaging learners. This investigation presents a review of game design strategies and the implications of appropriating these strategies for instructional design. Specifically, this study presents an overview of the trajectory of player positioning or point of view, the role of narrative, and methods of interactive design. A comparison of engagement strategies in popular games and characteristics of engaged learning is also presented to examine how strategies of game design might be integrated into the existing framework of engaged learning.
Full-text available
We introduce the WASABI ([W]ASABI [A]ffect [S]imulation for [A]gents with [B]elievable [I]nteractivity) Affect Simulation Architecture, in which a virtual human’s cognitive reasoning capabilities are combined with simulated embodiment to achieve the simulation of primary and secondary emotions. In modeling primary emotions we follow the idea of “Core Affect” in combination with a continuous progression of bodily feeling in three-dimensional emotion space (PAD space), that is subsequently categorized into discrete emotions. In humans, primary emotions are understood as onto-genetically earlier emotions, which directly influence facial expressions. Secondary emotions, in contrast, afford the ability to reason about current events in the light of experiences and expectations. By technically representing aspects of each secondary emotion’s connotative meaning in PAD space, we not only assure their mood-congruent elicitation, but also combine them with facial expressions, that are concurrently driven by primary emotions. Results of an empirical study suggest that human players in a card game scenario judge our virtual human MAX significantly older when secondary emotions are simulated in addition to primary ones.
Full-text available
Although most agree that games can be engaging and that games can be instructive, there is little consensus regarding the essential characteristics of instructional games. Implicit in the research literature is the notion that if we pair instructional content with certain game features, we can harness the power of games to engage users and achieve desired instructional goals. In this article, the authors present an input-process- output model of instructional games and learning that elaborates (a) the key features of games that are of interest from an instructional perspective; (b) the game cycle of user judgments, behavior, and feedback that is a hallmark of engagement in game play; and (c) the types of learning outcomes that can be achieved. The authors discuss the implications of this approach for the design and implementation of effective instruc- tional games.
Games and simulations are not only a rapidly growing source of entertainment in today's world; they are also quite beneficial. They enable players to develop quick-reaction and motor skills, engage cognitive processes, and interact with peers around the globe, thereby enhancing social skills. However, as a result of the rise of games and simulations, educators are struggling to engage their students through more traditional ways of learning. Educational Gameplay and Simulation Environments: Case Studies and Lessons Learned presents a remarkable collection of cases demonstrating how to conceptualize, design, and implement games and simulations effectively for learning. This paramount publication will aid educators, researchers, and game developers in broadening their work to effectively create and implement engaging learning environments for present and future students.
This chapter argues that although educational games have not always been taken seriously, they are forms of play that offer strong interactive communication support and should be a component of 21st century education. It reports on a systematic review of studies highlighting the game elements that support motivation and learning: repetition, learning content segmentation, feedback, challenge and competition, active participation in learning, teamwork, and interaction, and illustrates these mechanisms with examples.
Over the last 40 years, computer games have become an extremely popular leisure activity and more recently there has also been interest in the potential of serious games to help in learning, skill acquisition and attitude and behaviour change. Initially public interest in computer games focused on concerns about their violent and gender stereotyped content and their potentially addictive properties, but more recently the benefits of games have also been recognised. Psychology is at the interface between science, cognitive science and social science and in this paper we examine the role that theories and research in psychology have played in understanding the impacts of playing games, the appeal of games and the potential of games in supporting learning and behaviour change.
Conference Paper
We present a probabilistic model t hat assesses s tudent emotional reaction du ring interaction with an educational game. Following a well-known cognitive theory o f emotions (the OCC theory), the model predicts a student's emotional state by assessing the student's appraisal of her interaction with the game, in light of the student's goals and personality. We illustrate how the model relies on a Dynamic Decision Network that is based on both the OCC theory and observations from two user studies. Learner motivation is a key component for the success of any pedagogical activity. Electronic games for education are learning environments that t ry to increase the learner's motivation by embedding pedagogical activities in highly engaging, game- like interactions. However, several studies show that, while these educational games are usually successful in increasing student engagement, they often fail in triggering learning (7). The studies indicate that t his happens because many students play the games without actively reasoning about the underlying instructional domain, and thus fail to learn from the game activities. To ov ercome this limitation o f educational games, we a re designing p edagogical agents that, as part of game playing, generate tailored interventions aimed at stimulating the student t o learn b etter fr om t he game. However, in o rder not t o interfere with the high level of engagement that is a key asset of educational games, these agents need to take into account the players' affective states in addition to their cognitive states when deciding how to act. Toward this end, we a re devising a probabilistic model of student affect t hat our pedagogical agents can use, along with an assessment of student learning, to generate interventions that im prove learning without compromising engagement. The model relies on a Dynamic Decision Network (DDN) (9) to p robabilistically integrate information on both the possible ca uses of affective reaction and its observable effects (e.g., a change in the student's facial expression). Leveraging any evidence available on the student's emotional state is crucial, because in this modeling task the different sources of evidence are often ambiguous, and vary significantly with both the student and each particular interaction. In this paper, we focus on the part of the model t hat reasons about t he ca uses of student's affect by relying on the OCC cognitive theory of emotions, developed by Orthony, Clore and Collins (8). The OCC theory describes a person's emotions as the result of that person's appraisal of how the c urrent situation fits with the person's goals and p references. Our DDN models a student's appraisal m echanism during interaction with an educational game. It does s o by representing ho w game e vents