Conference PaperPDF Available

A mobile game system for improving the speech therapy experience

September 2017

DOI:10.1145/3098279.3119925

Conference: the 19th International Conference

Authors:

Northern Arizona University

A lack of intrinsic motivation to practice speech is attributed to tedious and repetitive speech curriculums, but mobile games have been widely recognized as a valid motivator for jaded individuals. SpokeIt is an interactive storybook style speech therapy game that intends to turn practicing speech into a motivating and productive experience for individuals with speech impairments as well as provide speech therapists an important diagnostic tool. In this paper, I discuss the novel intellectual contributions SpokeIt can provide such as an offline critical conversational speech recognition system, and the application of therapy curriculums to mobile platforms, I present conducted research, and consider exciting future work and research directions.

Content uploaded by Jared Duval

Content may be subject to copyright.

A Mobile Game System for Improving

the Speech Therapy Experience

Author Keywords

Speech Processing; Speech Therapy; Human Computer

Interaction; Games; Mobile Applications

ACM Classification Keywords

H. Information Systems; H.5. Information interfaces

and presentation (e.g., HCI); H.5.2. Voice I/O; I/O J.3.

Life and Medical Sciences: Health

Abstract

A lack of intrinsic motivation to practice speech is

attributed to tedious and repetitive speech curriculums,

but mobile games have been widely recognized as a

valid motivator for jaded individuals. SpokeIt is an

interactive storybook style speech therapy game that

intends to turn practicing speech into a motivating and

productive experience for individuals with speech

impairments as well as provide speech therapists an

important diagnostic tool. In this paper, I discuss the

novel intellectual contributions SpokeIt can provide

such as an offline critical conversational speech

recognition system, and the application of therapy

curriculums to mobile platforms, I present conducted

research, and consider exciting future work and

research directions.

Introduction

SpokeIt, shown in Figure 1, is a large collaborative

interdisciplinary project that incorporates the medical

expertise from UC Davis’s medical team, developmental

psychologists from UCSC, and my team of researchers

in the Interactive Systems for Individuals with Special

Needs Lab at UCSC [7]. We hope to create a solution

that speech language pathologists (SLPs) can use with

their patients and offer the following functionality:

 A diagnostic tool that assists SLPs in listening to

the patient's speech

Permission to make digital or hard copies of part or all of this work for

personal or classroom use is granted without fee provided that copies are

not made or distributed for profit or commercial advantage and that

copies bear this notice and the full citation on the first page. Copyrights

or third-party components of this work must be honored. For all other

uses, contact the Owner/Author.

MobileHCI '17, September 04-07, 2017, Vienna, Austria

ACM 978-1-4503-5075-4/17/09.

https://doi.org/10.1145/3098279.3119925

Jared Duval

University of California Santa Cruz

Santa Cruz, CA 95060, USA

jduval@ucsc.edu

 A critical offline speech recognition system that

accurately scores speech, responds accordingly,

and records progress data made accessible to the

SLP

 A dynamic curriculum that adjusts practice session

difficulty according to performance

 A patient profiling system that assigns words and

phrases to target each individual’s unique speech

goals

Research Goals, Questions, and Approach

The repetitive nature of speech therapy curriculums

often leads to unmotivated individuals, but games have

been shown to make repetition a fun and necessary

component required for progress in the game [6]. To

make a speech therapy game we first were interested

in validating that we could employ an offline accurate

real-time speech recognition system that was capable

of being used as game input [3]. Now that we have

one, we have many research questions:

 What style of game is most motivating for

individuals with speech impairments?

 Can we verify that our solution scores speech

accurately enough for use as a diagnostic tool?

 How do we incorporate therapeutic input

mechanics that induce flow and immersion rather

than distract from gameplay?

 Can we verify that our solution leads to improved

speech?

 Can we improve the efficiency of time spent in-

office with SLPs?

 Can SpokeIt provide insight into phonological

versus phonetic errors?

 Can procedural content generation effectively

produce content for as long as therapy is required?

 Can our games be abstracted in a way that allows

for other sensory inputs and mechanics to be

employed for other types of therapy?

The number of novel contributions SpokeIt is capable of

producing is exciting. As development continues and

high fidelity prototypes are iterated, we actively employ

user-centered design by including individuals with

speech impairments as well as SLPs and our

interdisciplinary team in the design of SpokeIt. We

incorporate feedback from experts and employ

practices from credible contributions to the field. We

test our speech therapy solutions on a diverse range of

individuals with speech impairments in both short and

long term studies. We regularly meet within our

interdisciplinary team to collaborate and ensure that

everyone’s research questions are being appropriately

addressed.

Studies, Conducted Research, and Results

Our lab has produced multiple iterations of speech

therapy games, shown in Figure 2, 3, and 4, that have

resulted in five publications, and three games

[1,2,3,4,5]. SpokeIt is our newest iteration and has

recently been highlighted in an accepted publication to

IDC [1]. SpokeIt was chosen to be presented in the

Three-Minute Thesis Challenge, Grad Slam, at UCSC.

In the past year, SpokeIt has been tested in user

studies on four children with cleft speech at the UC

Davis medical center, eight individuals with varying

physical and mental disabilities at Hope Services, and

three Stoke Survivors at Cabrillo College Stroke Center.

We will soon be submitting to the TAACCESS journal.

The studies have yielded extremely valuable insights

into making games usable and accessible for individuals

with different physical and mental disabilities. We have

Figure 1: SpokeIt

Figure 2: Speech with Sam

Minigames

Figure 3: Speech Adventure

learned that storybook style games are the preferred

style for practicing speech and have validated that our

games motivate practice more than existing

curriculums [1]. We have designed our characters,

shown in Figure 5, and plot to be as engaging as

possible, which has been validated by users in our

studies.

Remaining Research Efforts

There is massive potential for research opportunities as

we continue to develop SpokeIt. We are currently

building a sandbox level creator so researchers with no

coding experience can build levels for SpokeIt. We are

iterating on our procedural art generator, shown in

Figure 6, that utilizes evolutionary algorithms to learn

how to make game art. We are developing a better

data visualization and scoring system for reporting

speech data. We hope to design a narrative generator

that will allow SpokeIt to be available indefinitely to

accommodate different speech therapy lengths. We

hope to learn what other therapy applications SpokeIt

can be used for.

Expected Results

We plan to continue to use user-centered design

throughout our development, but once development

has ended, we hope to run long term studies that

answer the questions listed above. We believe that

SpokeIt will motivate individuals to practice speech

more than conventional methods, will make in-office

visits with SLPs more efficient, can be applied to other

forms of therapy, and produce guidelines for designing

therapeutic games that employ procedural content

generation.

References

1. Duval, J. in press. Designing Towards Maximum

Motivation and Engagement in an Interactive

Speech Therapy Game. ACM.

2. Rubin, Z. 2017. Development and evaluation of

software tools for speech therapy (Doctoral

dissertation, University of California, Santa Cruz).

3. Rubin, Z., & Kurniawan, S. 2013. Speech

adventure: using speech recognition for cleft

speech therapy. In Proceedings of the 6th

International Conference on PErvasive Technologies

Related to Assistive Environments (p. 35). ACM.

4. Rubin, Z., Kurniawan, S., & Tollefson, T. 2014.

Results from Using Automatic Speech Recognition

in Cleft Speech Therapy with Children. In

International Conference on Computers for

Handicapped Persons (pp. 283-286). Springer

International Publishing.

5. Rubin, Z., Kurniawan, S., Gotfrid, T., & Pugliese, A.

2016. Motivating Individuals with Spastic Cerebral

Palsy to Speak Using Mobile Speech Recognition. In

Proceedings of the 18th International ACM

SIGACCESS Conference on Computers and

Accessibility (pp. 325-326). ACM.

6. Sauvé, L. 2010. Effective educational games.

Educational gameplay and simulation

environments: Case studies and lessons learned,

27-50.

7. Sri Kurniawan, Su-hua Wang, Christina Roth, Travis

Tollefson. 2016. CHS: Small: Game for Cleft

Speech Therapy (No. 1617253). National Science

Foundation.

Figure 4: SpokeIt Prototype

Figure 5: SpokeIt race of

characters known as Migs

Figure 6: Procedural Art

Generator

Chasing Play on TikTok from Populations with Disabilities to Inspire Playful and Inclusive Technology Design

Conference Paper

Full-text available

May 2021

There is an open call for technology to be more playful and for the design of tech to be more inclusive of people with disabilities. In the era of COVID19, it is often unsafe for the public in general and people with disabilities in particular to engage in in-person design exercises using traditional methods. This presents a missed opportunity as these populations are already sharing playful content that is rich with tacit design knowledge that can be used to inspire the design of playful everyday technology. We present our process of scraping play potentials from TikTok from content creators with disabilities to generate design concepts that may inspire the design of future technology. We share 7 emerging themes from the scraped content, a catalog of design concepts that may inspire designers, and a discussion on the relevance of the emerging themes and possible implications for the design concepts.

Using Connected Learning Design Principles to Further Co-Create a Critical Speech Therapy Game

Conference Paper

Full-text available

Dec 2019

Therapy can be costly, time-consuming, repetitive, and difficult. Games have the power to teach transferable skills, can turn repetitive tasks into engaging mechanics, have been proven to be effective at delivering various forms of therapy, and can be deployed at large scales. Therapy games represent fertile ground for connected learning. In this work, we collaborate with 7 children with corrected cleft palate aged 2-10 and their parents during their yearly visit to UC Davis Medical center to co-create and evaluate SpokeIt, a speech therapy game. Each of these children come from low socioeconomic statuses with limited access to speech therapy and would benefit from the amplified opportunities of new media in connected learning. Throughout the study, we ran multiple cascading participatory design sessions using design principles of connected learning, which culminated in the design of two new medium-fidelity prototypes presented in this paper.

SpokeIt: A Co-Created Speech Therapy Experience

Conference Paper

Full-text available

Apr 2018

We present SpokeIt, a novel speech therapy game co-created with users, medical experts, speech pathologists, developmental psychologists, and game designers. Our serious game for health aims is to augment and support in-office speech therapy sessions with an at-home supported speech therapy experience. SpokeIt was co-created using participatory methods involving multiple stakeholders and target users. We describe the technical details of SpokeIt, the process of working with multiple stakeholders, and the methods that allowed us to create medium fidelity prototypes in real-time with players. We share our emerging designs, tools, insights for the broader CHI audience, and plans for future work.

Un videojuego para apoyar la terapia del lenguaje: el caso de la descripción estática

Conference Paper

Full-text available

Nov 2017

Al realizar procesos de (re)habilitación de niños con pérdida auditiva oralizados, es importante ofrecer una intervención terapéutica que favorezca tanto al desarrollo del lenguaje como los procesos cognitivos, y a la vez, ofrecer a las familias las herramientas necesarias para llevar a buen término dichos procesos. Aplicar técnicas de ludificación en actividades del proceso de terapia, puede motivar a los niños y permitir que se desempeñen de una forma más adecuada. Estas actividades podrían plantearse de forma que cumplan con las dinámicas utilizadas en los videojuegos y permitan cumplir los objetivos planteados para cada actividad dentro del proceso de intervención terapéutica. El proceso de selección de las actividades adecuadas para trabajar la habilidad de observación requerida para desarrollar la descripción estática, y su transformación en actividades lúdicas en las que es posible controlar sus objetivos, interacción con los usuarios y dificultad, permite crear una herramienta lúdica que apoya el proceso de terapia del lenguaje realizada por los fonoaudiólogos. Este proyecto es una propuesta de trabajo interdisciplinar de diseño y desarrollo de un videojuego con actividades para ejercitar habilidades de observación requeridas en la descripción estática, en el proceso de desarrollo de la narrativa en el niño con pérdida auditiva oralizado.

Approaches for creating therapy games

Article

Full-text available

Mar 2020

Jared Duval

Therapy games have the potential to offer people with disabilities a cost-effective, personalized, data-driven, connected, and motivating context for otherwise tedious and repetitive therapy. The paramount challenge in creating therapy games is creating a motivating experience with mechanics that translate into improved health outcomes---a wicked problem. To this end, I use research through design to explore multiple approaches to the co-creation of therapy games for various populations, including children with speech impairments, adults with developmental disabilities, children with Sensory-Based Motor Disorder (SBMD), and stroke survivors. I have collaborated on 3 therapy games, which serve as case studies where I explore identifying best practices, unique insights, and suggestions for future therapy game creators. Specifically, I discuss game-first versus therapy-first approaches, closed-game systems versus more open-ended playful systems, and potential future research directions.

Designing Towards Maximum Motivation and Engagement in an Interactive Speech Therapy Game

Conference Paper

Full-text available

Jun 2017

Children with speech impairments often find speech curriculums tedious, limiting how often children are motivated to practice. A speech therapy game has the potential to make practice fun, may help facilitate increased time and quality of at-home speech therapy, and lead to improved speech. We explore using conversational real-time speech recognition, game methodologies theorized to improve immersion and flow, and user centered approaches to design an immersive interactive speech therapy solution. Our preliminary user evaluation showed that compared to traditional methods, children were more motivated to practice speech using our system.

Results from Using Automatic Speech Recognition in Cleft Speech Therapy with Children

Conference Paper

Jul 2014
Lect Notes Comput Sci

Most children with cleft are required to undertake speech therapy after undergoing surgery to repair their craniofacial defect. However, the untrained ear of a parent can lead to incorrect practice resulting in the development of compensatory structures. Even worse, the boring nature of the cleft speech therapy often causes children to abandon home exercises and therapy altogether. We have developed a simple recognition system capable of detecting impairments on the phoneme level with high accuracy. We embed this into a game environment and provide it to a cleft palate specialist team for pilot testing with children 2 to 5 years of age being evaluated for speech therapy. The system consistently detected cleft speech in high-pressure consonants in 3 out of our 5 sentences. Doctors agreed that this would improve the quality of therapy outside of the office. Children enjoyed the game overall, but grew bored due to the delays of phrase-based speech recognition.

Motivating Individuals with Spastic Cerebral Palsy to Speak Using Mobile Speech Recognition

Conference Paper

Oct 2016

Individuals with cerebral palsy (CP) struggle with conditions such as dysarthria, dysphagia, and dyspraxia as they speak. While speech therapy is successful in practice, outside practice requires increased commitment and effort from caregivers. Researchers developed a speech recognition game designed to encourage out-of-office exercises and motivate users to practice. Next they recruited a participant with cerebral palsy to investigate the performance of the system in a live environment. The participant joined the game after demonstration from the caregiver and temporarily increased speech loudness and clarity during play. The participant found sound effects more rewarding than animations. The total number of sentences spoken during the session was found to be less than half that of a speaker without any impairment. Researchers also observed two instances of cheating. This work provides insight into the automated motivation of motivating speech production with individuals with cerebral palsy.

Educational Gameplay and Simulation Environments: Case Studies and Lessons Learned

Book

Jan 2010

Games and simulations are not only a rapidly growing source of entertainment in today's world; they are also quite beneficial. They enable players to develop quick-reaction and motor skills, engage cognitive processes, and interact with peers around the globe, thereby enhancing social skills. However, as a result of the rise of games and simulations, educators are struggling to engage their students through more traditional ways of learning. Educational Gameplay and Simulation Environments: Case Studies and Lessons Learned presents a remarkable collection of cases demonstrating how to conceptualize, design, and implement games and simulations effectively for learning. This paramount publication will aid educators, researchers, and game developers in broadening their work to effectively create and implement engaging learning environments for present and future students.

Effective Educational Games

Article

Jan 2010

Louise Sauvé

This chapter argues that although educational games have not always been taken seriously, they are forms of play that offer strong interactive communication support and should be a component of 21st century education. It reports on a systematic review of studies highlighting the game elements that support motivation and learning: repetition, learning content segmentation, feedback, challenge and competition, active participation in learning, teamwork, and interaction, and illustrates these mechanisms with examples.

Speech adventure: using speech recognition for cleft speech therapy

Conference Paper

May 2013

Children with cleft palate or lip learn to speak in a way that compensates for the cleft. Their speech becomes unintelligible even after surgery to correct the cleft. Most children undergo speech therapy for many years after the surgery. The boring nature of the cleft speech therapy often causes children to abandon home exercises. We posit that a game that can effectively motivate these children to practice at home would speed up recovery. However, currently, very little has been done in gaining an understanding of the children's, parents' and speech pathologists' needs and requirements. This paper discusses the design and development of the speech recognition system. From this work we developed a speech recognition system capable of detecting the omissions, substitutions, and compensations characteristic in cleft speech with extremely high accuracy.

Development and evaluation of software tools for speech therapy (Doctoral dissertation University of California Santa Cruz). Rubin Z. 2017. Development and evaluation of software tools for speech therapy

Z Rubin

Rubin, Z. 2017. Development and evaluation of software tools for speech therapy (Doctoral dissertation, University of California, Santa Cruz).

CHS: Small: Game for Cleft Speech Therapy (No. 1617253) National Science Foundation

Sri Kurniawan
Su-Hua Wang
Christina Roth
Travis Tollefson

Designing Towards Maximum Motivation and Engagement in an Interactive Speech Therapy Game

J Duval
In

A mobile game system for improving the speech therapy experience

Abstract

Recommended publications

Designing Towards Maximum Motivation and Engagement in an Interactive Speech Therapy Game

SpokeIt: A Co-Created Speech Therapy Experience

SpokeIt: building a mobile speech therapy experience

Motivating Individuals with Spastic Cerebral Palsy to Speak Using Mobile Speech Recognition