Zachary Rubin’s research while affiliated with University of California, Santa Cruz and other places

SpokeIt is a mobile serious game for health designed to support speech articulation therapy. Here, we present SpokeIt as well as 2 preceding speech therapy prototypes we built, all of which use a novel offline critical speech recognition system capable of providing feedback in real-time. We detail key design motivations behind each of them and report on their potential to help adults with speech impairment co-occurring with developmental disabilities. We conducted a qualitative within-subject comparative study on 5 adults within this target group, who played all 3 prototypes. This study yielded refined functional requirements based on user feedback, relevant reward systems to implement based on user interest, and insights on the preferred hybrid game structure, which can be useful to others designing mobile games for speech articulation therapy for a similar target group.

In this paper, we present All the Feels, a tool that provides an overlay of biometric and webcam-derived data onto the interface of the popular video game streaming service, Twitch. This overlay provides visualization of heart rate, skin conductance, and facial emotion recognition. It is intended to enhance the spectator experience and improve streamer-spectator connection. We present results of a preliminary study investigating spectator reactions to the introduction of this tool, via a live test with a streamer. Analysis of the stream combined with survey responses from 17 spectators indicates enhanced enjoyment of the stream, and suggests that spectators may feel more connected to the streamer. This work opens up opportunities to improve stream quality and spectator engagement through the introduction of an interface that includes biometric signals.

Children with speech impairments often find speech curriculums tedious, limiting how often children are motivated to practice. A speech therapy game has the potential to make practice fun, may help facilitate increased time and quality of at-home speech therapy, and lead to improved speech. We explore using conversational real-time speech recognition, game methodologies theorized to improve immersion and flow, and user centered approaches to design an immersive interactive speech therapy solution. Our preliminary user evaluation showed that compared to traditional methods, children were more motivated to practice speech using our system.

All the Feels is a tool developed to incorporate biometric data into online gameplay streams. The tool displays heart rate and GSR (galvanic skin response) information to stream spectators, taken from the player in real-time, through the use of a commercially available wearable device. The tool also provides emotion labeling information input from existing auto-detection software that uses the player's webcam. In this paper, we motivate and discuss the initial prototype of the tool, and present results of a preliminary study with stream watchers and streamers to assess the potential of this novel addition to streaming gameplay.

This study describes development of an iPad-based speech therapy game to improve speech accuracy in children with cleft lip and palate. Children with cleft palate (CP) require multidisciplinary care, including palatoplasty and speech therapy (ST). There is currently an unmet need for ST for children with CP in resource-poor and rural areas owing to financial burden and shortages of transportation and health care professionals.¹ Poor speech in these children, compounded by the aesthetic stigmata of cleft lip and/or palate, can have social and economic consequences in adulthood.

Most children with cleft are required to undertake speech therapy after undergoing surgery to repair their craniofacial defect. However, the untrained ear of a parent can lead to incorrect practice resulting in the development of compensatory structures. Even worse, the boring nature of the cleft speech therapy often causes children to abandon home exercises and therapy altogether. We have developed a simple recognition system capable of detecting impairments on the phoneme level with high accuracy. We embed this into a game environment and provide it to a cleft palate specialist team for pilot testing with children 2 to 5 years of age being evaluated for speech therapy. The system consistently detected cleft speech in high-pressure consonants in 3 out of our 5 sentences. Doctors agreed that this would improve the quality of therapy outside of the office. Children enjoyed the game overall, but grew bored due to the delays of phrase-based speech recognition.

Design patterns are meant to provide a rubric of problem-solving strategies for common design problems in software engineering. This paper presents the patterns that 34 commercial video games have come to use in representing the birth of a child, and asks, "Where do babies come from?", "What are babies for?", and "How do you support a virtual mother (in labor)?" to gauge how deeply the design patterns have entrenched themselves in modern video games. We found few commonalities between the 34 games; indeed, we found that design patterns as solutions to open problems do not seem to exist for childbirth scenes in games. However, patterns of depicting birth, the mother, and the child, do exist. We suggest that these patterns are a crutch -- causing more harm than good for the players of the games by the inaccurate and incomplete scenes surrounding birth.