Jackson Liscombe

Purpose We investigate the extent to which automated audiovisual metrics extracted during an affect production task show statistically significant differences between a cohort of children diagnosed with autism spectrum disorder (ASD) and typically developing controls. Method Forty children with ASD and 21 neurotypical controls interacted with a mu...

We present a cloud-based multimodal dialogue platform for the remote assessment and monitoring of speech, facial and fine motor function in Parkinson's Disease (PD) at scale, along with a preliminary investigation of the efficacy of the various metrics automatically extracted by the platform. 22 healthy controls and 38 people with Parkinson's Disea...

We propose a cloud-based multimodal dialog platform for the remote assessment and monitoring of Amyotrophic Lateral Sclerosis (ALS) at scale. This paper presents our vision, technology setup, and an initial investigation of the efficacy of the various acoustic and visual speech metrics automatically extracted by the platform. 82 healthy controls an...

We explore the utility of an on-demand multimodal conversational platform in extracting speech and facial metrics in children with Autism Spectrum Disorder (ASD). We investigate the extent to which these metrics correlate with objective clinical measures, particularly as they pertain to the interplay between the affective, phonatory and motoric sub...

We demonstrate a multimodal conversational platform for remote patient diagnosis and monitoring. The platform engages patients in an interactive dialog session and automatically computes metrics relevant to speech acoustics and articulation, oro-motor and oro-facial movement , cognitive function and respiratory function. The dialog session includes...

We present NEMSI, a cloud-based multimodal dialog system designed to have naturalistic interactions with individuals for the purpose of screening neurological or mental conditions. The system has been used by thousands of people capturing audio and video responses to open-ended questions and structured health surveys. CCS CONCEPTS • Human-centered...

A system and a method to generate statistical utterance classifiers optimized for the individual states of a spoken dialog system is disclosed. The system and method make use of large databases of transcribed and annotated utterances from calls collected in a dialog system in production and log data reporting the association between the state of th...

A method and apparatus for continuously improving the performance of semantic classifiers in the scope of spoken dialog systems are disclosed. Rule-based or statistical classifiers are replaced with better performing rule-based or statistical classifiers and/or certain parameters of existing classifiers are modified. The replacement classifiers or...

The design of commercial spoken dialog systems is most commonly based on hand-crafting call flows. Voice interaction designers write prompts, predict caller responses, set speech recognition parameters, implement interaction strategies, all based on "best design practices". Recently, we presented the mathematical framework "Contender" (similar to r...

In a recent publication [1], we laid out the mathematical foundations for an optimization technique—Contender— applicable to commercially deployed spoken dialog sys-tems similar to what the research community would refer to as a light version of reinforcement learning. In particu-lar, we showed how Contender respects the notion of sta-tistical sign...

Contender (or what the academic community would refer to as a light version of reinforcement learning) is a simple technique to experiment with a number of competing paths in a (commercial) spoken dialog system. By randomly routing certain portions of traffic to individual paths and computing average rewards for each of the routes, the goal is to f...

The online prediction of task success in Interactive Voice Response (IVR) systems is a comparatively new field of research. It helps to identify problemantic calls and enables the dialog system to react before the caller gets overly frustrated. This publication investigates, to which extent it is possible to predict task completion and how existing...

Satisfying callers’ goals and expectations is the primary objective of every customer care contact center. However, quantifying how successfully interactive voice response (IVR) systems satisfy callers’ goals and expectations has historically proven to be a most difficult task. Such difficulties in assessing automated customer care contact centers...

In commercial spoken dialog systems, call flows are built by call flow designers implementing a predefined business logic. While it may appear obvious from this logic how the call flow has to look like, i.e., which pieces of information have to be gathered from the caller or back-end systems and in which sequence, there are, in fact, strong argumen...

Transcription and semantic annotation (annoscription) of utterances is crucial part of speech performance analysis and tuning of spoken dialog systems and other natural language processing disciplines. However, the fact that these are manual tasks makes them expensive and slow. In this paper, we will discuss how anno-scription can be partially auto...

The localization of speech recognition for large-scale spoken dialog systems can be a tremendous exercise. Usually, all in- volved grammars have to be translated by a language expert, and new data has to be collected, transcribed, and annotated for statistical utterance classifiers resulting in a time-c onsuming and expensive undertaking. Often tho...

In this paper we discuss the recent evolution of spoken dialog systems in commercial deployments. Yet based on a simple finite state machine design paradigm, dialog systems reached today a higher level of complexity. The availability of massive amounts of data during deployment led to the development of continuous optimization strategy pushing the...

Statistical Spoken Language Understanding grammars (SSLUs) are often used only at the top recognition contexts of modern large-scale spoken dialog systems. We propose to use SSLUs at every recognition context in a dialog system, effectively replacing conventional, manually written grammars. Further-more, we present a methodology of continuous impro...

The annotation of hundreds of thousands of utterances for the training of statistical utterance classifiers requires a careful quality assurance procedure to make the data consistent and reliable. In this paper, we present five methods to analyze different aspects of annotated data to ensure their Completeness, Consistency, Correlation, Congruence...

In this paper we introduce a subjective metric for evaluating the performance of spoken dialog systems, caller experience (CE). CE is a useful metric for tracking the overall performance of a system in deployment, as well as for isolating individual problematic calls in which the system underperforms. The proposed CE metric differs from most perfor...

Most studies on speech-based emotion recognition are based on prosodic and acoustic features, only employing artifi- cial acted corpora where the results cannot be generalized to telephone-based speech applications. In contrast, we present an approach based on utterances from 1,911 calls from a deployed telephone-based speech application, taking ad...

We present a set of metrics describing classification performance for individual contexts of a spoken dialog system as well as for the entire system. We show how these metrics can be used to train and tune system components and how they are re- lated to Caller Experience, a subjective measure describing how well a caller was treated by the dialog s...

Most studies on speech-based emotion recognition are based on prosodic and acoustic features, only employing artificial acted corpora where the results cannot be generalized to telephone-based speech applications. In contrast, we present an approach based on utterances from 1,911 calls from a deployed telephone-based speech application, taking adva...

We present a supervised machine learning approach for detecting problematic human-computer dialogs between callers and an automated agent in a call center. The proposed model can distinguish problematic from non-problematic calls after only five caller turns with an accuracy of over 90%. Based on a corpus of more than 69,000 dialogs we further empl...

Current speech-enabled Intelligent Tutoring Systems do not model student question behavior the way human tutors do, despite ev- idence indicating the importance of doing so. Our study exam- ined a corpus of spoken tutorial dialogues collected for develop- ment of ITSpoke, an Intelligent Tutoring Spoken Dialogue Sys- tem. The authors extracted proso...

Successful Intelligent Tutoring Systems (ITSs)must be able to rec- ognize when their students are asking a question. They must iden- tify question form as well as function in order to respond appro- priately. Our study examines whether intonational features, specif- ically, F0 height and rise range, are useful cues to student question type in a cor...

The goal of my proposed dissertation work is to help answer two fundamental questions: (1) How is emotion communicated in speech? and (2) Does emotion modeling improve spo- ken dialogue applications? In this paper I de- scribe feature extraction and emotion classi- cation experiments I have conducted and plan to conduct on three different domains:...

Most research that explores the emotional state of users of spo- ken dialog systems does not fully utilize the contextual nature that the dialog structure provides. This paper reports results of machine learning experiments designed to automatically clas- sify the emotional state of user turns using a corpus of 5,690 dialogs collected with the "How...

What role does affect play in spoken tutorial systems and is it automatically detectable? We investigated the classification of student certainness in a corpus collected for ITSPOKE, a speech-enabled Intelligent Tutorial System (ITS). Our study suggests that tutors respond to indications of student uncertainty differently from student certainty. Re...

This paper presents results from a study examining emotional speech using acoustic features and their use in automatic ma- chine learning classification. In addition, we propose a clas- sification scheme for the labeling of emotions on continuous scales. Our findings support those of previous research as well as indicate possible future directions...

This paper describes work evaluating the extent to which descriptors of intonation and rhythm are useful for estimating human scores of the delivery proficiency of spoken English. It is shown that distributional measurements of syllable stress and intonational phrase boundary tones (prosodic events) correlate with human ratings in a corpus of speak...

A method to assess the quality of customer service phone interactions is to point callers to an online survey where they can ex-press their opinions, wishes, complaints, commendations, etc. by way of free-form text input. This paper investigates to which extend semantic classification can be applied to large amounts of surveys (thousands) in order...