Julie M Liss

• Ph.D.
• Head of Faculty at Arizona State University

Background Mild cognitive impairment (MCI) has been recognized as a possible precursor to Alzheimer’s disease (AD). Recent research focusing on connected speech has uncovered various features strongly correlated with MCI due to AD and related dementias. Despite these advancements, the impact of early cognitive decline on articulatory precision has...

There has been a surge of interest in leveraging speech as a marker of health for a wide spectrum of conditions. The underlying premise is that any neurological, mental, or physical deficits that impact speech production can be objectively assessed via automated analysis of speech. Recent advances in speech-based Artificial Intelligence (AI) models...

Speech foundation models are remarkably successful in various consumer applications, prompting their extension to clinical use-cases. This is challenged by small clinical datasets, which precludes effective fine-tuning. We tested the efficacy of two models to classify participants by segmental (Wav2Vec2.0) and suprasegmental (Trillsson) speech anal...

This perspective article explores the challenges and potential of using speech as a biomarker in clinical settings, particularly when constrained by the small clinical datasets typically available in such contexts. We contend that by integrating insights from speech science and clinical research, we can reduce sample complexity in clinical speech A...

Purpose This study explores speech motor planning in adults who stutter (AWS) and adults who do not stutter (ANS) by applying machine learning algorithms to electroencephalographic (EEG) signals. In this study, we developed a technique to holistically examine neural activity differences in speaking and silent reading conditions across the entire co...

Objective: Although studies have shown that digital measures of speech detected ALS speech impairment and correlated with the ALSFRS-R speech item, no study has yet compared their performance in detecting speech changes. In this study, we compared the performances of the ALSFRS-R speech item and an algorithmic speech measure in detecting clinically...

Objective This research note advocates for a methodological shift in clinical speech analytics, emphasizing the transition from high-dimensional speech feature representations to clinically validated speech measures designed to operationalize clinically relevant constructs of interest. The aim is to enhance model generalizability and clinical appli...

This study investigated the impact of alignment delay between assistive listening audio signals and acoustic signals from loudspeakers in large classrooms. Focusing on speech-in-noise comprehension, subjective comprehension confidence, and listening effort, the research aimed to illuminate auditory processing challenges and suggests improvements fo...

Cigna’s online stress management toolkit includes an AI-based tool that purports to evaluate a person’s psychological stress level based on analysis of their speech, the Cigna StressWaves Test (CSWT). In this study, we evaluate the claim that the CSWT is a “clinical grade” tool via an independent validation. The results suggest that the CSWT is not...

Purpose Oral diadochokinesis is a useful task in assessment of speech motor function in the context of neurological disease. Remote collection of speech tasks provides a convenient alternative to in-clinic visits, but scoring these assessments can be a laborious process for clinicians. This work describes Wav2DDK, an automated algorithm for estimat...

Objective: We demonstrated that it was possible to predict ALS patients' degree of future speech impairment based on past data. We used longitudinal data from two ALS studies where participants recorded their speech on a daily or weekly basis and provided ALSFRS-R speech subscores on a weekly or quarterly basis (quarter-annually). Methods: Using th...

Purpose Defined as the similarity of speech behaviors between interlocutors, speech entrainment plays an important role in successful adult conversations. According to theoretical models of entrainment and research on motoric, cognitive, and social developmental milestones, the ability to entrain should develop throughout adolescence. However, litt...

Approximately 1.2% of the world's population has impaired voice production. As a result, automatic dysphonic voice detection has attracted considerable academic and clinical interest. However, existing methods for automated voice assessment often fail to generalize outside the training conditions or to other related applications. In this paper, we...

Background and hypothesis: Automated language analysis is becoming an increasingly popular tool in clinical research involving individuals with mental health disorders. Previous work has largely focused on using high-dimensional language features to develop diagnostic and prognostic models, but less work has been done to use linguistic output to a...

Assistive listening systems (ALSs) dramatically increase speech intelligibility and reduce listening effort. It is very likely that essentially everyone, not only individuals with hearing loss, would benefit from the increased signal-to-noise ratio an ALS provides in almost any listening scenario. However, ALSs are rarely used by anyone other than...

The DIVA model is a computational model of speech motor control that combines a simulation of the brain regions responsible for speech production with a model of the human vocal tract. The model is currently implemented in Matlab Simulink; however, this is less than ideal as most of the development in speech technology research is done in Python. T...

Studies have shown deep neural networks (DNN) as a potential tool for classifying dysarthric speakers and controls. However, representations used to train DNNs are largely not clinically interpretable, which limits clinical value. Here, a model with a bottleneck layer is trained to jointly learn a classification label and four clinically-interpreta...

Approximately 1.2% of the world's population has impaired voice production. As a result, automatic dysphonic voice detection has attracted considerable academic and clinical interest. However, existing methods for automated voice assessment often fail to generalize outside the training conditions or to other related applications. In this paper, we...

The DIVA model is a computational model of speech motor control that combines a simulation of the brain regions responsible for speech production with a model of the human vocal tract. The model is currently implemented in Matlab Simulink; however, this is less than ideal as most of the development in speech technology research is done in Python. T...

Spectro-temporal dynamics of consonant-vowel (CV) transition regions are considered to provide robust cues related to articulation. In this work, we propose an objective measure of precise articulation, dubbed the objective articulation measure (OAM), by analyzing the CV transitions segmented around vowel onsets. The OAM is derived based on the pos...

Background and Hypothesis:Automated language analysis is becoming an increasingly popular tool in clinical research involving individuals with mental health disorders. Previous work has largely focused on using high-dimensional language features to develop diagnostic and prognostic models, but less work has been done to use linguistic output to ass...

Spectro-temporal dynamics of consonant-vowel (CV) transition regions are considered to provide robust cues related to articulation. In this work, we propose an objective measure of precise articulation, dubbed the objective articulation measure (OAM), by analyzing the CV transitions segmented around vowel onsets. The OAM is derived based on the pos...

We developed and evaluated an automatically extracted measure of cognition (semantic relevance) using automated and manual transcripts of audio recordings from healthy and cognitively impaired participants describing the Cookie Theft picture from the Boston Diagnostic Aphasia Examination. We describe the rationale and metric validation. We develope...

Clinical assessments often use complex picture description tasks to elicit natural speech patterns and magnify changes occurring in brain regions implicated in Alzheimer's disease and dementia. As The Cookie Theft picture description task is used in the largest Alzheimer's disease and dementia cohort studies available, we aimed to create algorithms...

Background/objective Changes in speech can be detected objectively before and during migraine attacks. The goal of this study was to interrogate whether speech changes can be detected in subjects with post-traumatic headache (PTH) attributed to mild traumatic brain injury (mTBI) and whether there are within-subject changes in speech during headache...

Digital health data are multimodal and high-dimensional. A patient’s health state can be characterized by a multitude of signals including medical imaging, clinical variables, genome sequencing, conversations between clinicians and patients, and continuous signals from wearables, among others. This high volume, personalized data stream aggregated o...

Background Cognitive decline is associated with deficits in attention to tasks and attention to relevant details. We developed a metric, semantic relevance (SemR) , which is algorithmically extracted from speech and measures overlap between a picture’s content and the words used to describe the picture. In this study, we validate it in a sample tha...

Background Neuropsychological testing requires an in‐person visit with a trained administrator using standard/fixed materials. Speech‐based cognitive testing on mobile devices enables more frequent and timely test administration, but head‐to‐head comparisons of in‐person and remote versions of tests are rare. We compare responses to a well‐validate...

Background Measures of language have demonstrated value in the detection of cognitive decline. Here we automatically extracted speech (audio) and language (transcripts) features from Cookie Theft picture descriptions (BDAE) to develop two classification models separating healthy participants from those with mild cognitive impairment (MCI) and demen...

In this study, we present and provide validation data for a tool that predicts forced vital capacity (FVC) from speech acoustics collected remotely via a mobile app without the need for any additional equipment (e.g. a spirometer). We trained a machine learning model on a sample of healthy participants and participants with amyotrophic lateral scle...

Background/Objective: Changes in speech can be detected objectively before and during migraine attacks. The goal of this study was to interrogate whether speech changes can be detected in subjects with post-traumatic headache (PTH) attributed to mild traumatic brain injury (mTBI) and whether there are within-subject changes in speech during headach...

Purpose Acoustic measurement of speech sounds requires first segmenting the speech signal into relevant units (words, phones, etc.). Manual segmentation is cumbersome and time consuming. Forced-alignment algorithms automate this process by aligning a transcript and a speech sample. We compared the phoneme-level alignment performance of five availab...

Objectives: Evaluation of hypernasality requires extensive perceptual training by clinicians and extending this training on a large scale internationally is untenable; this compounds the health disparities that already exist among children with cleft. In this work, we present the objective hypernasality measure (OHM), a speech-based algorithm that...

Introduction: Changes in speech have the potential to provide important information on the diagnosis and progression of various neurological diseases. Many researchers have relied on open-source speech features to develop algorithms for measuring speech changes in clinical populations as they are convenient and easy to use. However, the repeatabil...

A Correction to this paper has been published: https://doi.org/10.1038/s41746-020-00364-6.

Bulbar deterioration in amyotrophic lateral sclerosis (ALS) is a devastating characteristic that impairs patients' ability to communicate, and is linked to shorter survival. The existing clinical instruments for assessing bulbar function lack sensitivity to early changes. In this paper, using a cohort of N = 65 ALS patients who provided regular spe...

Aim. We compared the performance of five forced-alignment algorithms on a corpus of child speech.Method. The child speech sample included 42 children between 3 and 6 years of age. The corpus was force-aligned using the Montreal Forced Aligner with and without speaker adaptive training, triphone alignment from the Kaldi speech recognition engine, th...

Objectives: Evaluation of hypernasality requires extensive perceptual training by clinicians and extending this training on a large scale internationally is untenable; this compounds the health disparities that already exist among children with cleft. In this work, we present the objective hypernasality measure (OHM), a speech analytics algorithm t...

Hypernasality is a common characteristic symptom across many motor-speech disorders. For voiced sounds, hypernasality introduces an additional resonance in the lower frequencies and, for unvoiced sounds, there is reduced articulatory precision due to air escaping through the nasal cavity. However, the acoustic manifestation of these symptoms is hig...

Purpose In our previous studies, we showed that the brain modulates the auditory system, and the modulation starts during speech planning. However, it remained unknown whether the brain uses similar mechanisms to modulate the orofacial somatosensory system. Here, we developed a novel behavioral paradigm to (a) examine whether the somatosensory syst...

Objective To determine the potential for improving amyotrophic lateral sclerosis (ALS) clinical trials by having patients or caregivers perform frequent self‐assessments at home. Methods and Participants We enrolled ALS patients into a nonblinded, longitudinal 9‐month study in which patients and caregivers obtained daily data using several differe...

Purpose Despite the import of conversational entrainment to successful spoken dialogue, the systematic characterization of this behavioral syncing phenomenon represents a critical gap in the field of speech pathology. The goal of this study was to acoustically characterize conversational entrainment in the context of dysarthria using a multidimensi...

Hypernasality is a common symptom across many motor-speech disorders. For voiced sounds, hypernasality introduces an additional resonance in the lower frequencies and, for unvoiced sounds, there is reduced articulatory precision due to air escaping through the nasal cavity. However, the acoustic manifestation of these symptoms is highly variable, m...

It is widely accepted that information derived from analyzing speech (the acoustic signal) and language production (words and sentences) serves as a useful window into the health of an individual's cognitive ability. In fact, most neuropsychological testing batteries have a component related to speech and language where clinicians elicit speech fro...

No PDF available ABSTRACT Work in tonal languages has demonstrated that distributional information is important in speech processing. Probability of syllable + tone combinations play a role in speech processing in Mandarin (Wiener and Ito, 2015, 2016; Wiener and Turnbull, 2013), and transitional probabilities of tone-bearing vowels are important in...

Purpose Subjective speech intelligibility assessment is often preferred over more objective approaches that rely on transcript scoring. This is, in part, because of the intensive manual labor associated with extracting objective metrics from transcribed speech. In this study, we propose an automated approach for scoring transcripts that provides a...

It is widely accepted that information derived from analyzing speech (the acoustic signal) and language production (words and sentences) serves as a useful window into the health of an individual's cognitive ability. In fact, most neuropsychological batteries used in cognitive assessment have a component related to speech and language where clinici...

Presented at International Conference on Acoustics, Speech, and Signal Processing (ICASSP) in Brighton, UK on May 15, 2019. A summary of the paper found at: https://ieeexplore.ieee.org/document/8683367

A key initial step in several natural language processing (NLP) tasks involves embedding phrases of text to vectors of real numbers that preserve semantic meaning. To that end, several methods have been recently proposed with impressive results on semantic similarity tasks. However, all of these approaches assume that perfect transcripts are availa...

Detecting early signs of neurodegeneration is vital for planning treatments for neurological diseases. Speech plays an important role in this context because it has been shown to be a promising early indicator of neurological decline, and because it can be acquired remotely without the need for specialized hardware. Typically, symptoms are characte...

Hypernasal speech is a common symptom across several neurological disorders; however it has a variable acoustic signature, making it difficult to quantify acoustically or perceptually. In this paper, we propose the nasal cognate distinctiveness features as an objective proxy for hypernasal speech. Our method is motivated by the observation that inc...

Several studies have shown that speech and language features, automatically extracted from clinical interviews or spontaneous discourse, have diagnostic value for mental disorders such as schizophrenia and bipolar disorder. They typically make use of a large feature set to train a classifier for distinguishing between two groups of interest, i.e. a...

The modified barium swallow study (MBSS) is a commonly used radiographic procedure for diagnosis and treatment of swallowing disorders. Despite attempts by dysphagia specialists to standardize the MBSS, most institutions have not adopted such standardized procedures. High variability of assessment patterns arguably contribute to variability of trea...

Purpose Telemedicine, used to offset disparities in access to speech-language therapy, relies on technology that utilizes compression algorithms to transmit signals efficiently. These algorithms have been thoroughly evaluated on healthy speech; however, the effects of compression algorithms on the intelligibility of disordered speech have not been...

Objective: To design an ALS clinical study in which patients are remotely recruited, screened, enrolled and then assessed via daily data collection at home by themselves or caregivers. Methods: This observational, natural-history study included two academic medical centers, one providing overall clinical management and the other overseeing compu...

A key initial step in several natural language processing (NLP) tasks involves embedding phrases of text to vectors of real numbers that preserve semantic meaning. To that end, several methods have been recently proposed with impressive results on semantic similarity tasks. However, all of these approaches assume that perfect transcripts are availa...

The largely effortless process of segmenting a continuous speech stream into words has been shown cross-linguistically to be influenced by implicit knowledge about the distributional probabilities of phonotactics, stress assignment, and word size. Work in tonal languages provides compelling evidence that tones are also likely to be a source of prob...

Automatic pronunciation evaluation plays an important role in pronunciation training and second language education. This field draws heavily on concepts from automatic speech recognition (ASR) to quantify how close the pronunciation of non-native speech is to native-like pronunciation. However, it is known that the formation of accent is related to...

Training machine learning algorithms for speech applications requires large, labeled training data sets. This is problematic for clinical applications where obtaining such data is prohibitively expensive because of privacy concerns or lack of access. As a result, clinical speech applications are typically developed using small data sets with only t...

Purpose Across the treatment literature, behavioral speech modifications have produced variable intelligibility changes in speakers with dysarthria. This study is the first of two articles exploring whether measurements of baseline speech features can predict speakers’ responses to these modifications. Methods Fifty speakers (7 older individuals a...

Purpose Behavioral speech modifications have variable effects on the intelligibility of speakers with dysarthria. In the companion article, a significant relationship was found between measures of speakers' baseline speech and their intelligibility gains following cues to speak louder and reduce rate (Fletcher, McAuliffe, Lansford, Sinex, & Liss, 2...

Reductions in spoken language complexity have been associated with the onset of various neurological disorders. The objective of this study is to analyze whether similar trends are found in professional football players who are at risk for chronic traumatic encephalopathy. We compare changes in linguistic complexity (as indexed by the type-to-token...

Purpose The strength of the relationship between vowel centralization measures and perceptual ratings of dysarthria severity has varied considerably across reports. This article evaluates methods of acoustic-perceptual analysis to determine whether procedural changes can strengthen the association between these measures. Method Sixty-one speakers...

Articulatory precision is a critical factor that influences speaker intelligibility. In this paper, we propose a new measure we call ‘articulation entropy’ that serves as a proxy for the number of distinct phonemes a person produces when he or she speaks. The method is based on the observation that the ability of a speaker to achieve an articulator...

In English, the predominance of stressed syllables as word onsets aids lexical segmentation in degraded listening conditions. Yet it is unlikely that these findings would readily transfer to languages with differing rhythmic structure. In the current study, the authors seek to examine whether listeners exploit both common word size (syllable number...

Purpose Five experiments probed auditory-visual (AV) understanding of sentences by users of cochlear implants (CIs). Method Sentence material was presented in auditory (A), visual (V), and AV test conditions to listeners with normal hearing and CI users. Results (a) Most CI users report that most of the time, they have access to both A and V info...

State-of-the-art automatic speech recognition (ASR) engines perform well on healthy speech; however recent studies show that their performance on dysarthric speech is highly variable. This is because of the acoustic variability associated with the different dysarthria subtypes. This paper aims to develop a better understanding of how perceptual dis...

This pilot study investigated the tongue pull-back (TPB) exercise to improve tongue-base retraction as well as two methods to add resistance to the TPB. Surface electromyography (sEMG) to the submental triangle was used as an indication of tongue-base activity on 13 healthy adults during: (1) saliva swallow, (2) 15 mL water swallow, (3) effortful s...