Figure 1 - uploaded by Frank Rudzicz
Content may be subject to copyright.
The standardized Cookie Theft picture (Goodglass and Kaplan 1983). 

The standardized Cookie Theft picture (Goodglass and Kaplan 1983). 

Source publication
Article
Full-text available
Alzheimer's disease (AD) is an increasingly prevalent cognitive disorder in which memory, language, and executive function deteriorate, usually in that order. There is a growing need to support individuals with AD and other forms of dementia in their daily lives, and our goal is to do so through speech-based interaction. Given that 33% of conversat...

Contexts in source publication

Context 1
... consists of narrative speech during the standard Cookie Theft picture description task from the Boston Diagnostic Aphasia Examination ( Goodglass and Kaplan 1983). In this task, an examiner shows the picture in Figure 1 to the patient, requests a description of its contents, and is permitted to periodically encourage or prompt the patient. Each speech sample was recorded and manually transcribed at the word level following the TalkBank CHAT protocol (MacWhinney 2014). ...
Context 2
... participant turns, the SLPs instead annotated both the presence of specific TIBs and whether picture elements were mentioned. Those picture descriptions relate to the state of task completion and are, if extant, either SINK (i.e., information about the right part of the picture in Figure 1) or COOKIE (i.e., information about the left part of the picture). Fleiss' kappa, κ = 0.84, indicates satisfactory agreement, so we consider the annotations of the first SLP here. ...

Citations

... In [107], a prototype application has been proposed, based on Amazon's Alexa, to provide audio prompts with routine tasks for people diagnosed with dementia. In [108], ML algorithms have been applied to identify dialogue-related confusion from speech with individuals with Alzheimer's disease; accuracies above 80% were obtained and learn policies implemented to avoid conversation breakdowns. Several linguistic features were extracted as verbal indicators of confusion (e.g., vocabulary richness, parse tree structures, and acoustic cues). ...
Article
Full-text available
Socially assistive robots (SAR) hold significant potential to assist older adults and people with dementia in human engagement and clinical contexts by supporting mental health and independence at home. While SAR research has recently experienced prolific growth, long-term trust, clinical translation and patient benefit remain immature. Affective human-robot interactions are unresolved and the deployment of robots with conversational abilities is fundamental for robustness and humanrobot engagement. In this paper, we review the state of the art within the past two decades, design trends, and current applications of conversational affective SAR for ageing and dementia support. A horizon scanning of AI voice technology for healthcare, including ubiquitous smart speakers, is further introduced to address current gaps inhibiting home use. We discuss the role of user-centred approaches in the design of voice systems, including the capacity to handle communication breakdowns for effective use by target populations. We summarise the state of development in interactions using speech and natural language processing, which forms a baseline for longitudinal health monitoring and cognitive assessment. Drawing from this foundation, we identify open challenges and propose future directions to advance conversational affective social robots for: 1) user engagement, 2) deployment in real-world settings, and 3) clinical translation.
... Moreover, most of these studies are limited to data snapshots at a fixed time, and within particular genres or domains via lab-based tasks, very often the Cookie Theft picture description task. Some datasets permit analysis of more general language: e.g. the Carolinas dataset contains less directed conversations between people with dementia and health practitioners, and has been used to build models for automatic detection of confusion [21]. Learning models for longitudinal monitoring of individuals is not possible with the current datasets as either the datasets do not contain longitudinal information or any longitudinal data are either sparse or inconsistently available for participating individuals. ...
Preprint
Full-text available
Dementia is a family of neurogenerative conditions affecting memory and cognition in an increasing number of individuals in our globally aging population. Automated analysis of language, speech and paralinguistic indicators have been gaining popularity as potential indicators of cognitive decline. Here we propose a novel longitudinal multi-modal dataset collected from people with mild dementia and age matched controls over a period of several months in a natural setting. The multi-modal data consists of spoken conversations, a subset of which are transcribed, as well as typed and written thoughts and associated extra-linguistic information such as pen strokes and keystrokes. We describe the dataset in detail and proceed to focus on a task using the speech modality. The latter involves distinguishing controls from people with dementia by exploiting the longitudinal nature of the data. Our experiments showed significant differences in how the speech varied from session to session in the control and dementia groups.
... They claim to achieve an accuracy of 0.93 in the detection of dementia. Another approach used on diagnosis of dementia is proposed by Chinaei et al. [21]. They analyze some linguistic characteristics which are considered as verbal indicators of confusion in people with dementia, these are: the richness in the vocabulary, the analysis of the structure of the syntactic tree of the sentences and acoustic signals. ...
Article
Full-text available
Reminiscence therapy is a non-pharmacological intervention that helps mitigate unstable psychological and emotional states in patients with Alzheimer’s disease, where past experiences are evoked through conversations between the patients and their caregivers, stimulating autobiographical episodic memory. It is highly recommended that people with Alzheimer regularly receive this type of therapy. In this paper, we describe the development of a conversational system that can be used as a tool to provide reminiscence therapy to people with Alzheimer’s disease. The system has the ability to personalize the therapy according to the patients information related to their preferences, life history and lifestyle. An evaluation conducted with eleven people related to patient care (caregiver = 9, geriatric doctor = 1, care center assistant = 1) shows that the system is capable of carrying out a reminiscence therapy according to the patient information in a successful manner.
... We believe that the low capacity of the model, combined with generic features, should allow such an approach to avoid the overfitting problems that would impact a deep model trained in this manner. Now that we have demonstrated that this approach is sound in principle, future work will include the determination of a full set of linguistic features appropriate for dialogue evaluation (particularly features regarding dialogue acts and dialogue breakdowns (Higashinaka et al., 2015;Chinaei et al., 2017)), to extend our existing training procedure to train for overall response quality instead of relevance, and to obtain human response quality scores for dialogues from a variety of domains so that the final set of features can be fully evaluated and compared to existing methods. ...
Preprint
Full-text available
Automatically evaluating text-based, non-task-oriented dialogue systems (i.e., `chatbots') remains an open problem. Previous approaches have suffered challenges ranging from poor correlation with human judgment to poor generalization and have often required a gold standard reference for comparison or human-annotated data. Extending existing evaluation methods, we propose that a metric based on linguistic features may be able to maintain good correlation with human judgment and be interpretable, without requiring a gold-standard reference or human-annotated data. To support this proposition, we measure and analyze various linguistic features on dialogues produced by multiple dialogue models. We find that the features' behaviour is consistent with the known properties of the models tested, and is similar across domains. We also demonstrate that this approach exhibits promising properties such as zero-shot generalization to new domains on the related task of evaluating response relevance.
... During the last few years, new sophisticated techniques from Natural Language Processing (NLP) have been used to analyse written texts, clinically elicited utterances and spontaneous speech, in order to identify signs of psychiatric or neurological disorders and to extract automatically derived linguistic features for pathologies recognition, classification and description. Computational methods have been already successfully applied to the study of linguistic cues of cerebral functional disorders, both in the case of language modifications and disruption associated with depression (Jiang et al., 2017;Stasak et al., 2019), focal brain lesions (Fergadiotis and Wright, 2011), Parkinson's disease (Arias-Vergara et al., 2018;Benba et al., 2016;Sztah o and Vicsi, 2016;Upadhya et al., 2019) and for detecting dementia prodroms (MCI) (dos Santos et al., 2017;Matsuda Toledo et al., 2018;Meil an et al., 2018;Roark et al., 2007Roark et al., , 2011Satt et al., 2013;T oth et al., 2018;Vincze et al., 2016;Wang et al., 2019) or the different associated pathologies, like Alzheimer's Disease (Chinaei et al., 2017;Fraser et al., 2016;Jarrold et al., 2014;L opez-de-Ipiña et al., 2015;Yancheva and Rudzicz, 2016;Sirts, Piguet, Johnson), PPA (Fraser et al., 2014) and Fronto-Temporal Dementia (Jarrold et al., 2014). ...
Article
Almost 50 million people are living with dementia in 2018 worldwide, and the number will double every 20 years. The effectiveness of existing pharmacologic treatments for the disease is limited to symptoms control, and none of them are able to prevent, reverse or turn off the neurodegenerative process that leads to dementia; therefore, a prompt detection of the “disease signature” is a key problem, in order to develop and test new drugs and to support the management of clinical and domestic context. Recent studies showed that linguistic alterations may be one of the earliest signs of the pathology, years before other neurocognitive deficits become evident. Traditional tests fail to identify these slight but noticeable changes; whereas, the analysis of spoken language productions by Natural Language Processing (NLP) techniques can ecologically and inexpensively identify minor language modifications in potential patients. This interdisciplinary study aims at quantifying and describing alterations of linguistic features due to cognitive decline and build an automatic system for early diagnosis and screening purpose. To this aim, we enrolled 96 participants: 48 healthy controls and 48 impaired subjects. Of the latter, 32 was diagnosed with Mild Cognitive Impairment and 16 with early Dementia (eD). Each subject underwent a brief neuropsychological screening, and samples of semi-spontaneous speech productions was collected by means of three elicitation tasks. Recorded sessions were orthographically transcribed, PoS tagged and parsed building two different corpora: in the first we kept the automatic annotations, while in the second the transcripts were manually corrected in order to remove all mistakes. A multidimensional parameter computation was performed on the data, taking into consideration a set of 87 acoustical, rhythmical, morpho-syntactic and lexical feature as well as some readability indexes and demographic information. After these preparatory steps, some automatic classifiers were trained to distinguish healthy controls from MCI subjects employing two different algorithms, Support Vector (SVC) and Random Forest Classifiers (RFC). Our system was able to distinguish between controls and MCI subjects exhibiting high F1 scores, around 75%, thus it seems to be a promising approach for the identification of preclinical stages of dementia.
... 27 In addition, a large database of multimedia interactions and transcripts, DementiaBank, 28 is available for the study of communication in dementia patients, and it was used to study NLP techniques to classify and analyze the linguistic characteristics of AD patients. [29][30][31] Despite these exciting developments in the use of EHR data for identifying PLWD, a number of critical issues are left to address. It is imperative that investigators using their own "computable phenotype" to identify PLWD in a healthcare system validate and share their approaches. ...
Article
Embedded pragmatic clinical trials (ePCTs) are embedded in healthcare systems as well as their data environments. For people living with dementia (PLWD), settings of care can be different from the general population and involve additional people whose information is also important. The ePCT designs have the opportunity to leverage data that becomes available through the normal delivery of care. They may be particularly valuable in Alzheimer's disease and Alzheimer's disease‐related dementia (AD/ADRD), given the complexity of case identification and the diversity of care settings. Grounded in the objectives of the Data and Technical Core of the newly established National Institute on Aging Imbedded Pragmatic Alzheimer's Disease and AD‐Related Dementias Clinical Trials Collaboratory (IMPACT Collaboratory), this article summarizes the state of the art in using existing data sources (eg, Medicare claims, electronic health records) in AD/ADRD ePCTs and approaches to integrating them in real‐world settings. J Am Geriatr Soc 68:S49–S54, 2020.
... A key element in extracting instances of comprehension impairment is the assumption that breakdowns of language understanding within conversation result in unexpected responses to a given comment or question. As outlined by Chinaei et al. (2017), these unexpected responses may follow certain trends, such as lack of continuation of topic or requests for repetition. In Watson (1999), those with Alzheimer's Disease (AD) were most likely to respond during comprehension difficulties by either a lack of continuation (no contribution or elaboration on the topic, or complete change of topic) or reprise with dysfluency (a partial or complete repetition of the question with frequent pauses and filler words). ...
Book
Full-text available
RaPID3@LREC2020 - Preface Welcome to the LREC2020 Workshop on "Resources and ProcessIng of linguistic, para-linguistic and extra-linguistic Data from people with various forms of cognitive/psychiatric/developmental impairments" (RaPID-3). RaPID-3 aims to be an interdisciplinary forum for researchers to share information, findings, methods, models and experience on the collection and processing of data produced by people with various forms of mental, cognitive, neuropsychiatric, or neurodegenerative impairments, such as aphasia, dementia, autism, bipolar disorder, Parkinson’s disease or schizophrenia. Particularly, the workshop’s focus is on creation, processing and application of data resources from individuals at various stages of these impairments and with varying degrees of severity. Creation of resources includes e.g. annotation, description, analysis and interpretation of linguistic, paralinguistc and extra-linguistic data (such as spontaneous spoken language, transcripts, eyetracking measurements, wearable and sensor data, etc). Processing is done to identify, extract, correlate, evaluate and disseminate various linguistic or multimodal phenotypes and measurements, which then can be applied to aid diagnosis, monitor the progression or predict individuals at risk. A central aim is to facilitate the study of the relationships among various levels of linguistic, paralinguistic and extra-linguistic observations (e.g., acoustic measures; phonological, syntactic and semantic features; eye tracking measurements; sensors, signs and multimodal signals). Submission of papers are invited in all of the aforementioned areas, particularly emphasizing multidisciplinary aspects of processing such data and the interplay between clinical/nursing/medical sciences, language technology, computational linguistics, natural language processing (NLP) and computer science. The workshop will act as a stimulus for the discussion of several ongoing research questions driving current and future research by bringing together researchers from various research communities.
... The reduction in speech expressiveness is another language dysfunction typically observed in AD patients. This reduction is measured by the decrease in adjectives and indicators related to vocabulary richness [11,39]. Using a combination of these features, previous studies have succeeded in differentiating healthy controls and AD patients [22,37]. ...
Article
Background Identifying signs of Alzheimer disease (AD) through longitudinal and passive monitoring techniques has become increasingly important. Previous studies have succeeded in quantifying language dysfunctions and identifying AD from speech data collected during neuropsychological tests. However, whether and how we can quantify language dysfunction in daily conversation remains unexplored. Objective The objective of this study was to explore the linguistic features that can be used for differentiating AD patients from daily conversations. Methods We analyzed daily conversational data of seniors with and without AD obtained from longitudinal follow-up in a regular monitoring service (from n=15 individuals including 2 AD patients at an average follow-up period of 16.1 months; 1032 conversational data items obtained during phone calls and approximately 221 person-hours). In addition to the standard linguistic features used in previous studies on connected speech data during neuropsychological tests, we extracted novel features related to atypical repetition of words and topics reported by previous observational and descriptive studies as one of the prominent characteristics in everyday conversations of AD patients. Results When we compared the discriminative power for AD, we found that atypical repetition in two conversations on different days outperformed other linguistic features used in previous studies on speech data during neuropsychological tests. It was also a better indicator than atypical repetition in single conversations as well as that in two conversations separated by a specific number of conversations. Conclusions Our results show how linguistic features related to atypical repetition across days could be used for detecting AD from daily conversations in a passive manner by taking advantage of longitudinal data.
... Automated approaches for examining conversational dynamics between people with dementia and their caregivers have been previously investigated, 16,17 and machine learning approaches to assist with communication breakdown in dementia have already been achieved to some extent. [16][17][18][19] For example, Discursis is an automated text-analytic tool that provides quantification and visualization of communication behavior between 2 or more speakers. 20,21 Discursis has previously been applied to conversations of people with dementia to identify topics that facilitate conversational engagement 17 and also to identify the effectiveness of various communication strategies used by caregivers when conversing with people with dementia. ...
Article
Full-text available
Dementia is a common neurodegenerative condition involving the deterioration of cognitive and communication skills. Pausing in the speech of people with dementia is a dysfluency that may be used to signal conversational trouble in social interaction. This study aimed to examine the speech-pausing profile within picture description samples from people with dementia and healthy controls (HCs) within the DementiaBank database using the Calpy computational speech processing toolkit. Sixty English-speaking participants between the ages of 53 and 88 years (M age = 67.43, SD = 8.33; 42 females) were included in the study: 20 participants with mild cognitive impairment, 20 participants with moderate cognitive impairment, and 20 HCs. Quantitative analysis shows a progressive increase in the duration of pausing between HCs, the mild dementia group, and the moderate dementia group, respectively.
... They recorded the interactive data of spoken dialogues and extract different audiovisual features, then two machine learning algorithms were used achieving a 0.93 detection performance rate. Chinaei et al. [2] analyzed several linguistics features that are verbal indicators of confusion in AD like vocabulary richness, parse tree structures, and acoustic cues. They applied several machine learning algorithms to identify dialog-relevant confusion from speech with up to 82% accuracy. ...