Julian Hough

• PhD
• Associate Professor at Swansea University

The paper overviews the shared task on Real-Time Reverse Transliteration for Romanized Indo-Aryan languages. It focuses on the reverse transliteration of low-resourced languages in the Indo-Aryan family to their native scripts. Typing Romanized Indo-Aryan languages using ad-hoc transliterals and achieving accurate native scripts are complex and oft...

Ambiguous words are often found in modern digital communications. Lexical ambiguity challenges traditional Word Sense Disambiguation (WSD) methods, due to limited data. Consequently, the efficiency of translation, information retrieval, and question-answering systems is hindered by these limitations. This study investigates the use of Large Languag...

Social media posts and other digital communications are widespread with ambiguous words that can hold multiple meanings. Solving "lexical ambiguity" is a challenging task with traditional computation methods (Yadav, Patel and Shah, 2021). These methods struggle with limited data and a lack of contextual understanding, leading to the consequences of...

People often encounter difficulties in building shared understanding during everyday conversation. The most common symptom of these difficulties are self-repairs, when a speaker restarts, edits or amends their utterances mid-turn. Previous work has focused on the verbal signals of self-repair, i.e. speech disfluences (filled pauses, truncated words...

This paper summarizes the structure and findings from the first Workshop on Troubles and Failures in Conversations between Humans and Robots . The workshop was organized to bring together a small, interdisciplinary group of researchers working on miscommunication from two complementary perspectives. One group of technology-oriented researchers was...

Social robots have limited social competences. This leads us to view them as depictions of social agents rather than actual social agents. However, people also have limited social competences. We argue that all social interaction involves the depiction of social roles and that they originate in, and are defined by, their function in accounting for...

We present a conversational management act (CMA) annotation schema for one-to-one tutorial dialogue sessions where a tutor uses an analogy to teach a student a concept. CMAs are more fine-grained sub-utterance acts compared to traditional dialogue act markup. The schema achieves an inter-annotator agreement (IAA) Cohen Kappa score of at least 0.66...

We present two multimodal fusion-based deep learning models that consume ASR transcribed speech and acoustic data simultaneously to classify whether a speaker in a structured diagnostic task has Alzheimer's Disease and to what degree, evaluating the ADReSSo challenge 2021 data. Our best model, a BiLSTM with highway layers using words, word probabil...

Alzheimer’s disease (AD) is a progressive, neurodegenerative disorder mainly characterized by memory loss with deficits in other cognitive domains, including language, visuospatial abilities, and changes in behavior. Detecting diagnostic biomarkers that are noninvasive and cost-effective is of great value not only for clinical assessments and diagn...

This paper is a submission to the Alzheimer's Dementia Recognition through Spontaneous Speech (ADReSS) challenge, which aims to develop methods that can assist in the automated prediction of severity of Alzheimer's Disease from speech data. We focus on acoustic and natural language features for cognitive impairment detection in spontaneous speech i...

Despite the incremental nature of Dynamic Syntax (DS), the semantic grounding of it remains that of predicate logic, itself grounded in set theory, so is poorly suited to expressing the rampantly context-relative nature of word meaning, and related phenomena such as incremental judgements of similarity needed for the modelling of disambiguation. He...

We present a multi-task learning framework to enable the training of one universal incremental dialogue processing model with four tasks of disfluency detection, language modelling, part-of-speech tagging, and utterance segmentation in a simple deep recurrent setting. We show that these tasks provide positive inductive biases to each other with the...

In this paper, we explore the idea that independently developed Dynamic Syntax accounts of dialogue and interaction fit well within the general approach of radical embodied and enac-tive accounts of cognition (REEC). This approach enables a rethinking of the grounding of linguistic universal constraints, specifically tree structure restrictions, as...

We describe a study into how disfluencies are measures of communication efficacy and can indicate a demographic identity between participants in dyadic dialogue. The relative distribution of disfluency types are calculated for ternary (FF, FM, MM) gender context. Tests for association show that certain disfluency types are dependent on the demograp...

We introduce an annotation scheme and corpus study to investigate the use of base and target components of analogies in tutorial dialogues. We present the development of the scheme and test its final form on a corpus of one-to-one tutorial dialogues on computer science , for which we achieve over 0.77 multi-rater inter-annotator agreement. We then...

Head movement, and head nods in particular, are important communicative signals in face to face conversations. Listeners' head nods can be characterised as acknowledgements to the speaker or as showing support when the speaker encounters problems in completing their turn. Speakers' head movement, on the other hand, is often explained as mimicry. Th...

While flat representations of dialogue states can be useful for machine learning approaches to human-robot interaction , there is still a role for structured dialogue states classification, particularly for domains with little data. To address this, we propose a novel types-as-classifiers approach to dialogue processing for robots using probabilist...

This paper describes an initial corpus study of question-answer pairs in the Carolina Conversations Collection corpus of conversational interviews with older people. Our aim is to compare the behaviour of patients with and without Alzheimer's Disease (AD) on the basis of types of question asked and their responses in dialogue. It has been suggested...

While Voice User Interfaces (VUI) are becoming increasingly embedded into everyday life, their ability to tailor their output to individual users is limited. Research in VUIs has explored the use of static user models to encode general preferences; and, separately, dynamic models of dialogue context or short-term common ground have been used to inf...

This handbook is the first volume to provide a comprehensive, in-depth, and balanced discussion of ellipsis phenomena, whereby a perceived interpretation is fuller than would be expected based solely on the presence of linguistic forms. Natural language abounds in these apparently incomplete expressions, such as I laughed but Ed didn’t, in which th...

We present an immersive multi-person game developed for testing models of non-verbal behaviour in conversation. People interact in a virtual environment using avatars that are driven, by default, by their real-time head and hand movements. However, on the press of a button each participant's real movements can be substituted by 'fake' avatar moveme...

One of the fundamental requirements for models of semantic processing in dialogue is incrementality: a model must reflect how people interpret and generate language at least on a word-by-word basis, and handle phenomena such as fragments, incomplete and jointly-produced utterances. We show that the incremental word-by-word parsing process of Dynami...

We present a flexible, general model for a robot predicting affordances of potentially unfamiliar objects in a human-robot dialogue system with incremental speech understanding capabilities. We show how predicting affordances needs to be integrated within a general, continuous dialogue state update process in order to take advantage of dialogue-lev...

We present a collaborative virtual environment for comparing models of listening behaviours in conversation. Users can fake attention by pressing a button. While faking, automatic algorithms take control over their avatar to continuously produce socially appropriate responses. A scoring mechanism encourages both faking attention and trying to detec...

We propose a novel Types-As-Classifiers approach to dialogue processing for robots using probabilistic type judgments. In our proposal, incoming sensory data is converted to a world belief record in real time, and then derived beliefs such as intention attribution to a user, or the prediction of affordances of visible objects, are made as record ty...

Miscommunication phenomena such as repair in dialogue are important indicators of the quality of communication. Automatic detection is therefore a key step toward tools that can characterize communication quality and thus help in applications from call center management to mental health monitoring. However, most existing computational linguistic ap...

When giving descriptions, speakers often signify object shape or size with hand gestures. Such so-called 'iconic' gestures represent their meaning through their relevance to referents in the verbal content, rather than having a conventional form. The gesture form on its own is often ambiguous, and the aspect of the referent that it highlights is co...

Automatic speech recognition (asr) is not only becoming increasingly accurate, but also increasingly adapted for producing timely, incremental output. However, overall accuracy and timeliness alone are insufficient when it comes to interactive dialogue systems which require stability in the output and responsivity to the utterance as it is unfoldin...

Here we demonstrate our Intelligent Coaching Space, an immersive virtual environment in which users learn a motor action (e.g. a squat) under the supervision of a virtual coach. We detail how we assess the ability of the coachee in executing the motor action, how the intelligent coaching space and its features are realized and how the virtual coach...

For effective HRI, robots must go beyond having good legibility of their intentions shown by their actions, but also ground the degree of uncertainty they have. We show how in simple robots which have spoken language understanding capacities, uncertainty can be communicated to users by principles of grounding in dialogue interaction even without na...

We propose an incremental dialogue framework which combines probabilistic Type Theory with Records and order-theoretic models of probability. The probabilistic record type lattices at the core of the framework allow the efficient computation of type judgements of utterance meaning in situated dialogue. It models reference processing in simple refer...

Disfluencies such as self-repairs, filled pauses such as 'um' and silent pauses are pervasive in dialogue, but there is no consensus in the literature as to whether they reflect internal production pressures, or interactive issues -- or how their effects are manifest in dialogue. It is well-known that patients with schizophrenia have problems with...

In motor skill coaching interaction coaches use several techniques to improve the motor skill of the coachee. Through goal setting, explanations, instructions and feedback the coachee is motivated and guided to improve the motor skill. These verbal speech actions are often accompanied by iconic or deictic gestures and other nonverbal acts, such as...

Empirical evidence from dialogue, both corpus and experimental, highlights the importance of interaction in language use – and this raises some questions for Christiansen & Chater's (C&C's) proposals. We endorse C&C's call for an integrated framework but argue that their emphasis on local, individual production and comprehension makes it difficult...

We present the DUEL corpus, consisting of 24 hours of natural, face-to-face, loosely task-directed dialogue in German, French and Mandarin Chinese. The corpus is uniquely positioned as a cross-linguistic, multimodal dialogue resource controlled for domain. DUEL includes audio, video and body tracking data and is transcribed and annotated for disflu...

We present a multimodal coaching system that supports online motor skill learning. In this domain, closed-loop interaction between the movements of the user and the action instructions by the system is an essential requirement. To achieve this, the actions of the user need to be measured and evaluated and the system must be able to give corrective...

For dialogue systems to become robust, they must be able to detect disfluencies accurately and with minimal latency. To meet this challenge, here we frame incremental disfluency detection as a word-byword tagging task and, following their recent success in Spoken Language Understanding tasks, we test the performance of Recurrent Neural Networks (RN...

While tutorial dialogues have been well-studied, the nature of dialogue in physical coaching scenarios is much less well understood. We present a corpus study on coaching interactions wherein a coach trains a trainee to improve a motor skill. We show how our findings put novel requirements on pedagogic dialogue act tax-onomies, grounding criteria a...

In conversation, interlocutors routinely indicate whether something said or done has been processed and integrated. Such feedback includes backchannels such as 'okay' or 'mhm', the production of a next relevant turn, and repair initiation via clarification requests. Importantly, such feedback can be produced not only at sentence/turn boundaries, bu...

Truly interactive dialogue systems need to construct meaning on at least a word-byword basis. We propose desiderata for incremental semantics for dialogue models and systems, a task not heretofore attempted thoroughly. After laying out the desirable properties we illustrate how they are met by current approaches, comparing two incremental semantic...

We present STIR (STrongly Incremental Repair detection), a system that detects speech repairs and edit terms on transcripts incrementally with minimal latency. STIR uses information-theoretic measures from n-gram models as its principal decision features in a pipeline of classifiers detecting the different stages of repairs. Results on the Switchbo...

This chapter introduces the phenomenon of compound contributions in dialogue and discusses their implications for NLG in interactive systems. Contributions to dialogue are often split across multiple utterances – possibly by different speakers – with each adding to an incrementally emerging representation of meaning. A suitable NLG module must ther...

This paper describes a statistical corpus study of self-repairs in the disfluency-annotated Switchboard corpus which ex-amines the time-linear nature of self-repair processing for annotators and listeners in dialogue. The study suggests a strictly lo-cal detection and processing mechanism for self-repairs is sufficient, an advantage currently not u...

We describe a method for learning an incremental semantic grammar from a corpus in which sentences are paired with logical forms as predicate- argument structure trees. Working in the framework of Dynamic Syntax, and as- suming a set of generally available compositional mechanisms, we show how lexical entries can be learned as probabilistic procedu...

We present empirical evidence from dialogue that challenges some of the key assumptions in the Pickering & Garrod (P&G) model of speaker-hearer coordination in dialogue. The P&G model also invokes an unnecessarily complex set of mechanisms. We show that a computational implementation, currently in development and based on a simpler model, can accou...

Investigations into incidents are an important means of improving the safety and security of sociotechnical systems. Numerous models and approaches have been proposed, but little research has been done to understand the link between such theories and their practice in actual investigation. We propose an analytical framing of this link to facilitate...

We describe a method for learning an in-cremental semantic grammar from data in which utterances are paired with logical forms representing their meaning. Work-ing in an inherently incremental frame-work, Dynamic Syntax, we show how words can be associated with probabilistic procedures for the incremental projection of meaning, providing a grammar...

This paper describes recent work on the DynDial project * towards incremental semantic inter-pretation in dialogue. We outline our domain-general grammar-based approach, using a variant of Dynamic Syntax integrated with Type Theory with Records and a Davidsonian event-based seman-tics. We describe a Java-based implementation of the parser, used wit...

This document surveys the problems posed by ellipsis data, some of them very wellknown, but as a set still posing very considerable challenges and, as a proof of concept of the insights expressible by Dynamic Syntax analyses, uses the intrinsic incrementality of the DS framework to capture structural and semantic properties of elliptical fragments...