Julian Hough

Julian Hough
Queen Mary, University of London | QMUL · School of Electronic Engineering and Computer Science

PhD

About

67
Publications
7,752
Reads
How we measure 'reads'
A 'read' is counted each time someone views a publication summary (such as the title, abstract, and list of authors), clicks on a figure, or views or downloads the full-text. Learn more
620
Citations
Introduction
Julian Hough currently works at the School of Electronic Engineering and Computer Science, Queen Mary, University of London. Julian does research in Artificial Intelligence and Computational Linguistics.
Additional affiliations
January 2018 - present
Queen Mary University of London
Position
  • Lecturer
April 2014 - January 2018
Bielefeld University
Position
  • Research Associate
January 2013 - April 2013
Queen Mary, University of London
Position
  • Research Assistant
Description
  • RISER project

Publications

Publications (67)
Conference Paper
Full-text available
We present a conversational management act (CMA) annotation schema for one-to-one tutorial dialogue sessions where a tutor uses an analogy to teach a student a concept. CMAs are more fine-grained sub-utterance acts compared to traditional dialogue act markup. The schema achieves an inter-annotator agreement (IAA) Cohen Kappa score of at least 0.66...
Preprint
Full-text available
We present two multimodal fusion-based deep learning models that consume ASR transcribed speech and acoustic data simultaneously to classify whether a speaker in a structured diagnostic task has Alzheimer's Disease and to what degree, evaluating the ADReSSo challenge 2021 data. Our best model, a BiLSTM with highway layers using words, word probabil...
Article
Full-text available
Alzheimer’s disease (AD) is a progressive, neurodegenerative disorder mainly characterized by memory loss with deficits in other cognitive domains, including language, visuospatial abilities, and changes in behavior. Detecting diagnostic biomarkers that are noninvasive and cost-effective is of great value not only for clinical assessments and diagn...
Preprint
Full-text available
This paper is a submission to the Alzheimer's Dementia Recognition through Spontaneous Speech (ADReSS) challenge, which aims to develop methods that can assist in the automated prediction of severity of Alzheimer's Disease from speech data. We focus on acoustic and natural language features for cognitive impairment detection in spontaneous speech i...
Article
Full-text available
Despite the incremental nature of Dynamic Syntax (DS), the semantic grounding of it remains that of predicate logic, itself grounded in set theory, so is poorly suited to expressing the rampantly context-relative nature of word meaning, and related phenomena such as incremental judgements of similarity needed for the modelling of disambiguation. He...
Preprint
Full-text available
We present a multi-task learning framework to enable the training of one universal incremental dialogue processing model with four tasks of disfluency detection, language modelling, part-of-speech tagging, and utterance segmentation in a simple deep recurrent setting. We show that these tasks provide positive inductive biases to each other with the...
Conference Paper
Full-text available
In this paper, we explore the idea that independently developed Dynamic Syntax accounts of dialogue and interaction fit well within the general approach of radical embodied and enac-tive accounts of cognition (REEC). This approach enables a rethinking of the grounding of linguistic universal constraints, specifically tree structure restrictions, as...
Conference Paper
Full-text available
We describe a study into how disfluencies are measures of communication efficacy and can indicate a demographic identity between participants in dyadic dialogue. The relative distribution of disfluency types are calculated for ternary (FF, FM, MM) gender context. Tests for association show that certain disfluency types are dependent on the demograp...
Conference Paper
Full-text available
We introduce an annotation scheme and corpus study to investigate the use of base and target components of analogies in tutorial dialogues. We present the development of the scheme and test its final form on a corpus of one-to-one tutorial dialogues on computer science , for which we achieve over 0.77 multi-rater inter-annotator agreement. We then...
Conference Paper
Full-text available
Head movement, and head nods in particular, are important communicative signals in face to face conversations. Listeners' head nods can be characterised as acknowledgements to the speaker or as showing support when the speaker encounters problems in completing their turn. Speakers' head movement, on the other hand, is often explained as mimicry. Th...
Article
Full-text available
While flat representations of dialogue states can be useful for machine learning approaches to human-robot interaction , there is still a role for structured dialogue states classification, particularly for domains with little data. To address this, we propose a novel types-as-classifiers approach to dialogue processing for robots using probabilist...
Conference Paper
Full-text available
This paper describes an initial corpus study of question-answer pairs in the Carolina Conversations Collection corpus of conversational interviews with older people. Our aim is to compare the behaviour of patients with and without Alzheimer's Disease (AD) on the basis of types of question asked and their responses in dialogue. It has been suggested...
Conference Paper
Full-text available
While Voice User Interfaces (VUI) are becoming increasingly embedded into everyday life, their ability to tailor their output to individual users is limited. Research in VUIs has explored the use of static user models to encode general preferences; and, separately, dynamic models of dialogue context or short-term common ground have been used to inf...
Conference Paper
Full-text available
We present an immersive multi-person game developed for testing models of non-verbal behaviour in conversation. People interact in a virtual environment using avatars that are driven, by default, by their real-time head and hand movements. However, on the press of a button each participant's real movements can be substituted by 'fake' avatar moveme...
Preprint
Full-text available
One of the fundamental requirements for models of semantic processing in dialogue is incrementality: a model must reflect how people interpret and generate language at least on a word-by-word basis, and handle phenomena such as fragments, incomplete and jointly-produced utterances. We show that the incremental word-by-word parsing process of Dynami...
Conference Paper
Full-text available
We present a flexible, general model for a robot predicting affordances of potentially unfamiliar objects in a human-robot dialogue system with incremental speech understanding capabilities. We show how predicting affordances needs to be integrated within a general, continuous dialogue state update process in order to take advantage of dialogue-lev...
Poster
Full-text available
We present a collaborative virtual environment for comparing models of listening behaviours in conversation. Users can fake attention by pressing a button. While faking, automatic algorithms take control over their avatar to continuously produce socially appropriate responses. A scoring mechanism encourages both faking attention and trying to detec...
Conference Paper
Full-text available
We propose a novel Types-As-Classifiers approach to dialogue processing for robots using probabilistic type judgments. In our proposal, incoming sensory data is converted to a world belief record in real time, and then derived beliefs such as intention attribution to a user, or the prediction of affordances of visible objects, are made as record ty...
Article
Full-text available
Miscommunication phenomena such as repair in dialogue are important indicators of the quality of communication. Automatic detection is therefore a key step toward tools that can characterize communication quality and thus help in applications from call center management to mental health monitoring. However, most existing computational linguistic ap...
Conference Paper
Full-text available
When giving descriptions, speakers often signify object shape or size with hand gestures. Such so-called 'iconic' gestures represent their meaning through their relevance to referents in the verbal content, rather than having a conventional form. The gesture form on its own is often ambiguous, and the aspect of the referent that it highlights is co...
Chapter
Full-text available
Automatic speech recognition (asr) is not only becoming increasingly accurate, but also increasingly adapted for producing timely, incremental output. However, overall accuracy and timeliness alone are insufficient when it comes to interactive dialogue systems which require stability in the output and responsivity to the utterance as it is unfoldin...
Conference Paper
Full-text available
Here we demonstrate our Intelligent Coaching Space, an immersive virtual environment in which users learn a motor action (e.g. a squat) under the supervision of a virtual coach. We detail how we assess the ability of the coachee in executing the motor action, how the intelligent coaching space and its features are realized and how the virtual coach...
Conference Paper
Full-text available
For effective HRI, robots must go beyond having good legibility of their intentions shown by their actions, but also ground the degree of uncertainty they have. We show how in simple robots which have spoken language understanding capacities, uncertainty can be communicated to users by principles of grounding in dialogue interaction even without na...
Chapter
Full-text available
We propose an incremental dialogue framework which combines probabilistic Type Theory with Records and order-theoretic models of probability. The probabilistic record type lattices at the core of the framework allow the efficient computation of type judgements of utterance meaning in situated dialogue. It models reference processing in simple refer...
Conference Paper
Full-text available
Disfluencies such as self-repairs, filled pauses such as 'um' and silent pauses are pervasive in dialogue, but there is no consensus in the literature as to whether they reflect internal production pressures, or interactive issues -- or how their effects are manifest in dialogue. It is well-known that patients with schizophrenia have problems with...
Conference Paper
In motor skill coaching interaction coaches use several techniques to improve the motor skill of the coachee. Through goal setting, explanations, instructions and feedback the coachee is motivated and guided to improve the motor skill. These verbal speech actions are often accompanied by iconic or deictic gestures and other nonverbal acts, such as...
Conference Paper
Full-text available
We present the DUEL corpus, consisting of 24 hours of natural, face-to-face, loosely task-directed dialogue in German, French and Mandarin Chinese. The corpus is uniquely positioned as a cross-linguistic, multimodal dialogue resource controlled for domain. DUEL includes audio, video and body tracking data and is transcribed and annotated for disflu...
Article
Empirical evidence from dialogue, both corpus and experimental, highlights the importance of interaction in language use – and this raises some questions for Christiansen & Chater's (C&C's) proposals. We endorse C&C's call for an integrated framework but argue that their emphasis on local, individual production and comprehension makes it difficult...
Conference Paper
Full-text available
We present a multimodal coaching system that supports online motor skill learning. In this domain, closed-loop interaction between the movements of the user and the action instructions by the system is an essential requirement. To achieve this, the actions of the user need to be measured and evaluated and the system must be able to give corrective...
Conference Paper
Full-text available
For dialogue systems to become robust, they must be able to detect disfluencies accurately and with minimal latency. To meet this challenge, here we frame incremental disfluency detection as a word-byword tagging task and, following their recent success in Spoken Language Understanding tasks, we test the performance of Recurrent Neural Networks (RN...
Conference Paper
Full-text available
While tutorial dialogues have been well-studied, the nature of dialogue in physical coaching scenarios is much less well understood. We present a corpus study on coaching interactions wherein a coach trains a trainee to improve a motor skill. We show how our findings put novel requirements on pedagogic dialogue act tax-onomies, grounding criteria a...
Conference Paper
Full-text available
In conversation, interlocutors routinely indicate whether something said or done has been processed and integrated. Such feedback includes backchannels such as 'okay' or 'mhm', the production of a next relevant turn, and repair initiation via clarification requests. Importantly, such feedback can be produced not only at sentence/turn boundaries, bu...
Conference Paper
Full-text available
Truly interactive dialogue systems need to construct meaning on at least a word-byword basis. We propose desiderata for incremental semantics for dialogue models and systems, a task not heretofore attempted thoroughly. After laying out the desirable properties we illustrate how they are met by current approaches, comparing two incremental semantic...
Article
Full-text available
We present STIR (STrongly Incremental Repair detection), a system that detects speech repairs and edit terms on transcripts incrementally with minimal latency. STIR uses information-theoretic measures from n-gram models as its principal decision features in a pipeline of classifiers detecting the different stages of repairs. Results on the Switchbo...
Chapter
Full-text available
This chapter introduces the phenomenon of compound contributions in dialogue and discusses their implications for NLG in interactive systems. Contributions to dialogue are often split across multiple utterances – possibly by different speakers – with each adding to an incrementally emerging representation of meaning. A suitable NLG module must ther...
Conference Paper
Full-text available
This paper describes a statistical corpus study of self-repairs in the disfluency-annotated Switchboard corpus which ex-amines the time-linear nature of self-repair processing for annotators and listeners in dialogue. The study suggests a strictly lo-cal detection and processing mechanism for self-repairs is sufficient, an advantage currently not u...
Conference Paper
Full-text available
We describe a method for learning an incremental semantic grammar from a corpus in which sentences are paired with logical forms as predicate- argument structure trees. Working in the framework of Dynamic Syntax, and as- suming a set of generally available compositional mechanisms, we show how lexical entries can be learned as probabilistic procedu...
Article
We present empirical evidence from dialogue that challenges some of the key assumptions in the Pickering & Garrod (P&G) model of speaker-hearer coordination in dialogue. The P&G model also invokes an unnecessarily complex set of mechanisms. We show that a computational implementation, currently in development and based on a simpler model, can accou...
Article
Investigations into incidents are an important means of improving the safety and security of sociotechnical systems. Numerous models and approaches have been proposed, but little research has been done to understand the link between such theories and their practice in actual investigation. We propose an analytical framing of this link to facilitate...
Conference Paper
Full-text available
We describe a method for learning an in-cremental semantic grammar from data in which utterances are paired with logical forms representing their meaning. Work-ing in an inherently incremental frame-work, Dynamic Syntax, we show how words can be associated with probabilistic procedures for the incremental projection of meaning, providing a grammar...
Article
Full-text available
This paper describes recent work on the DynDial project * towards incremental semantic inter-pretation in dialogue. We outline our domain-general grammar-based approach, using a variant of Dynamic Syntax integrated with Type Theory with Records and a Davidsonian event-based seman-tics. We describe a Java-based implementation of the parser, used wit...
Article
Full-text available
This document surveys the problems posed by ellipsis data, some of them very wellknown, but as a set still posing very considerable challenges and, as a proof of concept of the insights expressible by Dynamic Syntax analyses, uses the intrinsic incrementality of the DS framework to capture structural and semantic properties of elliptical fragments...

Network

Cited By