Divesh Lala

Kyoto University | Kyodai · Graduate School of Informatics

This paper introduces the human-like embodied AI interviewer which integrates android robots equipped with advanced conversational capabilities, including attentive listening, conversational repairs, and user fluency adaptation. Moreover, it can analyze and present results post-interview. We conducted a real-world case study at SIGDIAL 2024 with 42...

In human conversations, short backchannel utterances such as "yeah" and "oh" play a crucial role in facilitating smooth and engaging dialogue. These backchannels signal attentiveness and understanding without interrupting the speaker, making their accurate prediction essential for creating more natural conversational agents. This paper proposes a n...

This study examined users' behavioral differences in a large corpus of Japanese human-robot interactions, comparing interactions between a tele-operated robot and an autonomous dialogue system. We analyzed user spoken behaviors in both attentive listening and job interview dialogue scenarios. Results revealed significant differences in metrics such...

In the realm of human-AI dialogue, the facilitation of empathetic responses is important. Validation is one of the key communication techniques in psychology , which entails recognizing, understanding, and acknowledging others' emotional states, thoughts, and actions. This study introduces the first framework designed to engender empathetic dialogu...

This paper tackles the challenging task of evaluating socially situated conversational robots and presents a novel objective evaluation approach that relies on multimodal user behaviors. In this study, our main focus is on assessing the human-likeness of the robot as the primary evaluation metric. While previous research often relied on subjective...

As the aging of society continues to accelerate, Alzheimer's Disease (AD) has received more and more attention from not only medical but also other fields, such as computer science, over the past decade. Since speech is considered one of the effective ways to diagnose cognitive decline, AD detection from speech has emerged as a hot topic. Neverthel...

Spoken dialogue systems must be able to express empathy to achieve natural interaction with human users. However, laughter generation requires a high level of dialogue understanding. Thus, implementing laughter in existing systems, such as in conversational robots, has been challenging. As a first step toward solving this problem, rather than gener...

An attentive listening system for autonomous android ERICA is presented. Our goal is to realize a humanlike natural attentive listener for elderly people. The proposed system generates listener responses: backchannels, repeats, elaborating questions, assessments, and generic responses. The system incorporates speech processing using a microphone ar...

Over the past year, research in various domains, including Natural Language Processing (NLP), has been accelerated to fight against the COVID-19 pandemic, yet such research has just started on dialogue systems. In this paper, we introduce an end-to-end dialogue system which aims to ease the isolation of people under self-quarantine. We conduct a co...

Many people are now engaged in remote conversations for a wide variety of scenes such as interviewing, counseling, and consulting, but there is a limited number of skilled experts. We propose a novel framework of parallel conversations with semi-autonomous avatars, where one operator collaborates with several remote robots or agents simultaneously....

Following the success of spoken dialogue systems (SDS) in smartphone assistants and smart speakers, a number of communicative robots are developed and commercialized. Compared with the conventional SDSs designed as a human-machine interface, interaction with robots is expected to be in a closer manner to talking to a human because of the anthropomo...

We demonstrate a job interview dialogue with the autonomous android ERICA which plays the role of an interviewer. Conventional job interview dialogue systems ask only pre-defined questions. The job interview system of ERICA generates follow-up questions based on the interviewee’s response on the fly. The follow-up questions consist of two kinds of...

We address an application of engagement recognition in human-robot dialogue. Engagement is defined as how much a user is interested in the current dialogue, and keeping users engaged is important for spoken dialogue systems. In this study, we apply a real-time engagement recognition model to laboratory guide by autonomous android ERICA which plays...

Example-based dialogue systems are often used in practice because of their robustness and simple architecture. However, when these systems are given out-of-database questions that are not registered in the question-response database, they have to respond with a fixed backup response, which can make users disengaged in the dialogue. In this study, w...

A spoken dialogue system that plays the role of an interviewer for job interviews is presented. In this work, ourgoal is to implement an automated job interview system where candidates can use it as practice before the real interview.Conventional job interview systems ask only pre-defined questions, which make the dialogue monotonous andfar from hu...

Automatic dialogue response evaluator has been proposed as an alternative to automated metrics and human evaluation. However, existing automatic evaluators achieve only moderate correlation with human judgement and they are not robust. In this work, we propose to build a reference-free evaluator and exploit the power of semi-supervised training and...

Turn-taking in human-robot interaction is a crucial part of spoken dialogue systems, but current models do not allow for human-like turn-taking speed seen in natural conversation. In this work we propose combining two independent prediction models. A continuous model predicts the upcoming end of the turn in order to generate gaze aversion and fille...

The demo shows ERICA, a highly realistic female android robot, and WikiTalk, an application that helps robots to talk about thousands of topics using information from Wikipedia. The combination of ERICA and WikiTalk results in more natural and engaging human-robot conversations.

We present a dialogue system for a conversational robot, Erica. Our goal is for Erica to engage in more human-like conversation, rather than being a simple question-answering robot. Our dialogue manager integrates question-answering with a statement response component which generates dialogue by asking about focused words detected in the user’s utt...

The task of identifying when to take a conversational turn is an important function of spoken dialogue systems. The turn-taking system should also ideally be able to handle many types of dialogue, from structured conversation to spontaneous and unstructured discourse. Our goal is to determine how much a generalized model trained on many types of di...

Engagement represents how much a user is interested in and willing to continue the current dialogue. Engagement recognition will provide an important clue for dialogue systems to generate adaptive behaviors for the user. This paper addresses engagement recognition based on multimodal listener behaviors of backchannels, laughing, head nodding, and e...

This article addresses the estimation of engagement level based on the listener’s behaviors such as backchannel, laughing, head nodding, and eye-gaze. Engagement is defined as the level of how much a user is being interested in and willing to continue the current interaction. When the engagement level is evaluated by multiple annotators, the criter...

Detection of engagement during a conversation is an important function of human-robot interaction. The level of user engagement can influence the dialogue strategy of the robot. Our motivation in this work is to detect several behaviors which will be used as social signal inputs for a real-time engagement recognition model. These behaviors are nodd...

Human beings have an ability to transition smoothly between individual and collaborative activities and to recognize these types of activity in other humans. Our long-term goal is to devise an agent which can function intelligently in an environment with frequent switching between individual and collaborative tasks. A basketball scenario is such an...

We address the annotation of engagement in the context of human-machine interaction. Engagement represents the level of how much a user is being interested in and willing to continue the current interaction. The conversational data used in the annotation work is a human-robot interaction corpus where a human subject talks with the android ERICA, wh...

We demonstrate an interactive conversation with an android named ERICA. In this demonstration the user can converse with ERICA on a number of topics. We demonstrate both the dialog management system and the eye gaze behavior of ERICA used for indicating attention and turn taking.

Research on embodied teammate agents which use dialog and gesture to coordinate their activities with the user is relatively sparse compared to conversational agents. We propose a dialog management model to handle interactions between user and agent in a virtual basketball environment. The model describes how a joint action should be initialized an...

Visual inspection of medical imagery such as MRI and CT scans is a major task for medical professionals who must diagnose and treat patients without error. Given this goal, visualizing search behavior patterns used to recognize abnormalities in these images is of interest. In this paper we describe the development of a system which automatically ge...

One barrier to creating truly intelligent autonomous virtual characters is the lack of common ground knowledge contained between the agent and its human interaction partner. Predefining this knowledge for the agent is infeasible so such agents generally can only interact within a limited task domain. This is particularly true for conversational age...

Believability is necessary for agents to establish intimate, real-time collaborations with humans in an interactive game environment. In this paper, the authors model sophisticated interaction patterns to improve believability by adapting Herbert Clark's joint activity theory. The authors use virtual basketball as an environment, where many communi...

Synthetic evidential study (SES) is a novel approach to understanding and augmenting collective thought process through substantiation by interactive media. It consists of a role-play game by participants, projecting the resulting play into a shared virtual space, critical discussions with mediated role-play, and componentization for reuse. We pres...

Synthetic evidential study (SES for short) is a novel technology-enhanced methodology for combining theatrical role play and group discussion to help people spin stories by bringing together partial thoughts and evidences. SES not only serves as a methodology for authoring stories and games but also exploits the framework of game framework to help...

In this paper, we describe a virtual basketball game where a human and an embodied agent can play together as a team. Our goal is to investigate whether the human prefers an agent who is highly competent at basketball or one which is not as competent but tries to actively communicate through body movements. The virtual basketball game was implement...

Virtual environments are a medium in which humans can effectively interact; however, until recently, research on body expression in these worlds has been sparse. This has changed with the recent development of markerless motion capture. This paper is a first step toward using this technology as part of an investigation into a collaborative task in...

A navigable mixed reality system where humans and agents can communicate and interact with each other in a virtual environment can be an appropriate tool for analyzing multi-human and multi-agent communication. We propose a prototype of our system, FCWorld, which has been developed to meet these requirements. FCWorld integrates various technologies...

In order to produce agents which are effective social actors, behavior must be modeled in an appropriate way. Models exist for a wide range of agent components, but this paper focuses on communication through body expression. Additionally, rather than formulating communication models from scratch, this paper discusses modeling of agents based on ex...

Spatial information plays an important role in social interaction with people. The ICIE (Immersive Collaborative Interaction Environment) is a platform which can present socio-spatial information, obtain human behavior with noncontact sensors, and have components to interpret socio-spatial information. In this paper, we explain the framework of ICI...

Creating agents which utilize natural verbal and non-verbal communication is an appropriate goal for many researchers involved in human-computer interaction. Using these types of agents enhances their capabilities as a communication tool for teaching humans inside a virtual environment. This paper describes how Herbert Clark's theory of joint activ...

Difficulties in living in a different culture are caused by different patterns of thinking, feeling and potential actions. People who enter into a new culture or unfamiliar social situation don't know how to behave toward other people. Queuing is a good example behavior of intercultural interaction in a human crowd. This research aims to develop a...

Virtual environments are a medium in which humans can effectively interact with each other, however up until recently research on body expression in these worlds has been sparse. This has changed with the recent development of markerless motion capture. This paper is a first step towards using these technologies as part of an investigation into a c...

In this paper we present VISIE, a software used to create immersive environments which utilize social and cultural interaction. We use the concept of a spatially immersive display to project information about the virtual world to the user in all directions. The user is able to interact in this world using spatial cognition as they do in the real wo...

Spacial information plays an important role in social interaction with people. The ICIE is a platform which can present socio-spacial information, obtain human behavior with non-contact sensors and have components to interpret the socio-spacial information. In this paper, we explain the framework of ICIE and main architectures to capture human beha...

Cultural behavior is an area of research that can allow us to further cross-cultural understanding, and is now starting to integrate itself within the field of information technology. One domain that expresses these behaviors is inside a crowd, however the analysis of micro-level crowd behavior is impractical in a real-world setting as passive obse...

Difficulties in living in a different culture are caused by different patterns of thinking, feeling and potential actions. A good way to experience cultural immersion is to walk in a crowd. This paper proposes a simulated crowd as a novel tool for allowing people to practice culture-specific nonverbal communication behaviors. We present a conceptua...

In this paper we present VISIE, a software used to create immersive environments which utilize social and cultural interaction. We use the concept of a spatially immersive display to project information about the virtual world to the user in all directions. The user is able to interact in this world using spatial cognition as they do in the real wo...