Patrick Gebhard

Deutsches Forschungszentrum für Künstliche Intelligenz | DFKI · Cognitive Assistants

Backchannels, i.e. short interjections of the listener, serve important meta-conversational purposes like signifying attention or indicating agreement. Despite their key role, automatic analysis of backchannels in group interactions has been largely neglected so far. The MultiMediate challenge addresses, for the first time, the tasks of backchannel...

Backlash is a form of social penalty occurring when people behave counter-stereotypically. When promoting themselves, dominant females compared to males are typically liked less and paid worse, because dominance is associated with males, and proscribed for females. Such backlash effects have been shown in human-human interactions, but attempts to r...

Computational modeling of human emotions is daunting, as emotions have a behavioral component represented by social signals and an internal component mainly reflecting individual experiences (Schneeberger et al., 2021). The challenge is how these two components are connected? We propose a terminology of emotions aiming to facilitate an understandin...

Cover Caption: The cover image is based on the Review Article Physical and mental well‐being of cobot workers: A scoping review using the Software‐Hardware‐Environment‐Liveware‐Liveware‐Organization model by Fabio A. Storm et al., https://doi.org/10.1002/hfm.20952

Job interviews are usually high-stakes social situations where professional and behavioral skills are required for a satisfactory outcome. Professional job interview trainers give educative feedback about the shown behavior according to common standards. This feedback can be helpful concerning the improvement of behavioral skills needed for job int...

In today's world, many patients with cognitive impairments and motor dysfunction seek the attention of experts to perform specific conventional therapies to improve their situation. However, due to a lack of neurorehabilitation professionals, patients suffer from severe effects that worsen their condition. In this paper, we present a technological...

Highly automated vehicles represent one of the most crucial development efforts in the automotive industry. In addition to the use of research vehicles, production vehicles for the general public are realistic in the near future. However, to fully exploit the benefits of these systems, it is fundamental that users have an appropriate level of trust...

The present scoping review investigated the current state of the art concerning factors affecting physical and mental health and well‐being of workers using collaborative robots (cobots) in manufacturing industries. Each identified factor was classified using the SHELLO (Software‐Hardware‐Environment‐Liveware‐Liveware‐Organization) conceptual model...

In the last decade, empirical sciences have faced a tremendous change in the way of conducting research. As a broad interdisciplinary field, research in Affective Computing often employs empirical user studies. The current paper analyzes research practices in Affective Computing and deduces recommendations for improving the quality of methods and r...

This chapter examines how notions of trust and empathy can be applied to human–robot interaction and how it can be used to create the next generation of emphatic agents, which address some of the pressing issues in multicultural aging societies: to harness cutting-edge AI research to develop solutions and new services. First, some examples of possi...

Understanding emotions is key to Affective Computing. Emotion recognition focuses on the communicative component of emotions encoded in social signals. This view alone is insufficient for deeper understanding and computational representation of the internal, subjectively experienced component of emotions. This paper presents the DEEP method as a st...

Computational emotion recognition focuses on observable expressions. In the case of highly unpleasant emotions that are rarely displayed openly and mostly unconsciously regulated -- such as shame -- this approach can be difficult. In previous studies, we found participants to smile and laugh while experiencing shame. Most current approaches interpr...

This paper presents an overview of AVASAG; an ongoing applied-research project developing a text-to-sign-language translation system for public services. We describe the scientific innovation points (geometry-based SL-description, 3D animation and video corpus, simplified annotation scheme, motion capture strategy) and the overall translation pipel...

Co-regulation in groups is essential for learning and may hold the key to training regulation skills. However, there is little possibility to train social co-regulation skills in authentic social situations. Existing systems mostly concentrate on training para-and nonverbal skills. While co-regulation in scripted scenarios may work at the behaviora...

The acceptance of driving automation is influenced by user trust. Integrating virtual agents into the interface between automation and user has been found to increase trust, also in driving automation. In this context, however, it remains unclear what virtual agents’ characteristics influence trust. This paper presents a pilot study about how inexp...

Using Social Agents as health-care assistants or trainers is one focus area of IVA research. While their use as physical health-care agents is well established, their employment in the field of psychotherapeutic care comes with daunting challenges. This paper presents our mobile Social Agent EmmA in the role of a vocational reintegration assistant...

In this paper, we focus on experience-based role play with virtual agents to provide young adults at the risk of exclusion with social skill training. We present a scenario-based serious game simulation platform. It comes with a social signal interpretation component, a scripted and autonomous agent dialog and social interaction behavior model, and...

Using Social Agents as health-care assistants or trainers is one focus area of IVA research. This paper presents a concept of our mobile Social Agent EmmA in the role of a vocational reintegration assistant for burn-out outpatient treatment. We follow a typical par- ticipatory design approach including experts and patients in order to address requi...

Virtual agents usually come in a virtual environment that can be designed in various ways which might affect users. This paper presents a study that examines whether the design of the virtual environment has an impact on the assessment of the virtual agent and the interaction. In a virtual job interview training, participants interacted with a virt...

Human interaction partners can deal with interruptions and then resume the interaction. This ability should be emulated by social agents. How fast interruptions are handled might influence the overall impression of an agent. In this paper, we present the results of a user study on how a human dialog partner perceives the be- havior of a virtual age...

In this demo, we present PARLEY, an interactive system to train difficult social situations in a safe environment with a Virtual Agent. The system realizes different phenomena studied by psychology research that are known to create a natural interaction. Moreover, we include an open learner model to ensure an explainable user experience.

Virtual agents play an important role when we interact with machines. They are in the role of assistants or companions with less or more human-like appearance. Such agents influence our behavior. With an increasing and broader distribution, their influence might become stronger, and at some point, they might even adopt roles with a degree of author...

Understanding emotions of others requires a theory of mind ap- proach providing knowledge of internal appraisal and regulation processes of emotions. Multi-modal social signal classification is insufficient for understanding emotional expressions. Mainly, be- cause many communicative, emotional expressions are not directly related to internal emoti...

Expanding research on employment interview training, this study introduces virtual employment interview (VI) training with focus on nonverbal behavior. In VI training, participants took part in a simulated interview with a virtual character. Simultaneously, the computer analyzed participants’ nonverbal behavior and provided real-time feedback for i...

Technology-enhanced learning environments are designed to help users practise social skills. In this paper, we present and evaluate a virtual job interview training game which has been adapted to the special requirements of young people with low chances on the job market. The evaluation spanned three days, during which we compared the technology-en...

The outcome of interpersonal interactions depends not only on the contents that we communicate verbally, but also on nonverbal social signals. As a lack of social skills is a common problem for a significant number of people, serious games and other training environments have recently become the focus of research. In this work we present NovA (Nonv...

Grounding is an important process that underlies all human interaction. Hence, it is crucial for building social robots that are expected to collaborate effectively with humans. Gaze behavior plays versatile roles in establishing, maintaining and repairing the common ground. Integrating all these roles in a computational dialog model is a comple...

Grounding is essential in human interaction and crucial for social robots collaborating with humans. Gaze plays versatile roles for establishing, maintaining and repairing the common ground. It is combined with parallel modalities and involved in several pro-cesses for behavior generation and recognition. We present a uniform modeling approach focu...

Grounding is essential in human interaction and crucial for social robots collaborating with humans. Gaze plays versatile roles for establishing, maintaining and repairing the common ground. It is combined with parallel modalities and involved in several pro-cesses for behavior generation and recognition. We present a uniform modeling approach focu...

Job interviews come with a number of challenges, especially for young people who are out of employment, education, or training (NEETs). This paper presents an approach to a job training simulation environment that employs two virtual characters and social cue recognition techniques to create an immersive interactive job interview. The two virtual c...

The TARDIS project aims to build a scenario-based serious-game simulation platform for NEETs and job-inclusion associations that supports so-cial training and coaching in the context of job interviews. This paper presents the general architecture of the TARDIS job interview simulator, and the serious game paradigm that we are developing.

This paper presents an approach that makes use of a virtual character and social signal processing techniques to create an immersive job interview simulation environment. In this environment, the virtual character plays the role of a recruiter which reacts and adapts to the user's behavior thanks to a component for the automatic recognition of soci...

In this chapter, the utilization of Digital Product Memories (DPMs) in the retail domain is discussed. We demonstrate a complete retail-related usage scenario from a consumer perspective that begins with the preparation of a shopping list before the physical in-store shopping activity. Throughout the entire shopping process in the retail store, the...

Creating intelligent interactive applications based on DPMs is a challenging task. The definition of system reactions, independent interface behaviors, and necessary context data requires expert programmers at various representation levels and often results in hard-to-maintain code. We use the Visual SceneMaker authoring tool, a visual authoring ap...

Creating interactive applications with multiple virtual characters comes along with many challenges that are related to different areas of expertise. The definition of context-sensitive interactive behavior requires expert pro- grammers and often results in hard-to-maintain code. To tackle these challenges, we suggest a visual authoring ap- proach...

In this paper, we present a visual authoring approach for the management of highly interactive, mixed-initiative, multi-party dialogues. Our approach enforces the separation of dialog-content and-logic and is based on a statechart language enfolding concepts for hierarchy , concurrency, variable scoping and runtime history. These concepts facilitat...

While current virtual characters may look photorealistic they often lack behavioral complexity. Emotion may be the key ingredient to create behavioral variety, social adaptivity and thus believability. While various models of emotion have been suggested, the concrete parametrization must often be designed by the implementer. We propose to enhance a...

Digital Object Memories, DOM, comprise concepts and technologies to physically and conceptually associate digital information with physical objects in an application-independent manner. By storing information about an object's properties, state, and history of use in its digital memory, objects become self-representative, which allows for novel kin...

Auto-ID as well as traditional identification technologies such as barcodes allow for linking physical products with digital data. Thus products become “smart items”, which may contribute to the consumer's retail experience in future retail environments. In this article, we discuss how digital assistants can utilize so-called digital product memori...

Generating coordinated multimodal behavior for an embodied agent (speech, gesture, facial expression...) is challenging. It requires a high degree of animation control, in particular when reactive behaviors are required. We suggest to distinguish realization planning, where gesture and speech are processed symbolically using the behavior markup lan...

What would the shopping experience of tomorrow look like? In this chapter we propose several human-centered and resource-adaptive ideas to this question. Throughout the whole chapter we explain our ideas with the recurrent theme of a shop that consists of instrumented shelves, public displays, audio systems, and mobile devices for each user.

Recently, a number of research projects have been started to create virtual agents that do not just serve as assistants to which tasks may be delegated, but that may even take on the role of a companion. Such agents require a great deal of social intelligence, such as the ability to detect the user's affective state and to respond to it in an empat...

Die vorliegende Arbeit beschreibt eine Erklärungskomponente einer natürlichsprachlichen Schnittstelle, die in Robotersystemen für die Kommunikation mit Menschen eingesetzt werden kann. Die speziell dafür benötigten Bereiche des Roboterwissens bestehen einerseits aus Plänen, in denen Roboteranweisungen formuliert sind, und andererseits aus Diagnose-...

This paper presents the results of a field study of the COHIBIT museum exhibit. Its purpose is to convey knowledge about car-technology and virtual characters in an entertaining way. Two life-sized virtual characters interact with the visitors and support them in constructing car models with specific tangible car modules. The evaluation should reve...

This paper describes a system to enhance the interaction between humans and virtual characters with emotional mimicry and role-taking. Such system increases the believability of virtual agents. Mimicking necessitates a model of emotional intelligence to understand and display user's emotions. A more complex processing is however necessary for a rea...

The continuous collection of digital information via smart labels attached to physical objects is a promising way to support information availability across all stages of a product's lifecycle. Since such "digital product memories" may contain a vast amount of heterogeneous data, we expect a strong demand for user support in tasks related to inform...

In this paper we present an interactive poker game in which one human user plays against two animated agents using RFID-tagged poker cards. The game is used as a showcase to illustrate how current AI technologies can be used for providing new features to computer games. A powerful and easy-to-use multimodal dialog authoring tool is used for modelin...

We present IGaze, a semi-immersive human-avatar interaction system. Using head tracking and an illusionistic 3D effect we let users interact with a talking avatar in an application interview scenario. The avatar features reactive gaze behavior that adapts to the user position according to exchangeable gaze strategies. In user studies we showed that...