Daniel Thalmann

École Polytechnique Fédérale de Lausanne | EPFL · Computer Science Section

Schizophrenia and depression are the two most common mental disorders associated with negative symptoms that contribute to poor functioning and quality of life for millions of patients globally. This study is part of a larger research project. The overall aim of the project is to develop an automated objective pipeline that aids clinical diagnosis...

Many on-line communities still rely on text messages as their privileged medium of communication and we believe it will remain so even if it can be complemented by images and video. Previous chapters have described how to infer emergent emotion dynamics from text corpus whereas we focus here on the instantaneous display of a detected emotion with a...

This paper presents an immersive volleyball game, where a player plays not only against virtual opponents but also with support on his/her side of virtual teammates. This volleyball game has been implemented for several 3D displays such as a stereoscopic display, an autostereoscopic display, Oculus Rift glasses, and a 320o Immersive Room. In this p...

Wearable devices such as Microsoft Hololens and Google glass are highly popular in recent years. As traditional input hardware is difficult to use on such platforms, vision-based hand pose tracking and gesture control techniques are more suitable alternatives. This demo shows the possibility to interact with 3D contents with bare hands on wearable...

This paper proposes a real time sociometric system to analyze social behavior from audio-visual recordings of two-person face-to-face conversations in English. The novelty of the proposed system lies in this automatic inference of ten social indicators in real time. The system comprises of a Microsoft kinect device that captures RGB and depth data...

Expanding the spectrum of agent social capabilities is an important challenge in agent-based simulation and other domains. While human-like emotionality has been vastly explored in the last 20years, little research addresses explicit, psychologically believable social situation modeling. Recently, some important elements have been underlined: hybri...

Human activity recognition in unconstrained RGB-D videos has extensive applications in surveillance, multimedia data analytics, human-computer interaction, etc, but remains a challenging prob-lem due to the background clutter, camera motion, viewpoint changes, etc. We develop a novel RGB-D activity recognition ap-proach that leverages the dense tra...

User ratings are the essence of recommender systems in e-commerce. Lack of motivation to provide ratings and eligibility to rate generally only after purchase restrain the effectiveness of such systems and contribute to the well-known data sparsity and cold start problems. This article proposes a new information source for recommender systems, call...

Healthcare can be brought to a principally new level by creating virtual bodies of real patients and using them for diagnosis and treatment planning along with the real patients. It will help to improve the quality of healthcare services, and reduce the healthcare cost, especially when dealing with the rise in musculoskeletal disorders in the aging...

The precise evaluation of joint movements, using reconstructed three-dimensional models of the joint tissues, is an important issue in many medical diagnosis and surgery planning. In order to achieve appropriate results by analyzing 3D joint models, efficient and accurate evaluation of contact among the virtual tissues is critical. In this chapter,...

Hand pose tracking and gesture recognition are useful for human-computer interaction, while a major problem is the lack of discriminative features for compact hand representation. We present a robust hand parsing scheme to extract a high-level description of the hand from the depth image. A novel distance-adaptive selection method is proposed to ge...

In this paper, a human-robot interaction system based on a novel combination of sensors is proposed. It allows one person to interact with a humanoid social robot using natural body language. The robot understands the meaning of human upper body gestures and expresses itself by using a combination of body movements, facial expressions, and verbal l...

New technical innovations in telepresence allow people to attend meetings or gather together anywhere they are and to share the same virtual space using 3D avatars. More sophisticated systems have been developed with 3D capturing and rendering, leading to a true 3D telepresence experience. If the 3D avatar is remotely guided by the real participant...

Simulating a crowded scene like a busy shopping street requires tight packing of virtual characters. In such cases, collisions are likely to occur, and the choice in collision detection shape will influence how characters are allowed to intermingle. Full collision detection is too expensive for crowds, so simplifications are needed. The most common...

Trust has been extensively studied and its effectiveness demon-strated in recommender systems. Due to lack of explicit trust information in most systems, many trust metric ap-proaches have been proposed to infer implicit trust from user ratings. However, previous works have not compared these different approaches, and oftentimes focus only on the p...

In this chapter, we present a system that provides real-time feedback about an ongoing discussion. Various speech statistics, such as speaking length, speaker turns and speaking turn duration, are computed and displayed in real time. In social monitoring, such statistics have been used to interpret and deduce talking mannerisms of people and gain i...

Pompeii was a Roman city, destroyed and completely buried during an eruption of the volcano Mount Vesuvius. We have revived its past by creating a D model of its previous appearance and populated it with crowds of Virtual Romans. In this paper, we detail the process, based on archaeological data, to simulate ancient Pompeii life in real time. In a...

This paper presents a parallel texture synthesis over arbitrary surfaces, generating consistent and spatially-varying visual appearances. A novel scaling field is represented to measure the geometry-aware appearance or geometric deformation, therefore the generated textures locally agree with the geometric structure and maintain the coherence durin...

In this tutorial, we will describe both virtual characters and realistic humanoid social robots using the same high level models. Particularly, we will describe: 1. How to capture real-time gestures and facial emotions from real people, how to recognize any real person, how to recognize certain sounds. We will present a state of the art and some ne...

In neurofeedback, brain waves are transformed into sounds or music, graphics, and other representations, to provide real-time information on ongoing waves and patterns in the brain. Here we present various forms of neurofeedback, including sonification, sonification in combination with visualization, and at last, immersive neurofeedback, where audi...

In this paper, we propose a method to achieve effective facial emotional expressivity for embodied conversational agents by considering two types of asymmetry when exploiting the valence–arousal–dominance representation of emotions. Indeed, the asymmetry of facial expressions helps to convey complex emotional feelings such as conflicting and/or hid...

Content-based human motion retrieval is important for animators with the development of motion editing and synthesis, which need to search similar motions in large databases. Obtaining text-based representation from quantization of mocap data turned ...

A real-time system is proposed to quantitatively assess speaking mannerisms and social behavior from audio recordings of two-person dialogs. Speaking mannerisms are quantitatively assessed by low-level speech metrics such as volume, rate, and pitch of speech. The social behavior is quantified by sociometrics including level of interest, agreement,...

Lack of motivation to provide ratings and eligibility to rate generally only after purchase restrain the effectiveness of recommender systems and contribute to the well-known data sparsity and cold start problems. This paper proposes a new information source for recommender systems, called prior ratings. Prior ratings are based on users' experience...

In this paper, a novel human-virtual human interaction system is proposed. This system supports a real human to communicate with a virtual human using natural body language. Meanwhile, the virtual human is capable of understanding the meaning of human upper body gestures and reacting with its own personality by the means of body action, facial expr...

In this paper we present a novel vision-based markerless hand pose estimation scheme with the input of depth image sequences. The proposed scheme exploits both temporal constraints and spatial features of the input sequence, and focuses on hand parsing and 3D fingertip localization for hand pose estimation. The hand parsing algorithm incorporates a...

Despite the demand for culturally placed agent models, an adequate simulation approach to the relationship between group-cultural and individual-psychological qualities, including culture emergence, is just appearing. It could be argued that we are at the beginning of a domain forming process, a dawn of generative, emergent artificial culture. In t...

We explore an approach to full-body motion editing with linear motion models, prioritized constraint-based optimization and latent-space interpolation. By exploiting the mathematical connections between linear motion models and prioritized inverse kinematics (PIK), we formulate and solve the motion editing problem as an optimization function whose...

The recent proliferation of virtual reality (VR) technology applications in the autism therapy to promote learning and positive behavior among such children has produced optimistic results in developing a variety of skills and abilities in them. Dolphin-assisted therapy has also become a topic of public and research interest for autism intervention...

Providing high quality recommendations is important for e-commerce systems to assist users in making effective selection decisions from a plethora of choices. Collaborative filtering is a widely accepted technique to generate recommendations based on the ratings of like-minded users. However, it suffers from several inherent issues such as data spa...

In this chapter, we focus on the technical aspect of real-time crowd simulation, and how an efficient architecture can be developed. But first, it is important to identify the main goals to achieve. We need: Quantity: real-time simulation of thousands of characters, Quality: state-of-the-art virtual humans, Efficiency: storage and real-time managem...

One of the important characteristics of behavioral animation is its ability to reduce the workload on the animators. This is achieved by letting a behavioral model automatically take care of the low-level details of the animation, freeing the animator to concentrate on the big picture. Freeing the animator from low-level animation details is even m...

While designing an animation engine usable in crowd simulations, several criteria have to be taken into account: animation computation should be efficient and scalable, it should allow for variability, and it should be compatible with levels of detail. To understand the problems, let us take the most important animation pattern used in crowd simula...

In this paper, we first present our crowd simulation method, Trajectory Variant Shift (TVS) based on real pedestrian trajectories re-use. We detail how to re-use and shift these trajectories to avoid collisions while retaining the liveliness of captured data. Second, we conducted a user study in a four-screen CAVE to compare our approach with three...

In this paper we present a model-based framework for hand pose estimation, which relies on the depth and color image sequence input. The proposed framework adopts a divide-and-conquer scheme, and combines fingertip tracking and articulated iterative closest point approach to restore the hand motion. The tracked fingertip positions are used to provi...

Traversing dense crowds can be challenging, but it is especially difficult in a virtual environment where the user is limited to simple input devices. Predicting near-future user movements with a hidden Markov model allows nearby autonomous agents to react politely, that is, to specifically avoid impeding the movement of the user. This predictive m...

We present a vision-based approach for robust 3D fingertip and palm tracking on depth images using a single Kinect sensor. First the hand is segmented in the depth images by applying depth and morphological constraints. The palm is located by performing distance transform to the hand contour and tracked with a Kalman filter. The fingertips are dete...

Robot localization is a key barrier to providing natural interaction between 3D virtual characters, human users and mobile robots. Knowing where the robot is, relative to a known world-frame, is essential to directed gestures, gazes and expressions between the robot and the other real and virtual participants in a telepresence system. The intrinsic...

This paper presents an interactive dynamic controller used to generate locomotion patterns for humanoid robots. The purpose of this work is to provide animators and artists easy and intuitive tools to design expressive motions for humanoid robots. A review of similar work in the computer animation community has guided our choices regarding the impl...

Providing high quality recommendations is important for online systems to assist users who face a vast number of choices in making effective selection decisions. Collaborative filtering is a widely accepted technique to provide recommendations based on ratings of similar users. But it suffers from several issues like data sparsity and cold start. T...

This paper proposes a new emotional model for Virtual Humans (VHs) in a conversational environment. As a part of a multi-users emotional 3D-chatting system, this paper focus on how to formulate and visualize the flow of emotional state defined by the Valence-Arousal-Dominance (VAD) parameters. From this flow of emotion over time, we successfully vi...

Non-verbal communication (NVC) makes up about two-thirds of all communication between two people or between one speaker and a group of listeners. However, this fundamental aspect of communicating is mostly omitted in 3D social forums or virtual world oriented games. This paper proposes an answer by presenting a multi-user 3D-chatting system enriche...

Autonomous virtual characters and social robots are meant to interact with humans. They should be able to communicate, express emotions and exhibit personality. Their social skills are highly dependant on their physical design, as well as on their motion capabilities. This paper presents a comparative survey of design choices and motion generation...

Since the identification of the first cases of Autism Spectrum Disorder (ASD) in 1943 by Dr Leo Kanner, the number of reported ASD cases as well as research studies expanding our understanding of the disorder has increased rapidly and significantly. In terms of ASD treatment, there is a wide range of conventional as well as complementary and altern...

We present an interface for navigating digital collections based on a one-dimensional analog control and a data visualization based on old analog radios. Our system takes advantage of inertial control to browse a large data collection in a compelling way, reducing the complexity of similar interfaces present in both desktop-based and portable media...

This paper presents an interactive multi-agent system based on a fully immersive virtual environment. A user can interact with the virtual characters in real time via an avatar by changing their moving behavior. Moreover, the user is allowed to select any character as the avatar to be controlled. A path planning algorithm is proposed to address the...

Autonomous virtual characters and social robots are meant to interact with humans. They should be able to communicate, express emotions and exhibit personality. Their social skills are highly dependent on their physical design, as well as on their motion capabilities. This paper presents a comparative survey of design choices and motion generation...

Computer-based simulations of human hip joints generally include investigating contacts happening among soft or hard tissues during hip movement. In many cases, hip movement is approximated as rotation about an estimated hip center. In this paper, we investigate the effect of different methods used for estimating hip joint center of rotation on the...

This paper essentially tries to examine all the roles that Virtual Humans can play in empowering human expression, and the research challenges we have to face to make this possible. It starts with a short history of Virtual Humans and how we contribute to the foundations of this field. We then define six typical Virtual Humans: the Performing Virtu...

Emotions are an important part of most societal dynamics. As with face to face meetings, Internet exchanges may not only include factual information but may also elicit emotional responses; how participants feel about the subject discussed or other group members. The development of automatic sentiment analysis has made large scale emotion detection...

Safety and efficiency are two major issues when it comes to long vehicle driving and operating, particularly those with very large lateral and longitudinal sizes. Trajectory planning for long vehicle tuning always plays an important role to secure safety and efficient operation, especially when turning in narrow spaces limited by surrounding object...

This paper is motivated by the objective of improving the realism of real-time simulated crowds by reducing short term collision avoidance through long term anticipation of pedestrian trajectories. For this aim, we choose to reuse outdoor pedestrian trajectories obtained with non-invasive means. This initial step is achieved by analyzing the record...

In order to be believable, embodied conversational agents (ECAs) must show expression of emotions in a consistent and natural looking way across modalities. The ECA has to be able to display coordinated signs of emotion during realistic emotional behaviour. Such a capability requires one to study and represent emotions and coordination of modalitie...

This paper presents a general-purpose simula-tion approach integrating a set of technological develop-ments and algorithmic methods in cellular automata (CA) domain. The approach provides a general-purpose comput-ing on graphics processor units (GPGPU) implementation for computing and multiple rendering of any direct-neighbor three-dimensional (3D)...

The acquisition of surround-view panoramas using a single hand-held or head-worn camera relies on robust real-time camera orientation tracking and relocalization. This paper presents robust methodology and evaluation for camera orientation relocalization, ...