Nadia Magnenat Thalmann

University of Geneva | UNIGE · Centre Universitaire d'informatique

In this work, we describe our approach to developing an intelligent and robust social robotic system for the Nadine social robot platform. We achieve this by integrating Large Language Models (LLMs) and skilfully leveraging the powerful reasoning and instruction-following capabilities of these types of models to achieve advanced human-like affectiv...

The use of social robots in healthcare systems or nursing homes to assist the elderly and their caregivers will be becoming common, where robots' understanding of engagement of the elderly is important. Traditional engagement estimation (EE) often requires expert involvement in a controlled dyadic interaction environment. In this article, we propos...

Motivated by how humans perceive geometry and color to recognize objects, we propose a boundary embedded attentional convolution (BEACon) network for point cloud instance segmentation. At the core of BEACon, we introduce the attentional weight in the convolution layer to adjust the neighboring features, with the weight being adapted to the relation...

Using intelligent virtual assistants for controlling employee population in workspaces is a research area that remains unexplored. This paper presents a novel application of virtual humans to enforce Covid-19 safety measures in a corporate workplace. For this purpose, we develop a virtual assistant platform, Chloe, equipped with automatic temperatu...

More and more companies have started to use nonhuman agents for employment interviews, making the selection process easier, faster, and unbiased. To assess the effectiveness of the above, in this paper, we systematically analyzed, reviewed, and compared human interaction with a social robot, a digital human, and another human under the same scenari...

Intangible heritage is often linked to human actions and performances. The use of digital humans (DHs) for its digital representation and therefore its preservation, allows reframing the way to transmit and deal with content that is difficult to visualize. To that end, the digital human becomes an important element establishing the connection betwe...

The publication of this article unfortunately contained mistakes. The articles mentioned in the text were not the correct ones.

A protocol for the representation of traditional crafts and the tools to implement this are proposed. The proposed protocol is a method for the systematic collection and organization of digital assets and knowledge, their representation into a formal model, and their utilization for research, education, and preservation. A set of digital tools acco...

Sketch recognition relies on two types of information, namely, spatial contexts like the local structures in images and temporal contexts like the orders of strokes. Existing methods usually adopt convolutional neural networks (CNNs) to model spatial contexts, and recurrent neural networks (RNNs) for temporal contexts. However, most of them combine...

Optical see-through near-eye display (NED) technologies for augmented reality (AR) have achieved significant advancements recently with investments from both academia and industry. Although various AR NED products have been successfully commercialized and even deployed into applications, there are still challenges with present AR NED technologies (...

In this paper, a representation based on digital assets and semantic annotations is established for Traditional Craft instances, in a way that captures their socio-historic context and preserves both their tangible and intangible Cultural Heritage dimensions. These meaningful and documented experiential presentations are delivered to the target aud...

This work regards the representation of handicrafts for craft training and demonstration in the environment of an ethnographic heritage museum. The craft of mastic cultivation is chosen as a use case. This paper presents the process of representation and presentation of this craft, following an articulated pipeline approach for data collection, ann...

Traditional crafts exhibit tangible and intangible dimensions. Intangible dimensions include the practitioner’s gestural know-how in craft practice and have received smaller attention than tangible dimensions in digitization projects. This work presents the process of representation and presentation of the glasswork and is exemplified in the re-cre...

From antiquity to today, some scientists, writers, and artists are passionate about representing humans not only as beautiful statues but as automatons that can perform actions. Already in ancient Greece, we can find some examples of automatons that replaced servants. In this chapter, we go through the development of automatons until the social rob...

In this paper, we present the design and development of MIDAS. MIDAS is intended to be a portable immersive system inclusive of an exoskeleton, a smell releasing device, and a VR game that will help stroke patients feel more motivated during a hand rehabilitation session. MIDAS engages four out of five senses during rehabilitation. MIDAS uses EMG s...

In this paper, we present the design and development of MIDAS. MIDAS is intended to be a portable immersive system inclusive of an exoskeleton, a smell releasing device, and a VR game that will help stroke patients feel more motivated during a hand rehabilitation session. MIDAS engages four out of five senses during rehabilitation. MIDAS uses EMG s...

This paper describes three case studies performed in everyday life with our social humanoid robot, Nadine. The development of AI, vision, and NLP over the last few years has made it possible to improve social robots’ awareness ability of their environment and has enabled them to understand spoken interactions. We have developed a software platform...

Virtual Humans are becoming a commodity in computing technology and lately have been utilized in the context of interactive presentations in Virtual Cultural Heritage environments and exhibitions. To this end, this research work underlines the importance of aligning and fine-tuning Virtual Humans’ appearance to their roles and highlights the import...

There are considerable advancements in medical health care in recent years, resulting in a rising older population. As the workforce for such a population is not keeping pace, there is an urgent need to address this problem. Having robots to stimulating recreational activities for older adults can reduce the workload for caretakers and give them ti...

Current research of designing prosthetic robotic hands mainly focuses on improving their functionality by devising new mechanical structures and actuation systems. Most of existing work relies on a single structure/system (e.g., bone-only or tissue-only) and ignores the fact that the human hand is composed of multiple functional structures (e.g., s...

In the field of robotic hand design, soft body and anthropomorphic design are two trends with a promising future. Designing soft body anthropomorphic robotic hands with human-like grasping ability, but with a simple and reliable structure, is a challenge that still has not been not fully solved. In this paper, we present an anatomically correct rob...

Recent methods including CoViAR and DMC-Net provide a new paradigm for action recognition since they are directly targeted at compressed videos (e.g., MPEG4 files). It avoids the cumbersome decoding procedure of traditional methods, and leverages the pre-encoded motion vectors and residuals in compressed videos to complete recognition efficiently....

Social Assistive Robotics is increasingly being used in care settings to provide psychosocial support and interventions for the elderly with cognitive impairments. Most of these social robots have provided timely stimuli to the elderly at home and in care centres, including keeping them active and boosting their mood. However, previous investigatio...

The current chapter serves as a state-of-the-art, presenting the limitations of the existing technology used in the broad area of human–computer interaction up to now. Although different kind of agents have been used to contribute to several domains, like education, health, entertainment, both in virtual and physical environments, the virtual chara...

As one of the fundamental problems of scene understanding and modeling, object detection has attracted extensive attention in the research communities of computer vision and artificial intelligence. Recently, inspired by the success of deep learning, various deep neural network-based models have been proposed and become the de facto solution for ob...

Socially intelligent robots are dedicated mostly for social interaction with humans. They can assume two useful roles: a functional and an affective one. They aim to serve as an interface between humans and technology, and to increase their quality of life by providing companionship and assisting them in everyday tasks and routines. Although there...

Visualization techniques for the usage of tools, handicrafts, and assembly operations are employed for demonstrating processes (e.g., assembly instructions). Until today, most commonly used techniques include written information, sketches in manuals, video instructions, etc. The evolution of technology nowadays has generated mature methods for tran...

With recent developments in the field of artificial intelligence, machine learning, and deep learning, the field of social robotics has gained momentum. Any social robot requires to interact with human and its environment. Any human–robot interaction involves two aspects, speech-based and speechless interactions. Among the two, the latter is an ess...

Virtual reality (VR) volleyball games can provide an immersive entertainment experience and facilitate users to get familiar with game rules. In real world, players often experience emotions from scores, strokes, passes, and the influence from their teammates or opponents, and so forth. However, seldom studies are carried out for this emotional exp...

There are considerable advancements in medical health care in recent years, resulting in rising older population. As the workforce for such a population is not keeping pace, there is an urgent need to address this problem. Having robots to stimulating recreational activities for older adults can reduce the workload for caretakers and give them time...

This edited book is one of the first to describe how Autonomous Virtual Humans and Social Robots can interact with real people and be aware of the surrounding world using machine learning and AI. It includes: · Many algorithms related to the awareness of the surrounding world such as the recognition of objects, the interpretation of various sources...

This book constitutes the refereed proceedings of the 38th Computer Graphics International Conference, CGI 2021, held virtually in September 2021. The 44 full papers presented together with 9 short papers were carefully reviewed and selected from 131 submissions. The papers are organized in the following topics: computer animation; computer vision;...

Most current anthropomorphic robotic hands can realize part of the human hand functions, particularly for object grasping. However, due to the complexity of the human hand, few current designs target at daily object manipulations, even for simple actions like rotating a pen. To tackle this problem, we introduce a gesture based framework, which adop...

Despite the great progress in human motion prediction, it remains a challenging task due to the complicated structural dynamics of human behaviors. In this paper, we address this problem in three aspects. First, to capture the long-range spatial correlations and temporal dependencies, we apply a transformer-based architecture with the global attent...

Current anthropomorphic robotic hands mainly focus on improving their dexterity by devising new mechanical structures and actuation systems. However, most of them rely on a single structure/system (e.g., bone-only) and ignore the fact that the human hand is composed of multiple functional structures (e.g., skin, bones, muscles, and tendons). This n...

In this work we have enhanced the perception of a humanoid robot by integrating it with a social state estimation system. We present a user study of the humanoid Nao robot as a social mediator, comprising two sets of experiments. In the first sets of experiments, the participants rate their understanding of feedback messages delivered via the Nao r...

Learning couple dance such as salsa is challenging as it requires to understand and assimilate all the dance skills (guidance, rhythm, style) correctly. Salsa is traditionally learned by attending a dancing class with a teacher and practice with a partner, the difficulty to access such classes though, and the variability of dance environment can im...

Compared with depth-based 3D hand pose estimation, it is more challenging to infer 3D hand pose from monocular RGB images, due to the substantial depth ambiguity and the difficulty of obtaining fully-annotated training data. Different from the existing learning-based monocular RGB-input approaches that require accurate 3D annotations for training,...

The LNCS journal Transactions on Computational Science reflects recent developments in the field of Computational Science, conceiving the field not as a mere ancillary science but rather as an innovative approach supporting many other scientific disciplines. The journal focuses on original high-quality research in the realm of computational science...

This book constitutes the refereed proceedings of the 37th Computer Graphics International Conference, CGI 2020, held in Geneva, Switzerland, in October 2020. The conference was held virtually. The 43 full papers presented together with 3 short papers were carefully reviewed and selected from 189 submissions. The papers address topics such as: virt...

This book constitutes the revised selected papers of the 33rd International Conference on Computer Animation and Social Agents, CASA 2020, held in Bournemouth, UK*, in October 2020. The 1 full paper and 13 short papers presented were carefully reviewed and selected from a total of 86 submissions. The papers are organized in topical sections of mode...

In this work, we address the challenging issue of scene segmentation. To increase the feature similarity of the same object while keeping the feature discrimination of different objects, we explore to propagate information throughout the image under the control of objects' boundaries. To this end, we first propose to learn the boundary as an additi...

In recent years, we have witnessed a steady growth of the research in real-time 3D hand pose estimation with depth cameras, since this technology plays an important role in various human–computer interaction applications. In this chapter, we first review existing techniques and systems for real-time 3D hand pose estimation. Then, we will discuss tw...

Schizophrenia is a Xu, Shihao Yang, Zixu Chakraborty, Debsubhra that contributes to Tahir, Yasir Maszczyk, Tomasz Victoria, Chua Yi Han Dauwels, Justin Thalmann, Daniel life. Thalmann, Nadia Magnenat Tan, Bhing-Leet Keong, Jimmy Lee Chee to design objective assessment tools of schizophrenia. In earlier work, we investigated non-verbal quantitative...

In this work, we address the challenging issue of scene segmentation. To increase the feature similarity of the same object while keeping the feature discrimination of different objects, we explore to propagate information throughout the image under the control of objects' boundaries. To this end, we first propose to learn the boundary as an additi...

Full text of tutorial

Schizophrenia and depression are the two most common mental disorders associated with negative symptoms that contribute to poor functioning and quality of life for millions of patients globally. This study is part of a larger research project. The overall aim of the project is to develop an automated objective pipeline that aids clinical diagnosis...

The current preliminary study concerns the identification of the effects human-humanoid interaction can have on human emotional states and behaviors, through a physical interaction. Thus, we have used three cases where people face three different types of physical interaction with a neutral person, Nadine social robot and the person on which Nadine...

This paper presents a system that aims to achieve autonomous grasping for micro-controller based humanoid robots such as the Inmoov robot [1]. The system consists of a visual sensor, a central controller and a manipulator. We modify the open sourced objection detection software YOLO (You Only Look Once) v2 [2] and associate it with the visual senso...

Developing a social robot architecture is very difficult as there are countless possibilities and scenarios. In this paper, we introduce the design of a generic social robotics architecture deployed in Nadine social robot, that can be customized to handle any scenario or application, and allows her to express human-like emotions, personality, behav...