Séverin Lemaignan

• PhD · https://academia.skadge.org
• Senior Researcher at Bristol Robotics Laboratory

Accurate recognition of human emotions is a crucial challenge in affective computing and human-robot interaction (HRI). Emotional states play a vital role in shaping behaviors, decisions, and social interactions. However, emotional expressions can be influenced by contextual factors, leading to misinterpretations if context is not considered. Multi...

We introduce social embeddings as a compact, yet semantics-preserving, mathematical representation of social situations. Social embeddings are constructed by leveraging pre-trained large language models: we automatically generate a textual description of the social environment of a robot, and use pre-trained text embeddings to generate a vector rep...

Despite the many recent achievements in developing and deploying social robotics, there are still many underexplored environments and applications for which systematic evaluation of such systems by end-users is necessary. While several robotic platforms have been used in gerontological healthcare, the question of whether or not a social interactive...

Affect recognition, or the ability to detect and interpret emotional states, has the potential to be a valuable tool in the field of healthcare. In particular, it can be useful in gamified therapy, which involves using gaming techniques to motivate and keep the engagement of patients in therapeutic activities. This study aims to examine the accurac...

With the increase in life expectancy and staff shortage, there is an urgency to understanding the needs of older adults and exploring emerging fields such as social robotics to tackle the challenges of ageing. The paper highlights the importance of providing cognitive support, physical support, reducing loneliness and increasing social engagement a...

Social robots have the potential to support autistic school children with their wellbeing. This research reveals how a co-design approach with autistic children and their teachers was undertaken. Focus groups with autistic children and teachers collaboratively identified user requirements for the social robot and robot behaviours within the school...

Affect recognition, or the ability to detect and interpret emotional states, has the potential to be a valuable tool in the field of healthcare. In particular, it can be useful in gamified therapy, which involves using gaming techniques to motivate and keep the engagement of patients in therapeutic activities. This study aims to examine the effecti...

Game engines are increasingly used as simulation platforms by the autonomous vehicle community to develop vehicle control systems and test environments. A key requirement for simulation-based development and verification is determinism, since a deterministic process will always produce the same output given the same initial conditions and event his...

For a period of 3 weeks in June 2021, we embedded a social robot (Softbank Pepper) in a Special Educational Needs (SEN) school for autistic children. The robot’s behaviours and integration into the school were co-designed with the children and teachers, with a focus on improving the well-being of the pupils. Using a mix-method approach, we studied...

Fig. 1: Placing Sequence: an object is handed to the robot in an unknown pose (I). Then it is aligned with the placing surface solely based on tactile feedback (II). Finally it is placed on the table (III) and the robot retreats with its arm (IV). Abstract-Tactile sensors are promising tools for endowing robots with embodied intelligence and increa...

To achieve seamless interactions, robots have to be capable of reliably detecting affective states in real time. One of the possible states that humans go through while interacting with robots is frustration. Detecting frustration from RGB images can be challenging in some real-world situations; thus, we investigate in this work whether thermal ima...

We present a holistic grasping controller, combining free-space position control and in-contact force-control for reliable grasping given uncertain object pose estimates. Employing tactile fingertip sensors, undesired object displacement during grasping is minimized by pausing the finger closing motion for individual joints on first contact until f...

Participatory design (PD) has been used to good success in human-robot interaction (HRI) but typically remains limited to the early phases of development, with subsequent robot behaviours then being hardcoded by engineers or utilised in Wizard-of-Oz (WoZ) systems that rarely achieve autonomy. In this article, we present LEADOR (Led-by-Experts Autom...

We explore the effectiveness of a dynamically processed incremental referring description system using under-specified ambiguous descriptions that are then built upon using linguistic repair statements, which we refer to as a dynamic system. We build a dynamically processed incremental referring description generation system that is able to provide...

Participatory Design (PD) in Human-Robot Interaction (HRI) typically remains limited to the early phases of development, with subsequent robot behaviours then being hardcoded by engineers or utilised in Wizard-of-Oz (WoZ) systems that rarely achieve autonomy. We present LEADOR (Led-by-Experts Automation and Design Of Robots) an end-to-end PD method...

Game engines are increasingly used as simulation platforms by the autonomous vehicle (AV) community to develop vehicle control systems and test environments. A key requirement for simulation-based development and verification is determinism, since a deterministic process will always produce the same output given the same initial conditions and even...

Integrating real-time, complex social signal processing into robotic systems -- especially in real-world, multi-party interaction situations -- is a challenge faced by many in the Human-Robot Interaction (HRI) community. The difficulty is compounded by the lack of any standard model for human representation that would facilitate the development and...

Simulation-based verification is beneficial for assessing otherwise dangerous or costly on-road testing of autonomous vehicles (AV). This paper addresses the challenge of efficiently generating effective tests for simulation-based AV verification using software testing agents. The multi-agent system (MAS) programming paradigm offers rational agency...

Trust is a key dimension of human-robot interaction (HRI), and has often been studied in the HRI community. A common challenge arises from the difficulty of assessing trust levels in ecologically invalid environments: we present in this paper two independent laboratory studies, totalling 160 participants, where we investigate the impact of differen...

Striking the right balance between robot autonomy and human control is a core challenge in social robotics, in both technical and ethical terms. On the one hand, extended robot autonomy offers the potential for increased human productivity and for the off-loading of physical and cognitive tasks. On the other hand, making the most of human technical...

Generating spatial referring expressions is key to allowing robots to communicate with people in an environment. The focus of most algorithms for generation is to create a non-ambiguous description, and how best to deal with the combination explosion this can create in a complex environment. However, this is not how people naturally communicate. Hu...

In recent years, the field of Human-Robot Interaction (HRI) has seen an increasing demand for technologies that can recognize and adapt to human behaviors and internal states (e.g., emotions and intentions). Psychological research suggests that human movements are important for inferring internal states. There is, however, a need to better understa...

We conducted a study with 25 children to investigate the effectiveness of a robot measuring and encouraging production of spatial concepts in a second language compared to a human experimenter. Productive vocabulary is often not measured in second language learning, due to the difficulty of both learning and assessing productive learning gains. We...

The study of the fine-grained social dynamics between children is a methodological challenge, yet a good understanding of how social interaction between children unfolds is important not only to Developmental and Social Psychology, but recently has become relevant to the neighbouring field of Human-Robot Interaction (HRI). Indeed, child-robot inter...

Establishing common ground when attempting to disambiguate spatial locations is difficult at the best of times, but is even more challenging between children and robots. Here, we present a study that examined how 94 children (aged 5-8) communicate spatial locations to other children, adults and robots in face-to-face interactions. While standard HR...

The field of Human-Robot Interaction (HRI) lies at the intersection of several disciplines, and is rightfully perceived as a prime interface between engineering and the social sciences. In particular, our field entertains close ties with social and cognitive psychology, and there are many HRI studies which build upon commonly accepted results from...

Evaluating human-robot social interactions in a rigorous manner is notoriously difficult: studies are either conducted in labs with constrained protocols to allow for robust measurements and a degree of replicability, but at the cost of ecological validity; or in the wild, which leads to superior experimental realism, but often with limited replica...

This paper provides a qualitative review of different object recognition techniques relevant for near-proximity Human-Robot Interaction. These techniques are divided into three categories:2D correspondence, 3D correspondence and non-vision based methods. For each technique an implementation is chosen that is representative of the existing technolog...

In this article, we present Cellulo, a novel robotic platform that investigates the intersection of three ideas for robotics in education: designing the robots to be versatile and generic tools; blending robots into the classroom by designing them to be pervasive objects and by creating tight interactions with (already pervasive) paper; and finally...

An increasing number of human-robot interaction (HRI) studies are now taking place in applied settings with children. These interactions often hinge on verbal interaction to effectively achieve their goals. Great advances have been made in adult speech recognition and it is often assumed that these advances will carry over to the HRI domain and to...

A key challenge of HRI is allowing robots to be adaptable, especially as robots are expected to penetrate society at large and to interact in unexpected environments with non-technical users. One way of providing this adaptability is to use Interactive Machine Learning, i.e. having a human supervisor included in the learning process who can steer t...

When a robot is learning it needs to explore its environment and how its environment responds on its actions. When the environment is large and there are a large number of possible actions the robot can take, this exploration phase can take prohibitively long. However, exploration can often be optimised by letting a human expert guide the robot dur...

Child-robot interactions are increasingly being explored in domains which require longer-term application, such as healthcare and education. In order for a robot to behave in an appropriate manner over longer timescales, its behaviours should be coterminous with that of the interacting children. Generating such sustained and engaging social behavio...

Research in education has long established how children mutually influence and support each other's learning trajectories, eventually leading to the development and widespread use of learning methods based on peer activities. In order to explore children's learning behavior in the presence of a robotic facilitator during a collaborative writing act...

Social robots are increasingly being applied in educational environments such as schools. It is important to understand the views of the general public as social acceptance will likely play a role in the adoption of such technology. Other literature suggests that teacher attitudes are a strong predictor of technology use in classrooms, so willingne...

Human-Robot Interaction challenges Artificial Intelligence in many regards: dynamic, partially unknown environments that were not originally designed for robots; a broad variety of situations with rich semantics to understand and interpret; physical interactions with humans that requires fine, low-latency yet socially acceptable control strategies;...

Robots for education are not limited to support ICT teaching, and they are indeed finding new roles in the classroom. This article reports on such a new paradigm for educative robots, that involves learning by teaching and strong social engagement to help children struggling with handwriting. Our system relies on machine-learning and child-robot in...

Building over algorithms previously developed for digital pens, this article introduces a novel 2D localization technique for mobile robots, based on simple printed patterns. This method combines high absolute accuracy (below 0.3mm), unlimited scalability, low computational requirements (the presented open-source implementation runs at above 45Hz o...

Collaborative learning has often been associated with the construction of a shared understanding of the situation at hand. The psycholinguistics mechanisms at work while establishing common grounds are the object of scientific controversy. We postulate that collaborative tasks require some level of mutual modelling, i.e. that each partner needs som...

Human-Robot Interaction (HRI) research requires the integration and cooperation of multiple disciplines, technical and social, in order to make progress. In many cases using different motivations, each of these disciplines bring with them different assumptions and methodologies. We assess recent trends in the field of HRI by examining publications...

Measuring ``how much the human is in the interaction'' -- the level of engagement -- is instrumental in building effective interactive robots. Engagement, however, is a complex, multi-faceted cognitive mechanism that is only indirectly observable. This article formalizes with-me-ness as one of such indirect measures. With-me-ness, a concept borrowe...

The CoWriter activity involves a child in a rich and complex interaction where he has to teach handwriting to a robot. The robot must convince the child it needs his help and it actually learns from his lessons. To keep the child engaged, the robot must learn at the right rate, not too fast otherwise the kid will have no opportunity for improving h...

This volume is the proceedings of the 2nd workshop on Cognitive Architectures for Social Human-Robot Interaction, held at the ACM/IEEE HRI 2016 conference, which took place on Monday 7th March 2016, in Christchurch, New Zealand. Organised by Paul Baxter (Plymouth University, U.K.), J. Gregory Trafton (Naval Research Laboratory, USA), and Severin Le...

This paper presents the concept and a case study of a robject, a robotic entity embedded in an everyday object. Robjects use the affordance of the original object to ensure an efficient interaction and a high acceptance. The example of the ranger robot shows how this approach can be applied to a domestic environment. We explore the integration of a...

Robots are being increasingly used in schools by researchers keen to assess how they may be used to facilitate learning and provide support. Based on 15 school experiment visits at 9 different schools in the U.K., we outline our observations, specifically focusing on the broader implications of robots in the classroom primarily from the perspective...

We present a study on the impact of unexpected robot behaviors on the perception of a robot by children and their subsequent engagement in a playful interaction based on a novel “domino” task. We propose an original analysis methodology which blends behavioral cues and reported phenomenological perceptions into a compound index. While we found only...

In order to explore the impact of integrating a robot as a facilitator in a collaborative activity, we examined interpersonal distancing of children both with a human adult and a robot facilitator. Our scenario involves two children performing a collaborative learning activity, which included the writing of a word/letter on a tactile tablet. Based...

Mutual modelling, the reciprocal ability to establish a mental model of the other, plays a fundamental role in human interactions. This complex cognitive skill is however difficult to fully apprehend as it encompasses multiple neuronal, psychological and social mechanisms that are generally not easily turned into computational models suitable for r...

This article presents a novel robotic partner which children can teach handwriting. The system relies on the learning by teaching paradigm to build an interaction, so as to stimulate meta-cognition, empathy and increased self-esteem in the child user. We hypothesise that use of a humanoid robot in such a system could not just engage an unmotivated...

This video (that accompanies the paper "When Children Teach a Robot to Write: An Autonomous Teachable Humanoid Which Uses Simulated Handwriting" by the same authors, and presented as well during this conference) presents the first results of the EPFL CoWriter project. The project aims at building a robotic partner which children can teach handwriti...

Building over algorithms previously developed for digital pens, this article introduces a novel 2D localization technique for mobile robots, based on simple printed patterns. This method combines high absolute accuracy (below 0.3mm), unlimited scalability, low computational requirements (the presented open-source implementation runs at above 45Hz o...

Human-Robot interaction is an area full of challenges for artificial intelligence: dynamic, partially unknown environments that are not originally designed for autonomous machines; a large variety of situations and objects to deal with, with possibly complex semantics; physical interactions with humans that requires fine, low-latency control, repre...

Can robots in classroom reshape K-12 STEM education, and foster new ways of learning? To sketch an answer, this article reviews, side-by-side, existing literature on robot-based learning activities featuring mathematics and physics (purposefully putting aside the well-studied field of "robots to teach robotics") and existing robot platforms and too...

Simulation in robotics is often a love-hate relationship: while simulators do save us a lot of time and effort compared to regular deployment of complex software architectures on complex hardware, simulators are also known to evade many (if not most) of the real issues that robots need to manage when they enter the real world. Because humans are th...

In human relationships, responsiveness---behaving in a sensitive manner that is supportive of another person's needs---plays a major role in any interaction that involves effective communication, caregiving, and social support. Perceiving one's partner ...

We present the design approach and evaluation of our prototype called "Ranger". Ranger is a robotic toy box that aims to motivate young children to tidy up their room. We evaluated Ranger in 14 families with 31 children (2-10 years) using the Wizard-of-Oz technique. This case study explores two different robot behaviors (proactive vs. reactive) and...

Over the last four years, we have been slowly ramping up explicit knowledge representation and manipulation in the deliberative and executive layers of our robots. Ranging from situation assessment to symbolic task planning, from verbal interaction to event-driven execution control, we have built up a knowledge-oriented architecture which is now us...

The dissertation tackles the broad question of knowledge representation and manipulation for companion robots. It first builds a taxonomy of the knowledge manipulation skills required by service robots, then proposes a novel active knowledge base that integrates into large cognitive architectures, and finally explores several applications, includin...

The video presents in a didactic way several abilities and algorithms required to achieve interactive "pick and give" tasks in a human environment. Communication between the human and the robot relies on unconstrained verbal dialogue, the robot uses multi-modal perception to track the human and its environment, and implements real-time 3D motion pl...

The video presents in a didactic way the tools developed at LAAS-CNRS and related to symbol grounding and natural language processing for companion robots (Fig. 1). It mainly focuses on two of them: the ORO-server knowledge base and the Dialogs natural language processor. These two tools enable three cognitive functions that allow for better natura...