Christine Lisetti

• Professor
• Professor (Associate) at Florida International University

Independent and autonomous at-home care can solve many current societal issues. However, with increased life expectancy and the rise of aging populations, further improvements to interactive robots are needed to increase Quality-of-Life. The standalone Toyota Human Support Robot has the ability to conduct independent at-home care. However, it lacks...

Simulation technology provides opportunities for teachers to engage in extended practice using positive behavioral supports to promote student engagement and behavior. These training models are rapidly emerging and if effective, create an infrastructure for scaling up positive behavioral supports in classrooms and schools. However, there is limited...

Human support robots (mobile robots able to perform useful domestic manipulative tasks) might be better accepted by people if they can communicate in ways they naturally understand: e.g. speech, but also facial expressions, postures, among others. Subtle (unconscious) mirroring of nonverbal cues during conversations promotes rapport building, essen...

Background Digital health agents — embodied conversational agents designed specifically for health interventions — provide a promising alternative or supplement to behavioral health services by reducing barriers to access to care. Objective Our goals were to (1) develop an expressive, speech-enabled digital health agent operating in a 3-dimensiona...

Virtual training models to support teachers’ use of positive behavioral supports provide an unprecedented opportunity to support teacher professional development via practice, reflection, problem solving, and feedback in an authentic learning context. However, little is known regarding factors that influence teacher use and benefit from these model...

In human face-to-face conversations, non-verbal behaviors (NVB), such as gaze, facial expressions, gestures, and body postures, can improve communication effectiveness, by creating a smooth interaction between the interlocutors - called rapport. During human interactions with embodied conversational agents (ECAs) (a.k.a. virtual humans), a key issu...

In this article, we discuss the 3D Interactive Virtual Training for Teachers system (IVT-T) which we built for teachers to hone in their classroom management skills. By interacting with thirty fully autonomous and animated realistic 3D virtual students in two detailed and realistic 3D classrooms, teachers can practice their behavior management and...

Simulation training models, such as Interactive Virtual Training for Teachers (IVT-T), have relevance to consultation and show promise in supporting teachers who need behavior management support. Using an embedded mixedmethod design, we examined the feasibility and effectiveness of IVT-T. Twenty seven teachers working in three K-8 schools participa...

Partial subtle mirroring of nonverbal behaviors during conversations (also known as mimicking or parallel empathy), is essential for rapport building, which in turn is essential for optimal human-human communication outcomes. Mirroring has been studied in interactions between robots and humans, and in interactions between Embodied Conversational Ag...

BACKGROUND Digital health agents — embodied conversational agents designed specifically for health interventions — provide a promising alternative or supplement to behavioral health services by reducing barriers to access to care. OBJECTIVE Our goals were to (1) develop an expressive, speech-enabled digital health agent operating in a 3-dimensiona...

Simulation offers unique affordances over traditional training (e.g., remote access, mastery learning experiences, immediate feedback) relevant to teacher training in behavior management. This study describes a user-based evaluation of Interactive Virtual Training for Teachers (IVT-T). The study involved observing representative users (seven advanc...

We posit that human-robot interfaces that integrate multimodal communication features of a 3-dimensional graphical social virtual agent with a high degree of freedom robot are highly promising. We discuss the modular agent architecture of an interactive system that integrates two frameworks (our in-house virtual social agent and robot agent framewo...

Elementary school classrooms are emotionally stressful environments, for both students and teachers. Successful teachers use strategies that regulate students' emotions and behaviors while also controlling their own emotions (stress, nervousness). To prepare teachers for the challenges of teaching, teacher training should include emotional and beha...

Simulation-based training systems have proven effective in a variety of domains, both for facilitating the learning of skills as well for applying this knowledge to real life. Although difficulties managing students' disruptive behavior in classrooms has been identified as one of the main causes of teachers' turnover, only a handful of virtual trai...

Although socially interactive agents have emerged as a new metaphor for human-computer interaction, they are, to date, absent from the Internet. We describe the design choices, implementation, and challenges in building EEVA, the first fully integrated platform-independent framework for deploying realistic 3D web-based social agents: with real-time...

Early career teachers working in high poverty schools face of overwhelming challenges navigating disruptive behaviors with studies highlighting behavior problems as one of the strongest predictors of turnover (Ingersoll & Smith, 2003). Simulation-based technology leverages important pedagogical strengths (e.g., realistic training context, opportuni...

Interactive Virtual Training (IVT) is a web-based interactive simulation system created to help early career teachers work- ing in high-poverty schools learn effective classroom man- agement skills. IVT simulates disruptive students’ behavior (aggressive, off-task, non-compliant), as defined in vignettes developed by pedagogical experts on our team....

Real time user independent facial expression recognition is important for virtual agents but challenging. However, since in real time recognition users are not necessarily presenting all the emotions, some proposed methods are not applicable. In this paper, we present a new approach that instead of using the traditional base face normalization on w...

With the growing number of researchers interested in modeling the inner workings of affective social intelligence, the need for tools to easily model its associated expressions has emerged. The goal of this article is two-fold: 1) we describe HapFACS, a free software and API that we developed to provide the affective computing community with a reso...

This research focuses on the potential ability of animated avatars (a digital representation of the user) and virtual agents (a digital representation of a coach, buddy, or teacher) to deliver computer-based interventions for adolescents' chronic weight management. An exploration of the acceptance and desire of teens to interact with avatars and vi...

We discuss a large research project aimed at building socially expressive virtual health agents or assistants (VHA) that can deliver brief motivational interventions (BMI) for behavior change, in a communication style that individuals and patients not only accept, but also find emotionally supportive and socially appropriate. Because of their well-...

Cyberpsychology is a recent branch of psychology whose main research objects are the processes of change induced by new technologies. Some of these processes are related to and involve a variety of affective processes. The discipline’s overlap with affective computing and human-computer interaction in general are significant, yet its psychological...

There is still so much we do not know regarding how our emotional lives play out in digital experiences and how the design of systems, interfaces and interaction shape our emotions. How can technology support psychological wellbeing? How can we at least make sure that it does not hinder it? The majority of the research trying to address these quest...

We developed a virtual counseling system which can deliver brief alcohol health interventions via a 3D anthropomorphic speech-enabled interface—a new field for spoken dialog interactions with intelligent virtual agents in the health domain. We present our spoken dialog system design and its evaluation. We developed our dialog system based on Markov...

We focus on creating a programming tool which enables to create dialog managers for speech-enabled IVA applications in the standardized health screening and assessment domain. Our approach aims to bridge the gap between the intelligent virtual agents (IVA) and the spoken dialog systems (SDS) research communities for the delivery of standardized hea...

In this paper, we discuss a novel approach to design an emotionally responsive system in the context of virtual health interventions for behavior change. We describe the system’s design with a focus on enabling a multimodal Embodied Conversational Agent (ECA) to deliver the interventions empathetically. This is done by adapting its verbal and non-v...

There has recently been growing interest in valence and emotion sensing using a variety of signals. Text, as a communication channel, gathers a substantial amount of interest for recognizing its underlying sentiment (valence or polarity), affect or emotion (e.g. happy, sadness). We consider recognizing the valence of a sentence as a prior task to e...

We discuss our approach to developing a novel modality for the computer-delivery of Brief Motivational Interventions (BMIs) for behavior change in the form of a personalized On-Demand VIrtual Counselor (ODVIC), accessed over the internet. ODVIC is a multimodal Embodied Conversational Agent (ECA) that empathically delivers an evidence-based behavior...

We present HapFACS (ver. beta), a new open source software and API for generating FACS-based facial expressions on 3D virtual characters that have accompanying lip-synchronized animation abilities. HapFACS has two main usage scenarios: First, with the HapFACS software, users can generate repertoires of realistic FACS-validated facial expressions, e...

In this paper, we discuss a novel approach for the computer-delivery of Brief Motivational Interventions (BMIs) for health behavior change. We describe the basic elements of our system architecture, and focus on enabling a multimodal Embodied Conversational Agent (ECA) to deliver the health behavior change interventions empathetically by adapting,...

This paper describes the design of a multimodal spoken dialogue system using Markov Decision Processes (MDPs) to enable embodied conversational virtual health coach agents to deliver brief interventions for lifestyle behavior change - in particular excessive alcohol consumption. Its contribution is two fold. First, it is the first attempt to-date t...

Named-Entity Recognizers (NERs) are an important part of information extraction systems in annotation tasks. Although substantial progress has been made in recognizing domain-independent named entities (e.g. location, organization and person), there is a need to recognize named entities for domain-specific applications in order to extract relevant...

In this article, we present HapFACS 1.0, a new software/API for generating static and dynamic three-dimensional facial expressions based on the Facial Action Coding System (FACS). HapFACS provides total control over the FACS Action Units (AUs) activated at all levels of intensity. HapFACS allows generating faces with an individual AU or composition...

Emerging Health Information Technologies (HIT) such as Electronic Health Record (EHR) and PHR systems and Heath Information Exchanges (HIEs) facilitate access to and sharing of patients' medical data distributed in different HIT systems. One of the many ...

We discuss the design and implementation of the pro-totype of an avatar-based health system aimed at pro-viding people access to an effective behavior change intervention which can help them to find and cultivate motivation to change unhealthy lifestyles. An empathic Embodied Conversational Agent (ECA) delivers the in-tervention. The health dialog...

In this paper, we have proposed a user model for computer based drinking behavior change intervention and recommender systems. We discuss specific requirements of user modeling in health promotion and specifically alcohol interventions. We believe that making behavior change systems available pervasively may lead to better and sustainable results....

In this article, we propose the design of sensory motor level as part of a three-layered agent architecture inspired from the Multilevel Process Theory of Emotion (Leventhal 1979, 1980; Leventhal and Scherer, 1987). Our project aims at modeling emotions on an autonomous embodied agent, Petra, a more robust robot than our previous prototype - Cherry...

In this article we review some of the main contributions that the believable agents community has brought about and we review some of the main believable agent architectures existing to date. We also discuss the enormous potential that believable agents can bring in the domain of health, including health communication, health promotion, health coun...

In this article we describe work-in-progress about the development of avatar-based personalized assistants that can delivered motivational interviewing health behavior change interventions, tailored to its specific users Our approach combines the latest progress in Embodied Conversational Agents (ECAs), believable agents, and dialog systems. We dis...

In this article, we describe a new approach to enhance driving safety via multi-media technologies by recognizing and adapting to drivers’ emotions with multi-modal intelligent car interfaces. The primary objective of this research was to build an affectively intelligent and adaptive car interface that could facilitate a natural communication with...

This special section is focused on the novel and practical ways, and solid contributions, to improve affective computing and pervasive technologies. There are 19 papers in this special section.

In this article, we address some of the issues concerning emotion recognition from processing physiological signals captured by bio-sensors. We discuss some of our preliminary results, and propose future directions for emotion recognition based on our lessons learned.

We give an overview of the Second Life (SL) virtual world, explaining what objects can be manipulated to implement user-defined scenari in SL, and give an example of a work-in-progress scenario of exposure therapy for anxiety disorders, coupled with automatic processing of bio-sensed emotional signals. Both e-Health and emotion recognition research...

We explore how avatars can be used as social orthotics defined as therapeutic computer-based social companions aimed at promoting healthy behaviors. We review some of the health interventions deployed in helping at-risk populations along with some of the unique advantages that computer-based interventions can add to face-to-face interventions. We p...

The Association for the Advancement of Artificial Intelligence (AAAI) was pleased to present the AAAI 2008 Spring Symposium Series, held Wednesday through Friday, March 26-28, 2008, at Stanford University, California. The eight symposia were titled (1) AI Meets Business Rules and Process Management, (2) Architectures for Intelligent Theory-Based Ag...

ABSTRACT The impact of hurricanes is so devastating throughout different levels of society that there is a pressing need to provide a range of users with accurate and timely information that can enable effective planning,for and,response,to potential hurricane landfalls. The Weather Research and Forecasting (WRF) code is the,latest numerical model...

In this article, we explore the possibility of using animated characters as personal social companions for supporting interventions for promoting health behaviors. We explore how supportive feedback could be provided to users of such artificial companion systems, by coupling both personalized intervention content from a mental health perspective wi...

We have outlined a number of questions that should be addressed as we explore the emerging research and practice area of agent-augmented virtual environments for psychotherapy. The questions addressed theoretical, technological and 'cultural' challenges, the latter referring to issues of acceptance of these technologies by the clinical community. T...

In this article, we propose the design of sensory motor level as part of a three-layered agent architecture inspired from the Multilevel Process Theory of Emotion (Leventhal 1979, 1980; Leventhal and Scherer, 1987). Our project aims at modeling emotions on an autonomous embodied agent, a more robust robot than our previous prototype. Our robot has...

The interpretation of physiological signals in terms of emotion requires an appropriate mapping between physiological features and emotion representations. We present a user model associating psychological and physiological representation of emotion in order to bring findings from the psychophysiology domain into User-Modeling computational techniq...

In this paper we describe the User Modeling phase of our general research approach: developing Adaptive Intelligent User Interfaces to facilitate enhanced natural communication during the Human-Computer Interaction. Natural communication is established by recognizing users’ affective states (i.e., emotions experienced by the users) and responding t...

During last decade, an increasing interest for interpreting users' emotional subjective experience on the basis of physiological signals has led to various approaches. In this article, we focus on two different approaches toward emotion recognition : (1) the user-dependency of psycho-physiological data collected (i.e. do we choose to keep track of...

Starting from the assumptions that human-ambient in- telligence interaction will be improved by having more human-human like communications and that facial ex- pressions are fundamental in human-human communi- cations, we describe how we developed facial expres- sions for artificial agents based on a psychological the- ory developed by Scherer. In...

During face to face communication, it has been suggested that as much as 70% of what people communicate when talk- ing directly with others is through paralanguage involving multiple modalities combined together (e.g. voice tone and volume, body language). In an attempt to render human- computer interaction more similar to human-human com- municati...

Near to real-time emotion recognition is a promising task for human-computer interaction (HCI) and human-robot interaction (HRI). Using knowledge about the user's emotions depends upon the possibility to extract information about users' emotions during HCI or HRI without explicitly asking users about the feelings they are experiencing. To be able t...

In this article, we expose our research program in bio-affective-social computing aimed at guiding and participating to the development of multimodal socially intelligent agents in terms of: (1) decision-making and (2) communication. We present some of the results we obtained working on certain modules of the research paradigm which we chose to exp...

In this paper we describe the multimodal affective user interface (MAUI) we created to capture its users’ emotional physiological signals via wearable computers and visualize the categorized signals in terms of recognized emotion. MAUI aims at (1) giving feedback to the users about their emotional states via various modalities (e.g. mirroring the u...

In this article, we discuss the strong relationship between affect and cognition and the importance of emotions in multimodal human computer interaction (HCI) and user-modeling. We introduce the overall paradigm for our multi-modal system that aims at recognizing its users' emotions and at responding to them accordingly depending upon the current c...

We present a method applied to the emotional subjective ex-perience of the user for extracting emotional semantic information from Autonomic Nervous System (ANS) physiological signals (skin conduc-tance and heart rate) sensed via bio-sensors. We present a physiologically-inspired data structure based on intra-individual rule extraction used to reac...

This study exemplifies the integration of information systems (IS) behavioural science in the area of technology adoption and diffusion into the design science paradigm. We first present a research framework for triangulating design and behavioural science paradigms. From the design science perspective, we introduce an intelligent interface (Model...

Progress in computer graaphics over the last decade has ren- dered the creation of believable anthropomorphic graphical avatars pos- sible. Issues in rendering these animated graphical avatars believable and engaging during Human-Computer Interaction still remain. In this arti- cle, we focus on the animation of avatar's facial expressions. We expla...

In this article, we propose the design of a three-layered agent architecture in- spired from the Multilevel Process Theory of Emotion (Leventhal and Scherer, 1987). Our project aims at modeling emotions on an autonomous embodied robotic agent, ex- panding upon our previous work (Lisetti, et al., 2004). Our agent is designed to socially interact wit...

In this paper, we uncover a new potential application for multi-media technologies: affective intelligent car interfaces for enhanced driving safety. We also describe the experiment we conducted in order to map certain physiological signals (galvanic skin response, heart beat, and temperature) to certain driving-related emotions and states (Frustra...

In this paper we describe how to design a very simple tutor for Computer Assisted Instruction. In this work we adopted the basic ideas from the Belief Desire Intentions architecture and we implemented a simple model for the affective state of the avatar. The framework comprehends a simple model for personality, for the mood and for emotions and exp...

The development of an autonomous social robot, Cherry, is occurring in tandem with studies gaining potential user preferences, likes, dislikes, and perceptions of her features. Thus far, results have indicated that individuals 1) believe that service robots with emotion and personality capabilities would make them more acceptable in everyday roles...

Previous experiences show that it is possible for agents such as robots cooperating asynchronously on a sequential task to enter situations where one robot does not fulfill its obligations in a timely manner due to hardware or planning failure, unanticipated delays, etc., leaving the other robot in an infinite wait state. Our approach is derived fr...

In this article we describe a new approach to enhance presence technologies. First, we discuss the strong relationship between cognitive processes and emotions and how human physiology is uniquely affected when experiencing each emotion. Secondly, we introduce our prototype multimodal affective user interface. In the remainder of the paper we descr...

In this article we describe a new approach to enhance presence technologies. First, we discuss the strong relationship between cognitive processes and emotions and how human physiology is uniquely affected when experiencing each emotion. Secondly, we introduce our prototype multimodal affective user interface. In the remainder of the paper we descr...

u Abstract This study exemplifies the integration of IS behavioral science in the area of technology adoption and diffusion into the design science process. We first identify the computer-mediated paradox, as it exists in the tele-home health care setting. Specifically, we address the challenges of providing quality patient inclusive of affective a...

It has been long recognised in education that teaching and learning is a highly social and emotional activity. Students’ cognitive progress depends on their psychological predispositions such as their interest, confidence, sense of progress and achievement as well as on social interactions with their teachers and peers who provide them (or not) wit...

Previous experiences show that it is possible for agents such as robots cooperating asynchronously on a sequential task to enter situations where one robot does not fulfill its obligations in a timely manner due to hardware or planning failure, unanticipated delays, etc., leaving the other robot in an infinite wait state. Our approach is derived fr...

Accounting for a patient's emotional state is integral in medical care. Tele-health research attests to the challenge clinicians must overcome in assessing patient emotional state when modalities are limited (J. Adv. Nurs. 36(5) 668). The extra effort involved in addressing this challenge requires attention, skill, and time. Large caseloads may not...

In this paper we describe our work in progress for exploring the role of emotions in the behaviour and decision-making of artificial agents in modeling and simulation systems. The computational model of emotions we are using is based on a multi-level theory of emotions that accounts for the three layers identified in the human emotional system: the...

Human intelligence is being increasingly redefined to include the all-encompassing effect of emotions upon what used to be considered 'pure reason'. With the recent progress of research in computer vision, speech/prosody recognition, and bio-feedback, real-time recognition of affect will enhance human-computer interaction considerably, as well as a...

We present a general analytical model for predicting the reconstructed frame rate of an MPEG stream. Our model captures the temporal relationships between I-, P, and B-frames but is independent of the channel and media characteristics. We derive an adaptive ...

In this paper, we uncover a new potential application for multimedia technologies: car interfaces for enhanced driver's safety. We also describe the experiment we conducted in order to map certain physiological signals (galvanic skin response, heart beat, and temperature) to certain emotions (Neutral, Anger, Fear, Sadness, and Frustration). We demo...

This document is a summary of the panel discussion that was held at the workshop. Rather than recording the speakers' contributions, we asked them to summarize their views in a text. The document is organized according to the questions that were discussed in the panel. This discussion was collected and revised by Fiorella de Rosis and Eva Hudlicka.

In this article, we expose some of the issues raised by the critics of the neoclassical approach to rational agent modeling and we propose a formal approach for the design of artificial rational agents that includes some of the functions of emotions found in the human system. We suggest that emotions and rationality are closely linked in the human...

Our research combines two diverse strands of work in AI and cognitive science. We start from the principled paradigm of rational agent design based on decision theory. We then use this paradigm to formally dene the emotional states and personality of an arti- cial intelligent agent. We view the emotional states as the agent's decision-making modes,...

With the growing importance of information technology in our everydaylife, newtypes of applications are appearing that require the understanding of information in a broad sense. Information that includes affective and subjective content plays a major role not only in an individual’s cognitive processes but also in an individual’s interaction with o...

We describe our computational Affective Knowledge Repre- sentation (AKR) the hierarchical model of affect - includ- ing affect, mood, emotion, and personality - which we have developed for the design of socially intelligent agents. We describe a script implementation of emotion concepts used which we applied to create an intelligent user interface...

In this article, we describe an Affective Knowledge Repre- sentation (AKR) scheme to represent emotion schemata to be used in the design a variety of socially intelligent artificial agents. Our approach in this article and in the applications of our AKR scheme, focusses on the notion of "social expertise" of socially intelligent agents in terms of...

Generalized Partial Global Planning (GPGP) and its associated TÆMS hierarchical task network representation were developed as a domain-independent framework for coordinating the real-time activities of small teams of cooperative agents working ...

We use the formalism of decision theory to develop principled definitions of emotional states of a user, and we postulate that these notions should be useful in designing interactive entertainment systems.

We discuss here one of our projects, aimed at developing an automatic facial expression interpreter, mainly in terms of signaled emotions. We present some of the relevant findings on facial expressions from cognitive science and psychology that can be understood by and be useful to researchers in Human-Computer Interaction and Artificial Intelligen...

We use the formalism of decision theory to develop principled definitions of emotional states of a rational agent. We postulate that these notions are useful for rational agent design. First, they can serve as internal states controlling the allocation of computations and time devoted to cognitive tasks under external pressures. Second, they provid...

We use the formalism of decision theory to develop principled definitions of emotional states of a rational agent. We postulate that these notions are useful for rational agent design. First, they can serve as internal states controlling the allocation of computations and time devoted to cognitive tasks under external pressures. Second, they provid...

We use the formalisnl of decision theory to develop principled definitions of emotional states of a ra- tional agent. We postulate that these notions are useflfl for rational agent design. First, they can serve as internal states controlling the allocation of computations and time devoted to cognitive tasks under external pressures. Second, they pr...