No full-text available
To read the full-text of this research,
you can request a copy directly from the authors.
Affective Learning: Empathetic Agents with Emotional Facial and Tone of Voice Expressions
Abstract
Empathetic behavior has been suggested to be one effective way for Embodied Conversational Agents (ECAs) to provide feedback to learners' emotions. An issue that has been raised is the effective integration of parallel and reactive empathy. The aim of this study is to examine the impact of ECAs' emotional facial and tone of voice expressions combined with empathetic verbal behavior when displayed as feedback to students' fear, sad, and happy emotions in the context of a self-assessment test. Three identical female agents were used for this experiment: 1) an ECA performing parallel empathy combined with neutral emotional expressions, 2) an ECA performing parallel empathy displaying emotional expressions that were relevant to the emotional state of the student, and 3) an ECA performing parallel empathy by displaying relevant emotional expressions followed by emotional expressions of reactive empathy with the goal of altering the student's emotional state. Results indicate that an agent performing parallel empathy displaying emotional expressions relevant to the emotional state of the student may cause this emotion to persist. Moreover, the agent performing parallel and then reactive empathy appeared to be effective in altering an emotional state of fear to a neutral one.
... Aside from the engaging quality, the emotional/affective behavior of the VA not only serves as an alternative to captivate users' interest within group settings but also acts as another potent element for improving users' interactive and communicative experience [12,27,45]. Affective expressions encompass both direct verbal cues [74,101] and nonverbal ones, including facial expressions [11,12,27,52], bodily gestures [45,76,85], and voice tones [24,72,102]. ...
... Numerous studies have shown that VAs exhibiting a broad range of emotions and possessing unique personalities can enhance the user's perception of VAs [72,85]. Nam et al. [74] found that VA's dynamic emotions positively affected realism, learning effectiveness, and usefulness in the nursing training experiences. ...
... Nam et al. [74] found that VA's dynamic emotions positively affected realism, learning effectiveness, and usefulness in the nursing training experiences. Moridis et al. [72] observed that when a VA mirrors the user's emotions (e.g., happiness, sadness, and fear) through facial and vocal expression, it can effectively transform and reinforce one's emotions. The VA's rich facial expressions increased the user's belief, cooperation [11,91], and their co-presence [52]. ...
This study investigates how different virtual agent (VA) behaviors
influence subjects’ perceptions and group decision-making. Participants carried out experimental group discussions with a VA
exhibiting varying levels of engagement and affective behavior.
Engagement refers to the VA’s focus on the group task, whereas
affective behavior reflects the VA’s emotional state.
The findings revealed that VA’s engagements effectively captured participants’ attention even in the group setting and enhanced group synergy, thereby facilitating more in-depth discussion and producing better consensus.
On the other hand, VA’s affective behavior negatively affected the perceived social presence and trustworthiness.
Consequently, in the context of group discussion, participants preferred the engaged and non-affective VA to the non-engaged and affective VA.
The study provides valuable insights for improving the VA’s behavioral design as a team member for collaborative tasks.
... Aural responses from an AI system can be perceived in a distinct way compared to text responses from the system, due to not only modality effects, but also other stylistic elements used for delivering the content (e.g., tone). For instance, emotion (Moridis and Economides, 2012), conversational style and extroversion personality (e.g., Chang et al., 2018;Hoegen et al., 2019;Lee et al., 2006;Lee, 2000, 2001) embedded in computer "voices" are known to have significant effects on user perceptions of computer agents. Users react to certain computer voices differently compared to others even though they are all generated by machines, as shown by a long line of research in the Computers-Are-Social-Actors (CASA) paradigm: i.e., individuals show a tendency to treat computers as if they are humans, based on social norms affiliated with human-to-human communication, even when they know they are interacting with machines (Nass et al., 1994;Nass and Moon, 2000;Reeves and Nass, 1996). ...
... For instance, one study showed that users conformed more to computer voices that matched their gender . Another study found that when certain emotions (i.e., happiness, sadness, and fear) embedded in an embodied conversational agent's facial expression and tone were parallel to those of the user, the emotional state of the user tended to persist (Moridis and Economides, 2012). A more recent study showed that when an AI-driven conversational agent gradually matched the user as it detected and learned the users' considerate conversational style, it resulted in higher trustworthiness of the agent (Hoegen et al., 2019). ...
... For example, several speech emotion recognition (SER) algorithms are developed to detect users' emotions from acoustical cues, by labeling a voice input as a specific emotional category, e.g., happy, sad, or angry [23], [24], [25], or by predicting the valence and arousal of a voice input [26]. To respond to users' emotions, voice-based CAs have been relying on users' selfreported emotions [27] or conversational states such as "standby" or "listening" [28]. In human-human conversations, people show empathy-the ability to comprehend others' feelings and to re-experience them oneself-using emotive interjections (e.g., "WoW!") [29], [30]. ...
... Participants in a previous study regarded the perceived empathy of the therapist chatbots as the best thing of the experience [7]. In learning scenarios, previous works indicated that virtual agents performing empathy help students relieve emotions of fear and persist in learning, and get more perceptions of presence [27], [51]. The empathic CA also has potential in mental health applications [52], [53]. ...
Emotion is important for the conversational user interface. In prior research, conversational agents (CAs) employ natural language process techniques to create affective interaction based on text. However, the use of acoustic features of speech for voice-based CAs is under exploration. This work presents an acoustically emotion-aware CA that enables speech emotion recognition and stylizes responses with empathetic feedback and interjections. We conducted a user study in which 75 participants interacted with the CA under emotion stimulating to evaluate their perceived emotional intelligence (PEI). Our results show that the acoustically emotion-awareness increased the participants' PEI of the CA, and empathetic responses from the CA helped alleviate some participants' negative emotions. Our work provides implications for designing future CAs with better PEI.
... E-learning refers to using computers and online learning tools in a blended learning environment, focusing on collaborative online learning [2]. E-learning can be advantageous as it allows learners to progress at their own pace and access information at the time and place that suits them best [3,4]. However, it also presents some challenges. ...
... Claims have been made by Woolf et al. [18] that a computer-based tutor should be able to detect and analyze levels of motivation, confidence, boredom, frustration, and fatigue in order to provide relevant feedback for each of these states. Other researchers [3,4] have looked into detecting basic emotions in an e-learning situation. ey made use of ECAs to accomplish parallel empathy and thereafter reactive empathy that expresses emotion through voice and articulation. ...
Various studies have measured and analyzed learners’ emotions in both traditional classroom and e-learning settings. Learners’ emotions can be estimated using their text input, speech, body language, or facial expressions. The presence of certain facial expressions has shown to indicate a learner’s levels of concentration in both traditional and e-learning environments. Many studies have focused on the use of facial expressions in estimating the emotions experienced by learners. However, little research has been conducted on the use of analyzed emotions in estimating the learning affect experienced. Previous studies have shown that online learning can enhance students’ motivation, interest, attention, and performance as well as counteract negative emotions, such as boredom and anxiety, that students may experience. Thus, it is crucial to integrate modules into an existing e-learning platform to effectively estimate learners’ learning affect (LLA), provide appropriate feedback to both learner and lecturers, and potentially change the overall online learning experience. This paper proposes a learning affect estimation framework that employs relational reasoning for facial expression recognition and adaptive mapping between recognized emotions and learning affect. Relational reasoning and deep learning, when used for autoanalysis of facial expressions, have shown promising results. The proposed methodology includes estimating a learner’s facial expressions using relational reasoning; mapping the estimated expressions to the learner’s learning affect using the adaptive LLA transfer model; and analyzing the effectiveness of LLA within an online learning environment. The proposed research thus contributes to the field of facial expression recognition enhancing online learning experience and adaptive learning.
... E-learning refers to using computers and online learning tools in a blended learning environment, focusing on collaborative online learning [2]. E-learning can be advantageous as it allows learners to progress at their own pace and access information at the time and place that suits them best [3,4]. However, it also presents some challenges. ...
... Claims have been made by Woolf et al. [18] that a computer-based tutor should be able to detect and analyze levels of motivation, confidence, boredom, frustration, and fatigue in order to provide relevant feedback for each of these states. Other researchers [3,4] have looked into detecting basic emotions in an e-learning situation. ey made use of ECAs to accomplish parallel empathy and thereafter reactive empathy that expresses emotion through voice and articulation. ...
Relational Networks (RN), as one of the most widely used relational reasoning techniques, have achieved great success in many applications such as action and image analysis, speech recognition and text understanding. The use of relational reasoning via RN in neural networks has often been used in recent years. In these instances, RN is composed of various deep learning-based algorithms in simple plug-and-play modules. This is quite advantageous since it circumvents the need for features engineering. This paper surveys the emerging research of deep learning models that make use of RN in tasks such as Natural Language Processing (NLP), Action Recognition, Temporal Relational Reasoning as well as Facial Emotion Recognition (FER). Since, RNs are easy to integrate they have been used in various tasks such as NLP, Recurrent Neural Networks (RNN), Action Recognition, Image Analysis, Object Detection, Temporal Relational Reasoning, as well as for FER. This is due to the fact that RNs use bidirectional LSTM and CNN to solve relational reasoning problems at character and word level. In this paper a comparative review of all relational reasoning-based RN models using deep learning techniques is presented.
... These theories support that incorporating emotions is advantageous to dialogue system by allowing the dialogue system to emulate the conversational behavior of human beings and, at the same time, to strengthen the emotional connection with human users [37] . Emotion might also increase the user's engagement in the conversation [38,39] . On the other hand, it has been argued that emotion might introduce unpredictability into the system. ...
... However, emotion can be considered as explicit actions in the action space to display the affective behavior more straightforwardly. For example, a virtual agent could take different strategies for parallel empathy (mirroring the other one's emotion) and reactive empathy (providing insight to recover from other one's emotional states) [39] . 3. Long-term Empathy Modeling The display of empathy in dialogues is usually engaged in long-term activities. ...
Dialogue systems have achieved growing success in many areas thanks to the rapid advances of machine learning techniques. In the quest for generating more human-like conversations, one of the major challenges is to learn to generate responses in a more empathetic manner. In this review article, we focus on the literature of empathetic dialogue systems, whose goal is to enhance the perception and expression of emotional states, personal preference, and knowledge. Accordingly, we identify three key features that underpin such systems: emotion-awareness, personality-awareness, and knowledge-accessibility. The main goal of this review is to serve as a comprehensive guide to research and development on empathetic dialogue systems and to suggest future directions in this domain.
... These are systems capable of directly influencing the affect of learners. Emotional regulation systems are interventions that take many forms, for example, emotion-aware reinforcement systems aim to shift the learners' affective state from negative emotions to pro-learning emotions through various means, like changing the tone/expression/emotion of the tutoring agent (Gu et al., 2010;Moridis & Economides, 2012;Rathi & Deshpande, 2019), suggesting activities to increase the positive emotions of the students (Landowska, 2013), or changing the course materials to suit the emotional state of students (Moga & Antonya, 2012;Mohanan et al., 2017). The most common approach involves changing the emotions of the automated affective agent. ...
Affective computing is an emerging area of education research and has the potential to enhance educational outcomes. Despite the growing number of literature studies, there are still deficiencies and gaps in the domain of affective computing in education. In this study, we systematically review affective computing in the education domain. Methods: We queried four well-known research databases, namely the Web of Science Core Collection, IEEE Xplore, ACM Digital Library, and PubMed, using specific keywords for papers published between January 2010 and July 2023. Various relevant data items are extracted and classified based on a set of 15 extensive research questions. Following the PRISMA 2020 guidelines, a total of 175 studies were selected and reviewed in this work from among 3102 articles screened. The data show an increasing trend in publications within this domain. The most common research purpose involves designing emotion recognition/expression systems. Conventional textual questionnaires remain the most popular channels for affective measurement. Classrooms are identified as the primary research environments; the largest research sample group is university students. Learning domains are mainly associated with science, technology, engineering, and mathematics (STEM) courses. The bibliometric analysis reveals that most publications are affiliated with the USA. The studies are primarily published in journals, with the majority appearing in the Frontiers in Psychology journal. Research gaps, challenges, and potential directions for future research are explored. This review synthesizes current knowledge regarding the application of affective computing in the education sector. This knowledge is useful for future directions to help educational researchers, policymakers, and practitioners deploy affective computing technology to broaden educational practices.
... Researchers have identified various observable factors that influence human-AI interaction, including appearance (especially personified appearance) [118][119][120][121][122][123][124], voice characteristics like tone, pitch, and style [125][126][127][128][129][130][131][132][133][134][135], dialogue [136][137][138], movement and behavior [138][139][140], and emotional and social expression [138,[141][142][143][144][145]. Fine-tuning these factors allows AI system designers to enhance the impact of virtual agents or physical robots and strengthen their relationship with users. ...
This thesis investigates the psychological factors that influence belief in AI predictions, comparing them to belief in astrology- and personality-based predictions, and examines the "personal validation effect" in the context of AI, particularly with Large Language Models (LLMs). Through two interconnected studies involving 238 participants, the first study explores how cognitive style, paranormal beliefs, AI attitudes, and personality traits impact perceptions of the validity, reliability, usefulness, and personalization of predictions from different sources. The study finds a positive correlation between belief in AI predictions and belief in astrology- and personality-based predictions, highlighting a "rational superstition" phenomenon where belief is more influenced by mental heuristics and intuition than by critical evaluation. Interestingly, cognitive style did not significantly affect belief in predictions, while paranormal beliefs, positive AI attitudes, and conscientiousness played significant roles. The second study reveals that positive predictions are perceived as significantly more valid, personalized, reliable, and useful than negative ones, emphasizing the strong influence of prediction valence on user perceptions. This underscores the need for AI systems to manage user expectations and foster balanced trust. The thesis concludes with a proposal for future research on how belief in AI predictions influences actual user behavior, exploring it through the lens of self-fulfilling prophecy. Overall, this thesis enhances understanding of human-AI interaction and provides insights for developing AI systems across various applications.
... The current body of literature has documented various factors that affect how individuals perceive robots. For example, empirical research has documented that a robot's tone of voice (e.g., Moridis & Economides, 2012), pitch (e.g., Edwards et al., 2019), and gestures (e.g., Kose-Bagci et al., 2009) all influence perceptions and experiences of robot interactions. Though previous research has found that a variety of factors impact trust in robots (e.g., Bach et al., 2022;Kaplan et al., 2023), limited research focuses on how content or the subject of the conversation with AI or robots would affect trust. ...
... Despite the interesting results and significant implications, the current study also has several limitations that could be addressed by future studies. The commercially available VAs (e.g., Alexa, Siri, Cortana) all possess different skills and characteristics, such as emotional tone, vocal pitch and naturalness of voice(Moridis & Economides, 2012).Such differences may affect user responses to the examined tasks.Thus, future studies might consider testing various VAs, thereby improving the robustness of results. An examination of a more diverse list of VAs could also help identify other relevant design characteristics outside the scope of this study. ...
There is an emerging interest in examining user attitudes towards voice assistants (VAs); however, there is limited research on how user attitudes are formulated in different contexts. Drawing from the stereotype content models, the current study attempts to investigate how users perceive and evaluate voice assistants (VAs) in different contexts (i.e., functional vs. social tasks) based on warmth, competence and trustworthiness. Study 1 (N = 123) employs a within‐subjects design to examine how task type (functional vs. social) affects user perceptions and attitudes towards a VA (i.e., Google Assistant). Study 2 (N = 116) and Study 3 (N = 61) examine the boundary effect of perceived psychological power and ease of use. The findings show that attitude is significantly more positive in functional tasks (vs. social), and this effect is mediated by perceived competence. This indirect effect is also significantly moderated by perceived ease of use. Perceived warmth does not mediate the effect of social tasks on attitude, and trust in VAs is a direct outcome of functional tasks. Taken together, this study contributes to both theory and practice in many ways. Specifically, the findings are the first to demonstrate a direct effect of task type on consumer perceptions and attitudes. Additionally, the findings indicate that user evaluations of VAs are still dominated by user perceptions of the competence of the VAs.
... Another unique feature of any AI application, including CAI agents is their anthropomorphic nature. Researchers have been trying to incorporate emotions when designing the interactions of these devices with humans (Moridis & Economides, 2012). In this regard, the level of empathy and enjoyment that the CAI agents can provide is important, since both these factors are related to user's feelings and emotional state (Chin, Molefi, & Yi, 2020;Pal, Babakerkhell, & Zhang, 2021). ...
Artificial-intelligence (AI) powered conversational (CAI) agents have been growing in popularity. Like human personality, CAI agent personality can also impact the relationship users develop with them. In this work we adopt a multi-methodological systematic approach for generating CAI agent personality descriptors. 235 unique descriptors are obtained grouped into 8 personality dimensions. Having generated the personality framework, we propose a research model based on Stimulus Organism Response framework and Sternberg's Triangular Theory of Love for explaining how these personality traits lead to formation of love with these agents affecting their usage scenario. Results indicate that all the three components of love (passion, intimacy, and commitment) have significant effects on the usage scenario. However, depending upon the agent personality the nature of relationships varies. Cognitive personality results in fatuous love, affective personality results in consummate love, and social personality results in friendly relationship. A conceptual difference is observed between brand love, and the “love for AI.” The results of this research will not only help the HCI designers to create suitable machine personality for various AI-based agents, but it will also provide an unconventional approach towards examining adoption of emerging AI-based technologies by exploring the love aspect between man and machines.
... A speech synthesizer of this kind can endow an ECA with a personalised voice, which contributes to increasing human engagement in the ongoing conversation [3,19]. Furthermore, such a system can have a wider range of useful applications, such as speech-to-speech translation, navigation systems, and fairy tale-reciting. ...
A text-to-speech (TTS) synthesiser has to generate intelligible and natural speech while modelling linguistic and paralinguistic components characterising human voice. In this work, we present ITAcotron 2, an Italian TTS synthesiser able to generate speech in several voices. In its development, we explored the power of transfer learning by iteratively fine-tuning an English Tacotron 2 spectrogram predictor on different Italian data sets. Moreover, we introduced a conditioning strategy to enable ITAcotron 2 to generate new speech in the voice of a variety of speakers. To do so, we examined the zero-shot behaviour of a speaker encoder architecture, previously trained to accomplish a speaker verification task with English speakers, to represent Italian speakers’ voiceprints. We asked 70 volunteers to evaluate intelligibility, naturalness, and similarity between synthesised voices and real speech from target speakers. Our model achieved a MOS score of 4.15 in intelligibility, 3.32 in naturalness, and 3.45 in speaker similarity. These results showed the successful adaptation of the refined system to the new language and its ability to synthesise novel speech in the voice of several speakers.
... To provide an authentic experience, IVAs converse with humans via synthetic voice, allowing them to deliver humanlike expressions (Cho, et al., 2019). Indeed, the voice of IVAs has been the focus of a body of literature in terms of the vocal pitch (Elkins & Derrick, 2013), the naturalness of the voice (Nass & Gong, 1999), and the emotional tone (Moridis & Economides, 2012). Further, research has shown that humans tend to socially connect to computers showing human-like characteristics (Nass, et al., 1994). ...
... The framework must most likely perceive the client's feeling and play out the activities in like manner. It is fundamental to have a structure that incorporates different modules performing activities like discourse to content transformation, include extraction, and highlight determination and characterization of those highlights to recognize the feelings [4], [5]. ...
Facial expressions are the facial changes in light of a man's interior enthusiastic moods, aims, or social interchanges which are investigated by computer frameworks that endeavor to consequently examine and perceive facial movements and facial component changes from visual data. Now and again the facial expression recognition has been mistaken for feeling examination in the computer vision space prompts uncouth backings of acknowledgment process such as face detection, feature recognition and expression recognition in that way bringing about the issues of identifying impediments, enlightenments, posture varieties, acknowledgment, decrease in dimensionality, and so forth. Notwithstanding that, an appropriate computation and forecast of exact outcomes additionally enhances the execution of the facial Expression recognition. Henceforth, a detailed study was required about the strategies and systems utilized for unraveling the issues of facial expressions during the time of face detection, feature recognition and expression recognition. So thepaper displayed different current strategies and afterward basically considered the effort by the different researchers in the area of Facial Expression Recognition.
... In the past twenty years, many experiments have empirically demonstrated the power of empathic virtual agents to influence human affect including undoing negative feelings of frustration [27], increasing people's feelings of being cared for [28,29], altering people's feelings of fear into neutral feelings [30], and reducing public speaking anxiety [31]. While the fundamental tenets of affective mimicry and perspective taking drive computational empathy, the design space for computational empathy is combinatorically large. ...
How does empathy influence creative problem solving? We introduce a computational empathy intervention based on context-specific affective mimicry and perspective taking by a virtual agent appearing in the form of a well-dressed polar bear. In an online experiment with 1,006 participants randomly assigned to an emotion elicitation intervention (with a control elicitation condition and anger elicitation condition) and a computational empathy intervention (with a control virtual agent and an empathic virtual agent), we examine how anger and empathy influence participants' performance in solving a word game based on Wordle. We find participants who are assigned to the anger elicitation condition perform significantly worse on multiple performance metrics than participants assigned to the control condition. However, we find the empathic virtual agent counteracts the drop in performance induced by the anger condition such that participants assigned to both the empathic virtual agent and the anger condition perform no differently than participants in the control elicitation condition and significantly better than participants assigned to the control virtual agent and the anger elicitation condition. While empathy reduces the negative effects of anger, we do not find evidence that the empathic virtual agent influences performance of participants who are assigned to the control elicitation condition. By introducing a framework for computational empathy interventions and conducting a two-by-two factorial design randomized experiment, we provide rigorous, empirical evidence that computational empathy can counteract the negative effects of anger on creative problem solving.
... Affect is a key factor in human conversation and thus modelling empathy in embodied personal assistants is seen as an important research area (McQuiggan & Lester, 2007). For achieving affect and empathy it is on one side needed to understand well the user (Mairesse & Walker, 2010;Ortigosa et al., 2014), and on the other to provide suitable and expected reactions, which could be embodied in a personal agent via, for e.g., emotional facial and tone of voice expressions (Moridis & Economides, 2012) or even eye contact (Hardjasa & Nakazawa, 2020). ...
... This work can guide the implementation of HAIRs to increase trust in communication with HAIRs. A number of studies have been conducted to accumulate knowledge about how AI robot's paralinguistic nonverbal cues, such as tone (e.g., Moridis & Economides, 2012), pitch (e.g., Edwards et al., 2019;Niculescu et al., 2013), and gendered voice (e.g., Crowelly et al., 2009), as well as kinesic cues such as gestures (e.g., Kose-Bagci et al., 2009), and other body movements (e.g., Coleman, 2018), affect their communication with humans. Such studies should be continued in different communication settings and with different groups of human communication partners to provide more communication data for developing HAIRs. ...
As humanoid robot technology, anthropomorphized by artificial intelligence (AI), has rapidly advanced to introduce more human-resembling automated robots that can communicate , interact, and work like humans, we have begun to expect active interactions with Humanoid AI Robots (HAIRs) in the near future. Coupled with the HAIR technology development, the COVID-19 pandemic triggered our interest in using health care robots with many substantial advantages that overcome critical human vulnerabilities against the strong infectious COVID-19 virus. Recognizing the tremendous potential for the active application of HAIRs, this article explores feasible ways to implement HAIRs in health care and patient services and suggests recommendations for strategically developing and diffusing autonomous HAIRs in health care facilities. While discussing the integration of HAIRs into health care, this article points out some important ethical concerns that should be addressed for implementing HAIRs for health care services.
... According to the above findings and the strong potential of mobile sensing methodologies to detect LRES, more research is urgently needed to unobtrusively detect students' emotions and provide emotion-based personalized services or empathetic agents. Empathetic agents have been shown to be effective in persisting students' positive emotions and altering an emotional state of fear to a neutral one (C.N. Moridis & Economides, 2012). ...
This paper aims to provide the reader with a comprehensive background for understanding current knowledge on the use of non-intrusive Mobile Sensing methodologies for emotion recognition in Smartphone devices. We examined the literature on experimental case studies conducted in the domain during the past six years (2015-2020). Search terms identified 95 candidate articles, but inclusion criteria limited the key studies to 30. We analyzed the research objectives (in terms of targeted emotions), the methodology (in terms of input modalities and prediction models) and the findings (in terms of model performance) of these published papers and categorized them accordingly. We used qualitative methods to evaluate and interpret the findings of the collected studies. The results reveal the main research trends and gaps in the field. The study also discusses the research challenges and considers some practical implications for the design of emotion-aware systems within the context of Distance Education.
... Combining speech and gestures can also improve students' learning (Koumoutsakis et al., 2016). From the perspective of affective learning, facial and tone voice expressions can be used as feedback to learners' emotions (Moridis & Economides, 2012). Conversely, inappropriate facial expressions can harm students' learning (e.g. ...
This paper introduces a system that supports student-centered online one-to-one tutoring and evaluates the practical value of the system by running an experiment with 64 experienced mathematics teachers and 810 students in Grade 7. The experiment lasted for 50 days. A comprehensive evaluation was performed using students’ academic performance before and after usage of the system and the system log files. By classifying the students into active and inactive usage groups, it was determined that active students significantly outperformed inactive students on posttests, but with a small effect size. The results also suggested that high prior knowledge students tended to benefit more from using the system than low prior knowledge students. An explanation for this result was that students with a high level of prior knowledge were more likely to have good-quality interactions with their teachers. Therefore, although some advantages of this type of student-centered online one-to-one tutoring are observed, in this system, both the students and the teachers need to be further facilitated to produce more effective tutoring interactions.
... Although these studies are great examples of research on how to technically model an ECA's emotions, they do not primarily evaluate users' perceptions of emotional expressions. Other studies do focus on evaluating users' perceptions, researching effects of positive, negative and neutral emotional expressions of an ECA on students' social judgements, interest and self-efficacy (Kim et al., 2007) and effects of an ECA's emotional facial expressions and tone of voice on students' emotional states (Moridis & Economides, 2012). Although these studies show positive effects of an ECA's emotional expressions, they do not investigate effects of an ECA's emotional expressions on the quality of the relationship between the ECA and its users. ...
Embodied conversational agents (ECAs) could engage users in eHealth by building mutual understanding (i.e. rapport) via emotional expressions. We compared an ECA’s emotions expressed in text with an ECA’s emotions in facial expressions on users’ perceptions of rapport. We used a design, combining a happy or neutral facial expression with a happy or neutral textual expression. Sixty-three participants (mean, 4822 years) had a dialogue with an ECA on healthy living and rated multiple rapport items. Results show that participants’ perceived rapport for an ECA with a happy facial expression and neutral textual expression and an ECA with a neutral facial expression and happy textual expression was significantly higher than the neutral value of the rapport scale ( and , respectively). Furthermore, results show no significant difference in overall rapport between the conditions (), but a happy textual expression for an ECA with a neutral facial expression shows higher ratings of the individual rapport items helpfulness () and enjoyableness (). Future research should investigate users’ rapport towards an ECA with different emotions in long-term interaction and how a user’s age and personality and an ECA’s animations affect rapport building. Optimizing rapport building between a user and an ECA could contribute to achieving long-term interaction with eHealth.
... Overall it is accepted that, positive LREs serve as a determinant of students' emotional well-being and achievement [10], while negative LREs tend to yield negative impact on learning outcomes [14,15]. For this reason, several studies have proposed the use of emotional agents [16] or other emotional feedback methodologies to induce positive emotions to students. ...
Although there are multiple approaches for making lectures more interactive, Game-Based Learning (GBL) tends to achieve the highest impact on students’ emotional engagement. To this end, this study seeks to implement a mobile GBL approach in a Distance Education (DE) course to investigate the students’ gaming experience and learning related emotions. The experiment was conducted on 26 post-graduate distance students using a Kahoot! mobile game and a self-reported instrument. Quantitative analysis was implemented to measure the students’ perceived i) competence, ii) concentration and ii) immersion, and the learning related emotions of i) enjoyment, ii) boredom, iii) confusion, and iv) anxiety. Sentiment analysis revealed a highly positive emotional attitude towards mobile GBL in DE and highlighted the prevalent emotions of joy and competence. Thematic content analysis was applied to investigate the gaming features that caused negative or positive emotions. Time limit and music/sound were proved to cause negative emotions, while multimedia, colors, learnability, and sequencing were reported as positive emotional antecedes. Competition revealed mixed outcomes. Overall, this study provides with useful insights that can be used by educators and emotional designers to increase engagement and learning performance in DE.
... Moreover, emotion-based personalized services or empathetic agents can be provided. As a fact, empathetic agents have been shown to be effective in persisting positive emotions and altering an emotional state of fear to a neutral one [45]. ...
Emotion recognition is essential for assessing human emotional
states and predicting user behavior to provide appropriate and
personalized feedback. The wide range of Smartphones with accelerometers, microphones, GPSs, gyroscopes, and more motivate
researchers to explore the automatic emotion detection through
Smartphone sensors. To this end, mobile sensing can facilitate the
data retrieval process in a non-intrusive way without disturbing
the user’s experience. This study seeks to contribute to the field of
non-intrusive mobile sensing for emotion recognition by detecting
user emotions via accelerometer and gyroscope sensors in Smartphones. A prototype gaming app was designed and a sensor log app
for Android OS was used to monitor the users’ sensor data while
interacting with the game. The recorded data from 40 users was
processed and used to train different classifiers for two emotions: a
positive (enjoyment) and a negative (frustration) one. The validation study demonstrates a high prediction of 87.90% for enjoyment
and 89.45% for frustration. Our findings indicate that by analyzing
accelerometer and gyroscope data, it is possible to make efficient
predictions of a user’s emotional state. The proposed model and its
empirical development and validation are described in this paper.
... Positive affective states can be used to make personalized pedagogical decisions such as to detect what a student appreciates or provide empathic feedback [4,34,35]. However, this personalization could be delivered regardless of whether the student experiences one or several positive emotions. ...
Emotions in Intelligent Tutoring Systems (ITS) are often modeled as single affective states, however there is evidence that emotions co-occur during learning, with implications for affect-aware ITS that need to have a comprehensive understanding of a student’s affective state to react accordingly. In this paper we broaden the evidence that emotions co-occur in an educational context, and present a first attempt to predict these co-occurrences from data, using the MetaTutor ITS as a test-bed. We show that boredom+frustration, as well as curiosity+anxiety, frequently co-occur in MetaTutor, and that we can predict when these emotions co-occur significantly better than a baseline using eye-tracking and interaction data. These findings provide a first step toward building affect-aware ITS that can adapt to these complex co-occurring affective states.
... For example, several speech emotion recognition (SER) algorithms are developed to detect users' emotions from acoustical cues, by labeling a voice input as a certain emotional category, e.g., happy, sad or angry [1,22,46], or by predicting the valence and arousal of a voice input [5]. To respond to users' emotions, voice-based CAs have been relying on users' self-reported emotions, e.g., [23]. During human-to-human interaction, people often express emotions such as empathy-the ability to comprehend other's feelings and to re-experience them oneselfusing emotive interjections (e.g., "WoW!") [9,39]; because of this, such interjections and fllers are also inserted in voice-based CAs to improve users' PEI [6]. ...
... The provision of tutor or automated affective (cr3.1) feedback (e.g., an empathetic emotion or an encouragement provided by a computer agent) is also able to support perceived competence [34] and relatedness [60]. Moreover, it can promote student collaboration and engagement [37,59]. ...
Motivation is an important issue to consider when designing learning activities, including mobile learning and assessment. While previous research provides evidence for the motivational impact of mobile learning, not many pedagogical frameworks exist for the design of mobile-assisted learning and assessment. The current study is grounded in the Self-Determination Theory of motivation and proposes a pedagogical framework for mobile-assisted formative assessment, aiming at enhancing student motivation. For a preliminary evaluation of the framework, fifty-one students from a public European high school participated in a series of formative assessment activities. The tasks that were implemented according to the proposed mobile-based formative assessment framework had a significant positive impact on student perceived levels of autonomy, competence, and relatedness, enhancing students’ intrinsic motivation levels. Study findings highlighted the capacity of the proposed framework to guide the design of mobile-based formative assessment activities that enhance and promote student motivation. The study makes a theoretical contribution by proposing a framework that aligns mobile learning and assessment with elements of the Self-Determination Theory of motivation and also has a practical contribution by implementing mobile learning and assessment practices that have the potential to promote student motivation.
... Cues broadly constitute any sensory information (e.g., colors, setting, and dialogue) accessible in a communication environment (see Xu and Liao, 2020 for a review). Research to date primarily focuses on VHs' detection and response to cues originating from the user, including the tone of voice (Moridis and Economides, 2012), facial expression, and posture (Vinayagamoorthy et al., 2006;Karg et al., 2013). At present, there is limited knowledge on how and when VHs can and should attend to cues originating from the user's environment, which include cues within the immediate social context (e.g., pointing toward an object) as well as those outside (e.g., a phone ringing). ...
Virtual humans (VHs)—automated, three-dimensional agents—can serve as realistic embodiments for social interactions with human users. Extant literature suggests that a user’s cognitive and affective responses toward a VH depend on the extent to which the interaction elicits a sense of copresence, or the subjective “sense of being together.” Furthermore, prior research has linked copresence to important social outcomes (e.g., likeability and trust), emphasizing the need to understand which factors contribute to this psychological state. Although there is some understanding of the determinants of copresence in virtual reality (VR) (cf. Oh et al., 2018), it is less known what determines copresence in mixed reality (MR), a modality wherein VHs have unique access to social cues in a “real-world” setting. In the current study, we examined the extent to which a VH’s responsiveness to events occurring in the user’s physical environment increased a sense of copresence and heightened affective connections to the VH. Participants (N = 65) engaged in two collaborative tasks with a (nonspeaking) VH using an MR headset. In the first task, no event in the participant’s physical environment would occur, which served as the control condition. In the second task, an event in the participants’ physical environment occurred, to which the VH either responded or ignored depending on the experimental condition. Copresence and interpersonal evaluations of the VHs were measured after each collaborative task via self-reported measures. Results show that when the VH responded to the physical event, participants experienced a significant stronger sense of copresence than when the VH did not respond. However, responsiveness did not elicit more positive evaluations toward the VH (likeability and emotional connectedness). This study is an integral first step in establishing how and when affective and cognitive components of evaluations during social interactions diverge. Importantly, the findings suggest that feeling copresence with VH in MR is partially determined by the VHs’ response to events in the actual physical environment shared by both interactants.
... The expressive behaviors of iCat include spoken utterances that are divided into supportive categories of information support, tangible assistance, esteem support and emotional support. Moridis and Economides (2012) use parallel empathy and reactive empathy based on Davis's (1994) definition of empathy. In their implementation of an empathic tutoring agent, they use six basic emotions to show pedagogical feedback to students' happy, sad and fear emotions extracted from facial expressions. ...
Computational modeling of empathy has recently become an increasingly popular way of studying human relations. It provides a way to increase our understanding of the link between affective and cognitive processes and enhance our interaction with artificial agents. However, the variety of fields contributing to empathy research has resulted in isolated approaches to modeling empathy, and this has led to various definitions of empathy and an absence of common ground regarding underlying empathic processes. Although this diversity may be useful in that it allows for an in-depth examination of various processes linked to empathy, it also may not yet provide a coherent theoretical picture of empathy. We argue that a clear theoretical positioning is required for collective progress. The aim of this article is, therefore, to call for a holistic and multilayered view of a model of empathy, taken from the rich background research from various disciplines. To achieve this, we present a comprehensive background on the theoretical foundations, followed by the working definitions, components, and models of empathy that are proposed by various fields. Following this introduction, we provide a detailed review of the existing techniques used in AI research to model empathy in interactive agents, focusing on the strengths and weaknesses of each approach. We conclude with a discussion of future directions in this emerging field.
... Modality effects and the role of human likeness in VA interactions VA research in speech has tended to focus on how certain characteristics embedded in the virtual agent's voice (e.g., emotional tone, 5 vocal pitch, 6 naturalness of voice 7 ), usually coupled with the virtually embodied agent's facial cues (e.g., smiling vs. neutral facial expressions 6,8 ), can enhance humanlike perceptions of VAs. In reality, major VAs on the market are capable of delivering human-like impressions to users through simply one type of assigned synthetic voice even without signaling particular anthropomorphic physical cues (e.g., facial expression). ...
Paying attention to the rising popularity of virtual assistants (VAs) that offer unique user experiences through voice-centered interaction, this study examined the effects of modality, device, and task differences on perceived human likeness of, and attitudes toward, voice-activated VAs. To do so, a 2 (modality: voice vs. text) × 2 (device: mobile vs. laptop) × 2 (task type: hedonic vs. utilitarian) mixed factorial experimental design was employed. Findings suggest that voice (vs. text) interaction leads to more positive attitudes toward the VA system mediated by heightened perceived human likeness of the VA, but only with utilitarian (vs. hedonic) tasks. Interestingly, laptop (vs. mobile phone) interaction also enhanced perceived human likeness of the VA. This study offers theoretical and practical implications for VA research by exploring the combinational effects of modality, device, and task differences on user perceptions through human-like interactions.
... In a slightly different context, Andrade (2017) used multimodal data (motion and gaze) to show how students' explanations of feedback loops differ while controlling an embodied simulation. Finally, Moridis and Economides (2012) provided effective feedback using Embodied Conversational Agents based on emotional facial expression and speech. The exhaustive list of multimodal data applications are beyond the scope of the work presented in this paper. ...
Students' on‐task engagement during adaptive learning activities has a significant effect on their performance, and at the same time, how these activities influence students' behavior is reflected in their effort exertion. Capturing and explaining effortful (or effortless) behavior and aligning it with learning performance within contemporary adaptive learning environments, holds the promise to timely provide proactive and actionable feedback to students. Using sophisticated machine learning (ML) algorithms and rich learner data, facilitates inference‐making about several behavioral aspects (including effortful behavior) and about predicting learning performance, in any learning context. Researchers have been using ML methods in a “black‐box” approach, ie, as a tool where the input data is the learner data and the output is a given class from the chosen construct. This work proposes a methodological shift from the “black‐box” approach to a “grey‐box” approach that bridges the hypothesis/literature‐driven (feature extraction) “white‐box” approach with the computation/data‐driven (feature fusion) “black‐box” approach. This will allow us to utilize data features that are educationally and contextually meaningful. This paper aims to extend current methodological paradigms, and puts into practice the proposed approach in an adaptive self‐assessment case study taking advantage of new, cutting‐edge, interdisciplinary work on building pipelines for educational data, using innovative tools and techniques.
Practitioner Notes
What is already known about this topic Capturing and measuring learners' engagement and behavior using physiological data has been explored during the last years and exhibits great potential.
Effortless behavioral patterns commonly exhibited by learners, such as “cheating,” “guessing” or “gaming the system” counterfeit the learning outcome.
Multimodal data can accurately predict learning engagement, performance and processes.
What this paper adds Generalizes a methodology for building machine learning pipelines for multimodal educational data, using a modularized approach, namely the “grey‐box” approach.
Showcases that fusion of eye‐tracking, facial expressions and arousal data provide the best prediction of effort and performance in adaptive learning settings.
Highlights the importance of fusing data from different channels to obtain the most suited combinations from the different multimodal data streams, to predict and explain effort and performance in terms of pervasiveness, mobility and ubiquity.
Implications for practice and/or policy Learning analytics researchers shall be able to use an innovative methodological approach, namely the “grey‐box,” to build machine learning pipelines from multimodal data, taking advantage of artificial intelligence capabilities in any educational context.
Learning design professionals shall have the opportunity to fuse specific features of the multimodal data to drive the interpretation of learning outcomes in terms of physiological learner states.
The constraints from the educational contexts (eg, ubiquity, low‐cost) shall be catered using the modularized gray‐box approach, which can also be used with standalone data sources.
... Facial expression is one of the primary emotional displays used in VAs. Many of the VA facial systems (e.g., [39,62]) are designed following the psychological-based models of human emotion facial expression, such as Facial Action Coding System (FACS) [24] or the circumplex model of affect [8]. Gestures and body movements of a VA, if visible, can be augmented with special effects according to animation principles (e.g., exaggeration, slow in/out, arcs, timing [46]) and serve as non-verbal social emotional cues [87]. ...
An effective virtual agent (VA) that serves humans not only completes tasks efficaciously, but also manages its interpersonal relationships with users judiciously. Although past research has studied how agents apologize or seek help appropriately, there lacks a comprehensive study of how to design an emotionally intelligent (EI) virtual agent. In this paper, we propose to improve a VA's perceived EI by equipping it with personality-driven responsive expression of emotions. We conduct a within-subject experiment to verify this approach using a medical assistant VA. We ask participants to observe how the agent (displaying a dominant or submissive trait, or having no personality) handles user challenges when issuing reminders and rate its EI. Results show that simply being emotionally expressive is insufficient for suggesting VAs as fully emotionally intelligent. Equipping such VAs with a consistent, distinctive personality trait (especially submissive) can convey a significantly stronger sense of EI in terms of the ability to perceive, use, understand, and manage emotions, and can better mitigate user challenges.
... When it comes to audio cues signaled by virtual agents, studies tended to focus on particular variations in voice output (e.g., emotional tone of voice [21], vocal pitch [6]), with stronger interests in embodiment and non-verbal cues beyond voice (e.g., facial expression), to explore how human-like virtual agents can be perceived by users [9]. On the other hand, the pure voice effects have gained less attention. ...
MModern day voice-activated virtual assistants allow users to share and ask for information that could be considered as personal through different input modalities and devices. Using Google Assistant, this study examined if the differences in modality (i.e., voice vs. text) and device (i.e., smartphone vs. smart home device) affect user perceptions when users attempt to retrieve sensitive health information from voice assistants. Major findings from this study suggest that voice (vs. text) interaction significantly enhanced perceived social presence of the voice assistant, but only when the users solicited less sensitive health-related information. Furthermore, when individuals reported less privacy concerns, voice (vs. text) interaction elicited positive attitudes toward the voice assistant via increased social presence, but only in the low (vs. high) information sensitivity condition. Contrary to modality, the device difference did not exert any significant impact on the attitudes toward the voice assistant regardless of the sensitivity level of the health information being asked or the level of individuals' privacy concerns.
... AutoTutor comes with a long list of published results (D'Mello et al., 2011) successfully demonstrating both cognitive and affective skills (empathy). Moridis and Economides (Moridis and Economides, 2012) report on the impact of Embodied Conversational Agents (ECAs) on the respondent's affective state (sustain or modify) though corresponding empathy, either parallel (express harmonized emotions) or reactive (stimulate different or even contradictory emotions). EMOTE is another tool that can be integrated in existing agents, enriching their emotionality towards their improved learning performance and emotion wellbeing (Castellano et al., 2013). ...
Despite the visionary tenders that emerging technologies bring to education, modern learning environments such as MOOCs or Webinars still suffer from adequate affective awareness and effective feedback mechanisms, often leading to low engagement or abandonment. Artificial Conversational Agents hold the premises to ease the modern learner’s isolation, due to the recent achievements of Machine Learning. Yet, a pedagogical approach that reflects both cognitive and affective skills still remains undelivered. The current paper moves towards this direction, suggesting a framework to build pedagogical driven conversational agents based on Reinforcement Learning combined with Sentiment Analysis, also inspired by the pedagogical learning theory of Core Cognitive Skills.
Artificial intelligence (AI) driven chatbots provide instant feedback to support learning. Yet, the impacts of different feedback types on behavior and brain activation remain underexplored. We investigated how metacognitive, affective, and neutral feedback from an educational chatbot affected learning outcomes and brain activity using functional near-infrared spectroscopy. Students receiving metacognitive feedback showed higher transfer scores, greater metacognitive sensitivity, and increased brain activation in the frontopolar area and middle temporal gyrus compared to other feedback types. Such activation correlated with metacognitive sensitivity. Students receiving affective feedback showed better retention scores than those receiving neutral feedback, along with higher activation in the supramarginal gyrus. Students receiving neutral feedback exhibited higher activation in the dorsolateral prefrontal cortex than other feedback types. The machine learning model identified key brain regions that predicted transfer scores. These findings underscore the potential of diverse feedback types in enhancing learning via human-chatbot interaction, and provide neurophysiological signatures.
Background
Effective doctor-patient relationships hinge on robust communication skills, with non-verbal communication techniques (NVC) often overlooked, particularly in online synchronous interactions. This study delves into the exploration of NVC types during online feedback sessions for communication skill activities in a medical education module.
Methods
A cohort of 100 first-year medical students and 10 lecturers at the Faculty of Medicine, Universiti Kebangsaan Malaysia (UKM), engaged in communication skills activities via Microsoft Teams. Sessions were recorded, and lecturer NVC, encompassing body position, facial expressions, voice intonation, body movements, eye contact, and paralinguistics, were meticulously observed. Following these sessions, students provided reflective writings highlighting their perceptions of the feedback, specifically focusing on observed NVC.
Results
The study identified consistent non-verbal communication patterns during feedback sessions. Lecturers predominantly leaned forward and toward the camera, maintained direct eye contact, and exhibited dynamic voice intonation. They frequently engaged in tactile gestures and paused to formulate thoughts, often accompanied by filler sounds like “um” and “okay.” This consistency suggests proficient use of NVC in providing synchronous online feedback. Less observed NVC included body touching and certain paralinguistic cues like long sighs. Initial student apprehension, rooted in feelings of poor performance during activities, transformed positively upon observing the lecturer’s facial expressions and cheerful intonation. This transformation fostered an open reception of feedback, motivating students to address communication skill deficiencies. Additionally, students expressed a preference for comfortable learning environments to alleviate uncertainties during feedback reception. Potential contrivances in non-verbal communication (NVC) due to lecturer awareness of being recorded, a small sample size of 10 lecturers limiting generalizability, a focus solely on preclinical lecturers, and the need for future research to address these constraints and explore diverse educational contexts.
Conclusion
Medical schools globally should prioritize integrating NVC training into their curricula to equip students with essential communication skills for diverse healthcare settings. The study’s findings serve as a valuable reference for lecturers, emphasizing the importance of employing effective NVC during online feedback sessions. This is crucial as NVC, though occurring online synchronously, remains pivotal in conveying nuanced information. Additionally, educators require ongoing professional development to enhance proficiency in utilizing NVC techniques in virtual learning environments. Potential research directions stemming from the study’s findings include longitudinal investigations into the evolution of NVC patterns, comparative analyses across disciplines, cross-cultural examinations, interventions to improve NVC skills, exploration of technology’s role in NVC enhancement, qualitative studies on student perceptions, and interdisciplinary collaborations to deepen understanding of NVC in virtual learning environments.
Constructive interactions and knowledge integration activities are methods commonly used for learning; however, establishing successful coordination becomes a hurdle in computer-mediated collaborations. The development of systems to facilitate communication activities in such situations has been attempted, but models are still required for capturing learners’ interactions and detecting their quality. This study explored several types of verbal and nonverbal behaviors of learners that can be implemented while designing tutoring systems to effectively capture their interaction processes in scenarios where learners engage in collaborative learning mediated by a pedagogical conversational agent (PCA). This study focused on the degree of behavior recurrence of each speaker, which is considered suitable for observing levels of effectiveness. Specifically, this study focused on three indicators—gaze synchronization, language conformance, and emotional matching through facial expression—to establish a system-based index for measuring learners’ collaborative processes such as synchronization. This study experimentally examined the relationship between these indicators and the performance and process of collaborative learning among 44 learners while using PCA for facilitation. Subsequently, numerous dependent variables in the collaborative learning process were predicted using the three proposed indicators. However, no significant correlation was established between learning performance and the indicators used. These findings show that the recurrence of indicators is useful for estimating the collaborative learning process and that these indicators can be used in the development of learning support systems to trace learners’ achievements in successful interactions.
The unprecedented global movement of school education to find technological and intelligent solutions to keep the learning ecosystem working was not enough to recover the impacts of Covid-19, not only due to learning-related challenges but also due to the rise of negative emotions, such as frustration, anxiety, boredom, risk of burnout and the so-called “Covid fatigue”. Although this is not a new problem, it was deepened during the pandemic, and we need to face old and new challenges in different ways. Despite focusing only on the learning system’s inefficiencies and on the hegemony of solutions to tackle the learning gap, we also need to shed light on the strengths and the positive aspects of the learning process to promote wellbeing. As highlighted by John Self, an intelligent tutoring system would behave as if it genuinely cared about the student’s success. This note from John Self sheds light on the importance and reflection of what success means and for whom. This study presents a roadmap for positive psychology and artificial intelligence in education. It is intended to identify and understand how the intersection of Positive Psychology and Artificial Intelligence in Education can support the promotion of learning and wellbeing of students, teachers, and other educational stakeholders. As such, a bibliometric analysis of positive psychology and artificial intelligence in education was made as the so-called Positive Artificial Intelligence in Education (P-AIED). The search string was performed in 2021, and the total number of studies gathered was 10,777. After all the PRISMA steps, 256 studies were approved according to the inclusion criteria. The main conclusions were the high number of institutions and researchers with related publications indicate a new trend for the community of AIED; the high number of collaboration from different countries indicates a possible global movement toward P-AIED; Positive Emotion and Engagement were the main Positive Psychology constructs identified in the studies; the lack of well-grounded theories of Positive Psychology indicates an excellent research opportunity; Positive Learning Analytics (P-LA), Positive Educational Data Mining (P-EDM) and Positive Intelligent Tutoring Systems (P-ITS) are three hot topics for P-AIED.
This paper investigates basic prosodic features of speech (duration, pitch/F0 and intensity) for the analysis and classification of Ibibio (New Benue Congo, Nigeria) emotions, at the suprasegnental (sentence, word, and syllable) level. We begin by proposing generic hypothesis/baselines, representing a cache of research works documented over the years on emotion effects of neutral speech on western languages, and adopt the circumplex model for the effective representation of emotions. Our methodology uses standard approach to speech processing and exploits machine learning for the classification of seven emotions (anger, fear, joy, normal, pride, sadness, and surprise) obtained from male and female speakers. Analysis of feature-emotion correlates reveal that syllable (duration) units yield the least standard deviation across selected emotions for both genders, compared to other units. Also, there appear to be consistency for word and syllable units – as both genders show same duration correlate patterns. For F0 and intensity features, our findings agree with the literature, as high activation effects tend to produce higher F0 and intensity values, compared to low activation effects, but neutral and low activation effects produce the lowest pitch/F0 and intensity values (for both genders). A classification of the emotions yields interesting results, as classification accuracies and errors remarkably improved in emotion-F0 and emotion-intensity classification for support vector machine (SVM) and decision tree (DT) classifiers, but the highest classification accuracies were produced by the three classifiers at the sentence unit/level for fear emotion, with the k-nearest neighbour classifier (k-NN) leading (DT: 90%, SVM: 90%, k-NN: 92.40%).KeywordsEmotion recognitionIbibio affectsMachine learningProsodic featuresSpeech processingTone languages
Affective computing, a field that bridges the gap between human affect and computational technology, has witnessed remarkable technical advancement. However, theoretical underpinnings of affective computing are rarely discussed and reviewed. This paper provides a thorough conceptual analysis of the literature to understand theoretical questions essential to affective computing and current answers. Inspired by emotion theories, we proposed the function-component-representation (FCR) framework to organize different conceptions of affect along three dimensions that each address an important question: function of affect (why compute affect), component of affect (how to compute affect), and representation of affect (what affect to compute). We coded each paper by its underlying conception of affect and found preferences towards affect detection, behavioral component, and categorical representation. We also observed coupling of certain conceptions. For example, papers using the behavioral component tend to adopt discrete representation, whereas papers using the physiological component tend to adopt dimensional representation. The FCR framework is not only the first attempt to organize different theoretical perspectives in a systematic and quantitative way, but also a blueprint to help conceptualize an AC project and pinpoint new possibilities. Future work may explore how the identified frequencies of FCR framework combinations may be applied in practice.
The widespread use of chatbots is a reality and their application in higher education is promising. Understanding higher education users’ expectations for the use of chatbots in education is important for the design and development of new solutions. The present investigation documents how higher education users envision the pedagogical uses of chatbots in higher education, and how experts in the domain of education chatbots perceive the potential benefits and challenges related to the use of chatbots in education. A qualitative inquiry was undertaken based on 22 semi-structured interviews with higher-education students and instructors, and experts from the fields of Artificial Intelligence and educational chatbots. Based on our findings, the envisioned pedagogical uses of chatbots can be categorized in terms of chronological integration into the learning process: prospective, on-going, and retrospective. Under each one of those higher-order categories, specific learning domains can be supported (i.e., cognitive, affective), besides administrative tasks. Benefits and challenges foreseen in the use of pedagogical chatbots are presented and discussed. The findings of this study highlight the manner in which higher-education users envision the use of chatbots in education, with potential implications on the creation of specific pedagogical scenarios, accounting also for the learning context, chatbot technology, and pedagogies that are deemed appropriate in each scenario.
Affect has been the subject of increasing attention in cognitive accounts of learning. Many intelligent tutoring systems now seek to adapt pedagogy to student affective and motivational processes in an effort to increase the effectiveness of tutorial interaction and improve learning outcomes. To this end, recent work has begun to investigate the emotions experienced during learning in a variety of environments. In this paper we extend this line of research by investigating the affective transitions that occur throughout narrative-centered learning experiences. Further analysis differentiates the likelihood of affective transitions stemming from pedagogical agent empathetic responses to student affect.
Individual emotions play a crucial role during any learning interaction. Identifying a student's emotional state and providing personalized feedback, based on integrated pedagogical models, has been considered to be one of the main limits of traditional tools of e-learning. This paper presents an empirical study that illustrates how learner mood may be predicted during online self-assessment tests. Here a previous method of determining student mood has been refined based on the assumption that the influence on learner mood of questions already answered declines in relation to their distance from the current question. Moreover, this paper sets out to indicate that ldquoexponential logicrdquo may help produce more efficient models, if integrated adequately with affective modelling. The results show that these assumptions may prove useful to future research.
This paper describes the design and evaluation of an animated tutoring agent which used emotion regulation strategies to encourage better learning. The evaluation provided some preliminary evidence that the agent could reduce the amount of negative emotion experienced while using a computer aided learning tool.
The aim of this survey is to provide an overview of the various components of "computer aided affective learning systems." The research is classified into 3 main scientific areas that are integral parts of the development of these kinds of systems. The three main scientific areas are: I) emotions and their connection to learning; ii) affect recognition; and iii) emotional instruc- tion and design. Affective learning instructional technology is a new, multi- disciplinary research area, which has been developed during the last decade. This article depicts the development of the core relevant areas and describes the concerns.
This study investigated the effects of empathetic response (responsive vs. non-responsive) and gender (male vs. female) of pedagogical agents as learning companions (PALs) on learner interest, self-efficacy, social judgments of PAL persona, and learning. Fifty-six pre-service teachers were randomly assigned to one of four conditions. The results revealed that students who worked with a PAL who responded with empathy to learners' affect showed significantly higher interest (p < .05) and self-efficacy in the tasks (p < .05) than students with a non-responsive PAL. Also, students rated the persona of a male PAL more favorably than that of a female PAL (p < .05). The findings imply that PAL/learner relationships in computer-based learning might be consistent with human relationships in traditional classrooms.
The present study examines if and how the "gender" of computer-generated speech affects the user's perception of the computer and their conformity to the computer's recommendation. Presented with a series of social-dilemma situations, participants made a decision after listening to the computer's argument for one of the two choices in a 2 (TTS gender: male vs. female) by 2 (participant gender: male vs. female) experiment. Consistent with gender stereotypes, the male-voiced computer exerted greater influence on the user's decision than the female-voiced computer and was perceived to be more socially attractive and trustworthy. More strikingly, gendered synthesized speech triggered social identification processes, such that female subjects conformed more to the female-voiced computer, while males conformed more to the male-voiced computer (controlling for the main effect). Similar identification effects were found on social attractiveness and trustworthiness of the computer.
The role that affective states play in learning was investigated from the perspective of a constructivist learning framework. We observed six different affect states (frustration, boredom, flow, confusion, eureka and neutral) that potentially occur during the process of learning introductory computer literacy with AutoTutor, an intelligent tutoring system with tutorial dialogue in natural language. Observational analyses revealed significant relationships between learning and the affective states of boredom, flow and confusion. The positive correlation between confusion and learning is consistent with a model that assumes that cognitive disequilibrium is one precursor to deep learning. The findings that learning correlates negatively with boredom and positively with flow are consistent with predictions from Csikszentmihalyi's analysis of flow experiences.
This study investigated the transitions between affective states (i.e., boredom, flow, confusion, frustration, delight, and surprise) during learning while college students were tutored in computer literacy by AutoTutor, an automated tutoring system with natural language dialogue. Videos of participants' faces and the interaction histories were recorded and then played back for the participants to judge their own affective states. We developed a metric to measure the relative likelihood of transitioning from an affective state at time t i to a subsequent affective state at time t i+1 . Several significant trajectories between affective states were identified. Instructional implications are discussed in the context of an expanded version of a cognitive disequilibrium model.
This paper provides a synthesis of our research towards the devel-opment of an affect-sensitive Intelligent Tutoring System called AutoTutor. The affect-sensitive AutoTutor detects the emotions (boredom, flow/engagement, confusion, frustration) of a learner by monitoring conversa-tional cues, gross body language, and facial features. It is also mindful of the learners' affective and cognitive states in selecting its pedagogical and motiva-tional dialogue moves. Finally, the AutoTutor embodied pedagogical agent synthesizes affective responses through animated facial expressions and modu-lated speech. The paper provides an overview of our theoretical framework, methodology, implementation details, and results.
The Tutoring Research Group at the University of Memphis has developed a computer tutor (called AutoTutor) that simulates the discourse patterns and pedagogical strategies of a typical human tutor. The discourse patterns and pedagogical strategies were based on a previous project that dissected 100 hours of naturalistic tutoring sessions. AutoTutor is currently targeted for college students in introductory computer literacy courses, who learn the fundamentals of hardware, operating systems, and the Internet. Instead of merely being an information delivery system, AutoTutor serves as a discourse prosthesis or collaborative scaffold that assists the student in actively constructing knowledge. Evaluations of AutoTutor-1 have shown that the tutoring system improves learning and memory of the lessons by .5 standard deviation units, compared with a control condition in which college students reread chapters. This paper contrasts the teaching tactics of AutoTutor-1 and AutoTutor-2. In AutoTutor-1, a piece of information is considered covered if it mentioned by either the computer tutor or the student in a shared discourse space. In AutoTutor-2, a piece of information is considered covered only if it is articulated by the student.
Students ordinarily use both domain-dependent (such as seeking specific domain help) and domain-independent (such as trying to relax) strategies in regulating their affective states in learning. By contrast, current affective tutoring systems concentrate on the deployment of domain-dependent strategies only in regulating students' emotional states in learning. This paper reports the results of an experimental study that evaluated students' performance using the proposed framework. In a between-subjects experiment, students used two versions of a system for teaching data structures. The systems differed only in that one supported domain independent strategies, while both system systems supported domain-dependent strategies. Results provide some evidence that the experimental groups (presented with both the domain-dependent and domain-independent strategies) performed better than the control group (presented with only domain-dependent strategy) in learning data structures.
Affect has been the subject of increasing attention in cognitive accounts of learning. Many intelligent tutoring systems now seek to adapt pedagogy to student affective and motivational processes in an effort to increase the effectiveness of tutorial interaction and improve learning outcomes. To this end, recent work has begun to investigate the emotions experienced during learning in a variety of environments. In this paper we extend this line of research by investigating variances in transitions based upon individual characteristics. The findings reveal how affective trajectories vary among students and how these characteristics impact learning.
Affect has been the subject of increasing attention in cognitive accounts of learning. Many intelligent tutoring systems now
seek to adapt pedagogy to student affective and motivational processes in an effort to increase the effectiveness of tutorial
interaction and improve learning outcomes. To this end, recent work has begun to investigate the emotions experienced during
learning in a variety of environments. In this paper we extend this line of research by investigating the affective transitions
that occur throughout narrative-centered learning experiences. Further analysis differentiates the likelihood of affective
transitions stemming from pedagogical agent empathetic responses to student affect.
Building computerized mechanisms that will accurately, immediately and continually recognize a learner’s affective state and activate an appropriate response based on integrated pedagogical models is becoming one of the main aims of artificial intelligence in education. The goal of this paper is to demonstrate how the various kinds of evidence could be combined so as to optimize inferences about affective states during an online self-assessment test. A formula-based method has been developed for the prediction of students’ mood, and it was tested using data emanated from experiments made with 153 high school students from three different regions of a European country. The same set of data is analyzed developing a neural network method. Furthermore, the formula-based method is used as an input parameter selection module for the neural network method. The results vindicate to a great degree the formula-based method’s assumptions about student’s mood and indicate that neural networks and conventional algorithmic methods should not be in competition but complement each other for the development of affect recognition systems. Moreover, it becomes apparent that neural networks can provide an alternative for and improvements over tutoring systems’ affect recognition methods.
Our aim is to create an affective embodied conversational agent (ECA); that is an ECA able to display communicative and emotional signals. Nonverbal communication is done through certain facial expressions, gesture shapes, gaze direction, etc. But it can also carry a qualitative aspect through behavior expressivity: how a facial expression, a gesture is executed. In this paper we describe some of the work we have conducted on behavior expressivity, more particularly on gesture expressivity. We have developed a model of behavior expressivity using a set of six parameters that act as modulation of behavior animation. Expressivity may act at different levels of the behavior: on a particular phase of the behavior, on the whole behavior and on a sequence of behaviors. When applied at these different levels, expressivity may convey different functions.
We analyze the antecedents of affective states in a simulation problem-solving environment, The Incredible Machine: Even More Contraptions, through quantitative field observatio ns of high school students in the Philippines using that system. We investigate t he transitions between affective states over time, finding that several af fective states, including flow, boredom, and frustration, but not surprise, tend to persist over for relatively long periods of time. We also investigate how stude nts' usage choices influence their later affect, finding that gaming the system leads to reduced confusion but increased boredom.
Recent research has shown that empathic virtual agents enable to improve human-machine interaction. Virtual agent’s expressions
of empathy are generally fixed intuitively and are not evaluated. In this paper, we propose a novel approach for the expressions
of empathy using complex facial expressions like superposition and masking. An evaluation study have been conducted in order
to identify the most appropriate way to express empathy. According to the evaluation results people find more suitable facial
expressions that contain elements of emotion of empathy. In particular, complex facial expressions seem to be a good approach
to express empathy.
Humans continuously assess one another's situational context, modify their own affective state, and then respond based on these outcomes through empathetic expression. Virtual agents should be capable of similarly empathizing with users in interactive environments. A key challenge posed by empathetic reasoning in virtual agents is determining whether to respond with parallel or reactive empathy. Parallel empathy refers to mere replication of another's affective state, whereas reactive empathy exhibits greater cognitive awareness and may lead to incongruent emotional responses (i.e., emotions different from the recipient's and perhaps intended to alter negative affect). Because empathy is not yet sufficiently well understood, it is unclear as to which type of empathy is most effective and under what circumstances they should be applied. Devising empirically informed models of empathy from observations of "empathy in action" may lead to virtual agents that can accurately respond in social situations. This paper proposes a unified inductive framework for modeling parallel and reactive empathy. First, in training sessions, a trainer guides a virtual agent through a series of problem-solving tasks in a learning environment and encounters empathetic characters. The proposed inductive architecture tracks situational data including actions, visited locations, intentions, and the trainer's physiological responses to generate models of empathy. Empathy models are used to drive runtime situation-appropriate empathetic behaviors by selecting suitable parallel or reactive empathetic expressions. An empirical evaluation of the approach in an interactive learning environment suggests that the induced empathy models can accurately assess social contexts and generate appropriate empathetic responses for virtual agent control.
Recent research has shown that virtual agents expressing empathic emotions toward users have the potentiality to en- hance human-machine interaction. To identify under which circumstances a virtual agent should express empathic emo- tions, we have analyzed real human-machine dialog situa- tions that have led users to express emotion. The results of this empirical study have been combined with theoretical descriptions of emotions to construct a model of empathic emotions. Based on this model, a module of emotions has been implemented as a plug-in for JSA agents. It determines the empathic emotions (their types and intensity) of such agents in real time. It has been used to develop a demon- strator where users can interact with an empathic dialog agent to obtain information on their emails. An evaluation of this agent has enabled us to both validate the proposed model of empathic emotions and highlight the positive user's perception of the virtual agent.
Automatic facial expression recognition is a research topic with interesting applications in the field of human-computer interaction, psychology and product marketing. The classification accuracy for an automatic system which uses static images as input is however largely limited by the image quality, lighting conditions and the orientation of the depicted face. These problems can be partially overcome by using a holistic model based approach called the Active Appearance Model. A system will be described that can classify expressions from one of the emotional categories joy, anger, sadness, surprise, fear and disgust with remarkable accuracy. It is also able to detect smaller, local facial features based on minimal muscular movements described by the Facial Action Coding System (FACS). Finally, we show how the system can be used for expression analysis and synthesis.
Theories and technologies are needed to understand and integrate the knowledge of student affect (e.g., frustration, motivation and self-confidence) into learning models. Our goals are to redress the cognitive versus affective imbalance in teaching systems, develop tools that model student affect and build tutors that elicit, measure and respond to student affect. This article describes our broad approach towards this goal and our three main objectives: develop tools for affect recognition, interventions in response to student affect, and emotionally animated agents.
This survey describes recent progress in the field of Affective Computing (AC), with a focus on affect detection. Although many AC researchers have traditionally attempted to remain agnostic to the different emotion theories proposed by psychologists, the affective technologies being developed are rife with theoretical assumptions that impact their effectiveness. Hence, an informed and integrated examination of emotion theories from multiple areas will need to become part of computing practice if truly effective real-world systems are to be achieved. This survey discusses theoretical perspectives that view emotions as expressions, embodiments, outcomes of cognitive appraisal, social constructs, products of neural circuitry, and psychological interpretations of basic feelings. It provides meta-analyses on existing reviews of affect detection systems that focus on traditional affect detection modalities like physiology, face, and voice, and also reviews emerging research on more novel channels such as text, body language, and complex multimodal systems. This survey explicitly explores the multidisciplinary foundation that underlies all AC applications by describing how AC researchers have incorporated psychological theories of emotion and how these theories affect research questions, methods, results, and their interpretations. In this way, models and methods can be compared, and emerging insights from various disciplines can be more expertly integrated.
& In this paper, we report on our efforts in developing affective character-based interfaces, i.e., interfaces that recognize and measure affective information of the user and address user affect by employing embodied characters. In particular, we describe the Empathic Companion, an ani- mated interface agent that accompanies the user in the setting of a virtual job interview. This inter- face application takes physiological data (skin conductance and electromyography) of a user in realtime, interprets them as emotions, and addresses the user's affective states in the form of empathic feedback. The Empathic Companion is conceived as an educational agent that supports job seekers preparing for a job interview. We also present results from an exploratory study that aims to evaluate the impact of the Empathic Companion by measuring users' skin conductance and heart rate. While an overall positive effect of the Empathic Companion could not be shown, the outcome of the experiment suggests that empathic feedback has a positive effect on the interviewee's stress level while hearing the interviewer question. The idea of a computer sensing the user's autonomic nervous system (ANS) activity is becoming increasingly popular in the human-computer interface community, partly because of the availability of affordable high-specifi- cation sensing technologies, and also due to the recent progress in inter- preting users' physiological states as affective states or emotions (Picard 1997). The general vision is that if a user's emotion could be recognized by a computer, human-computer interaction would become more natural, enjoyable, and productive. The computer could offer help and assistance to a confused user or try to cheer up a frustrated user, and hence react in ways that are more appropriate than simply ignoring the user's affective state as is the case with most current interfaces.
Many of the intelligent tutoring systems that have been developed during the last 20 years have proven to be quite successful, particularly in the domains of mathematics, science, and technology. They produce significant learning gains beyond classroom environments. They are capable of engaging most students' attention and interest for hours. We have been working on a new generation of intelligent tutoring systems that hold mixed-initiative conversational dialogues with the learner. The tutoring systems present challenging problems and questions to the learner, the learner types in answers in English, and there is a lengthy multiturn dialogue as complete solutions or answers evolve. This article presents the tutoring systems that we have been developing. AUTO TUTOR is a conversational agent, with a taking head, that helps college students learn about computer literacy. ANDES, ATLAS, and WHY2 help adults learn about physics. Instead of being mere information-delivery systems, our systems help students actively construct knowledge through conversations.
Can human beings relate to computer or television programs in the same way they relate to other human beings? Based on numerous psychological studies, this book concludes that people not only can but do treat computers, televisions, and new media as real people and places. Studies demonstrate that people are "polite" to computers; that they treat computers with female voices differently than "male" ones; that large faces on a screen can invade our personal space; and that on-screen and real-life motion can provoke the same physical responses. Using everyday language to engage readers interested in psychology, communication, and computer technology, Reeves and Nass detail how this knowledge can help in designing a wide range of media.
Lifelike animated agents for knowledge-based learning environments can provide timely, customized advice to support learnersâ problem-solving activities. By drawing on a rich repertoire of emotive behaviors to exhibit contextually appropriate facial expressions and emotive gestures, these agents could exploit the visual channel to more effectively communicate with learners. To address these issues, this article proposes the emotive-kinesthetic behavior sequencing framework for dynamically sequencing lifelike pedagogical agentsâ full-body emotive expression. By exploiting a rich behavior space populated with emotive behaviors and structured by pedagogical speech act categories, a behavior sequencing engine operates in realtime to select and assemble contextually appropriate expressive behaviors. This framework has been implemented in a lifelike pedagogical agent, Cosmo, who exhibits full-body emotive behaviors in response to learners' problem-solving activities. (http://aied.inf.ed.ac.uk/members99/archive/vol_10/lester/full.html)
More than 30 years after its initial publication, this new edition of The Cognitive Structure of Emotions refines and updates Ortony, Clore, and Collins's OCC model of emotions. Starting from a three-way classification of construals of the world––events, the attribution of responsibility for events, and objects––the authors propose a systematic account of emotion differentiation. Rejecting the oft-favored features of bodily feelings, emotion-related behaviors, and facial expressions as too intensity-dependent and insufficiently diagnostic, they provide a detailed analysis of emotion differentiation in terms of the cognitive underpinnings of emotion types. Using numerous examples, they explain how different variables influence emotion intensity, and show how emotions can be formalized for computational purposes. Now with a contributed chapter describing the OCC model's influence, this book will interest a wide audience in cognitive, clinical, and social psychology, as well as in artificial intelligence and affective computing, and other cognitive science disciplines.
1. Introduction The study of emotion Types of evidence for theories of emotion Some goals for a cognitive theory of emotion 2. Structure of the theory The organisation of emotion types Basic emotions Some implications of the emotions-as-valenced-reactions claim 3. The cognitive psychology of appraisal The appraisal structure Central intensity variables 4. The intensity of emotions Global variables Local variables Variable-values, variable-weights, and emotion thresholds 5. Reactions to events: I. The well-being emotions Loss emotions and fine-grained analyses The fortunes-of-others emotions Self-pity and related states 6. Reactions to events: II. The prospect-based emotions Shock and pleasant surprise Some interrelationships between prospect-based emotions Suspense, resignation, hopelessness, and other related states 7. Reactions to agents The attribution emotions Gratitude, anger, and some other compound emotions 8. Reactions to objects The attraction emotions Fine-grained analyses and emotion sequences 9. The boundaries of the theory Emotion words and cross-cultural issues Emotion experiences and unconscious emotions Coping and the function of emotions Computational tractability.
Abstract This paper describes our FaceReader system, which we presented at Measuring ,Behavior 2005. This system is able to describe facial expressions and other facial features online with a remarkable accuracy. In this paper we ,will describe the possibilities of sucha s ystemand,will discuss the technology used to make the system work. Currently, emotional expressions can be recognized with anaccur acy of89% and it can also classify a number ,of other facial features. Keywords Facial expressions, classification, model-based, Active AppearanceM odel,re al-time.
The Tutoring Research Group at the University of Memphis has developed a computer tutor (called AutoTutor) that simulates the discourse patterns and pedagogical strategies of a typical human tutor. The dialog tactics were based on a previous project that dissected 100 hours of naturalistic tutoring sessions. AutoTutor is currently targeted for college students in introductory computer literacy courses, who learn the fundamentals of hardware, operating systems, and the Internet. Instead of merely being an information delivery system, AutoTutor serves as a discourse prosthesis (or collaborative scaffold) that assists the student in actively constructing knowledge. A dialog manager coordinates the conversation that occurs between a learner and a pedagogical agent, whereas lesson content and world knowledge are represented in a curriculum script and latent semantic analysis. The agent is a talking head with discourse- sensitive facial expressions and synthesized speech. Evaluations of AutoTutor have shown that the tutoring system improves learning and memory of the lessons by .5 to .6 standard deviation units. This article describes the components of AutoTutor and contrasts two versions that follow somewhat different teaching tactics.
[This book examines] empathy from the standpoint of contemporary social/personality psychology—emphasizing these disciplines' traditional subject matter (e.g., emotion, cognition, helping, aggression) and its research techniques (survey research, laboratory experiments).
[The author's] goal was to provide a thorough, readable . . . summary of contemporary empathy research [primarily for advanced undergraduate and graduate students]. (PsycINFO Database Record (c) 2012 APA, all rights reserved)
Following Langer (1992), this article reviews a series of experimental studies that demonstrate that individuals mindlessly apply social rules and expectations to computers. The first set of studies illustrates how individuals overuse human social categories, applying gender stereotypes to computers and ethnically identifying with computer agents. The second set demonstrates thatpeople exhibit overlearned social behaviors such as politeness and reciprocity toward computers. In the third set of studies, premature cognitive commitments are demonstrated: A specialist television set is perceived as providing better content than a generalist television set. A final series of studies demonstrates the depth of social responses with respect to computer ‘personality.’ Alternative explanations for these findings, such asanthropomorphism and intentional social responses, cannot explain the results. We conclude with an agenda for future research.
Over the last years, the animation of interface agents has been the target of increasing interest. Largely, this increase in attention is fuelled by speculated effects on human motivation and cognition. However, empirical investigations on the effect of animated agents are still small in number and differ with regard to the measured effects. Our aim is two-fold. First, we provide a comprehensive and systematic overview of the empirical studies conducted so far in order to investigate effects of animated agents on the user's experience, behaviour and performance. Second, by discussing both implications and limitations of the existing studies, we identify some general requirements and suggestions for future studies.
Embodied computer agents are becoming an increasingly popular human–computer interaction technique. Often, these agents are programmed with the capacity for emotional expression. This paper investigates the psychological effects of emotion in agents upon users. In particular, two types of emotion were evaluated: self-oriented emotion and other-oriented, empathic emotion. In a 2 (self-oriented emotion: absent vs. present) by 2 (empathic emotion: absent vs. present) by 2 (gender dyad: male vs. female) between-subjects experiment (N=96), empathic emotion was found to lead to more positive ratings of the agent by users, including greater likeability and trustworthiness, as well as greater perceived caring and felt support. No such effect was found for the presence of self-oriented emotion. Implications for the design of embodied computer agents are discussed and directions for future research suggested.
There is now growing interest in the development of computer systems which respond to users' emotion and affect. We report three small-scale studies (with a total of 42 participants), which investigate the extent to which affective agents, using strategies derived from human–human interaction, can reduce user frustration within human–computer interaction. The results confirm the previous findings of Klein et al. [Klein, J., Moon, Y., Picard, R.W., 2002. This computer responds to user frustration: theory, design and results. Interacting with Computers 142, 119–140] that such interventions can be effective. We also obtained results that suggest that embodied agents can be more effective at reducing frustration than non-embodied agents, and that female embodied agents may be more effective than male embodied agents. These results are discussed in light of the existing research literature.
The use of embodied agents, defined as visual human-like representations accompanying a computer interface, is becoming prevalent in applications ranging from educational software to advertisements. In the current work, we assimilate previous empirical studies which compare interfaces with visually embodied agents to interfaces without agents, both using an informal, descriptive technique based on experimental results (46 studies) as well as a formal statistical meta-analysis (25 studies). Results revealed significantly larger effect sizes when analyzing subjective responses (i.e., questionnaire ratings, interviews) than when analyzing behavioral responses such as task performance and memory. Furthermore, the effects of adding an agent to an interface are larger than the effects of animating an agent to behave more realistically. However, the overall effect sizes were quite small (e.g., across studies, adding a face to an interface only explains approximately 2.5% of the variance in results). We discuss the implications for both designers building interfaces as well as social scientists designing experiments to evaluate those interfaces. Author Keywords
Recently, more and more attention has been paid to emotions in the domain of Human-Computer Interaction. When evaluating a product, one can no longer ignore the emotions a product induces. This paper examines the value of a new instrument to measure emotions: the FaceReader. We will assess the extent to which the FaceReader is useful when conducting usability evaluations. To do this, we will compare the data gained from the FaceReader with two other sources: user questionnaires and researcher's loggings. Preliminary analysis shows that the FaceReader is an effective tool to measure instant emotions and fun of use. However, a combination of the FaceReader with another observation method (e.g. researcher's loggings) is necessary. As regards the user questionnaire, our results indicate that it is rather a reflection of the content of the application or the outcome of a task, than a correct self- reflection of how the user felt when accomplishing the task. Author Keywords Emotions, usability, FaceReader, instant fun of use
AutoTutor is a computer tutor that simulates the discourse patterns and pedagogical strategies of a typical human tutor. AutoTutor is designed to assist college students in learning the fundamentals of hardware, operating systems, and the Internet in an introductory computer literacy course. Most tutors in school systems are not highly trained in tutoring techniques and have only a modest expertise on the tutoring topic, but they are surprisingly effective in producing learning gains in students. We have dissected the discourse and pedagogical strategies these unskilled tutors exhibit by analyzing approximately 100 hours of naturalistic tutoring sessions. These mechanisms are implemented in AutoTutor. AutoTutor presents questions and problems from a curriculum script, attempts to comprehend learner contributions that are entered by keyboard, formulates dialog moves that are sensitive to the learner’s contributions (such as short feedback, pumps, prompts, elaborations, corrections, and hints), and delivers the dialog moves with a talking head. AutoTutor has seven modules: a curriculum script, language extraction, speech act classification, latent semantic analysis, topic selection, dialog move generation, and a talking head.
We pres ent a probabilistic model to monitor a user's emotions and engagement during the interaction with educational games. We illustrate how our probabilistic model assesses affect by integrating evidence on both possible causes of the user's emotional arousal (i.e., the state of the interaction) and its effects (i.e., bodily expressions that are known to be influenced by emotional reactions). The probabilistic model relies on a Dynamic Decision Network to leverage any indirect evidence on the user's emotional state, in order to estimate this state and any other related variable in the model. This is crucial in a modeling task in which the available evidence usually varies with the user and with each particular interaction. The probabilistic model we present is to be used by decision theoretic pedagogical agents to generate interventions aimed at achieving the best tradeoff between a user's learning and engagement during the interaction with educational games.
We present a probabilistic model of user affect designed to allow an intelligent agent to recognise multiple user emotions during the interaction with an educational computer game. Our model is based on a probabilistic framework that deals with the high level of uncertainty involved in recognizing a variety of user emotions by combining in a Dynamic Bayesian Network information on both the causes and effects of emotional reactions. The part of the framework that reasons from causes to emotions (diagnostic model) implements a theoretical model of affect, the OCC model, which accounts for how emotions are caused by one's appraisal of the current context in terms of one's goals and preferences. The advantage of using the OCC model is that it provides an affective agent with explicit information not only on which emotions a user feels but also why, thus increasing the agent's capability to effectively respond to the users' emotions. The challenge is that building the model requires having mechanisms to assess user goals and how the environment fits them, a form of plan recognition. In this paper, we illustrate how we built the predictive part of the affective model by combining general theories with empirical studies to adapt the theories to our target application domain. We then present results on the model's accuracy, showing that the model achieves good accuracy on several of the target emotions. We also discuss the model's limitations, to open the ground for the next stage of the work, i.e., complementing the model with diagnostic information.
We study the incidence (rate of occurrence), persistence (rate of reoccurrence immediately after occurrence), and impact (effect on behavior) of students’ cognitive–affective states during their use of three different computer-based learning environments. Students’ cognitive–affective states are studied using different populations (Philippines, USA), different methods (quantitative field observation, self-report), and different types of learning environments (dialogue tutor, problem-solving game, and problem-solving-based Intelligent Tutoring System). By varying the studies along these multiple factors, we can have greater confidence that findings which generalize across studies are robust. The incidence, persistence, and impact of boredom, frustration, confusion, engaged concentration, delight, and surprise were compared. We found that boredom was very persistent across learning environments and was associated with poorer learning and problem behaviors, such as gaming the system. Despite prior hypothesis to the contrary, frustration was less persistent, less associated with poorer learning, and did not appear to be an antecedent to gaming the system. Confusion and engaged concentration were the most common states within all three learning environments. Experiences of delight and surprise were rare. These findings suggest that significant effort should be put into detecting and responding to boredom and confusion, with a particular emphasis on developing pedagogical interventions to disrupt the “vicious cycles” which occur when a student becomes bored and remains bored for long periods of time.
The realism of avatars in terms of behavior and form is critical to the development of collaborative virtual environments. In the study we utilized state of the art, real- time face tracking technology to track and render facial expressions unobtrusively in a desktop CVE. Participants in dyads interacted with each other via either a video- conference (high behavioral realism and high form realism), voice only (low behav- ioral realism and low form realism), or an "emotibox" that rendered the dimensions of facial expressions abstractly in terms of color, shape, and orientation on a rectan- gular polygon (high behavioral realism and low form realism). Verbal and non-verbal self-disclosure were lowest in the videoconference condition while self-reported copresence and success of transmission and identification of emotions were lowest in the emotibox condition. Previous work demonstrates that avatar realism in- creases copresence while decreasing self-disclosure. We discuss the possibility of a hybrid realism solution that maintains high copresence without lowering self-disclosure, and the benefits of such an avatar on applications such as dis- tance learning and therapy.