Figure 3 - uploaded by Minjuan Wang
Content may be subject to copyright.
Source publication
Using emotion detection technologies from biophysical signals, this study explored how emotion evolves during learning process and how emotion feedback could be used to improve learning experiences. This article also described a cutting-edge pervasive e-Learning platform used in a Shanghai online college and proposed an affective e-Learning model,...
Context in source publication
Context 1
... The core of the platform includes a number of "smart classrooms" distributed around Shanghai, the Yangtze River delta, and even in remote western regions of China such as Tibet, Yan’an, Xing Jiang, and Nin Xia. These classrooms are equipped with smart devices/sensors and specially developed software for distance learning. For example, the touch screen of the room displays presentations (e.g. PowerPoint), while also acts as a whiteboard for handwriting. The instructor can write on materials projected on the screen using a laser E-Pen. To optimize the video quality, a pan-camera can follow the instructor when he/she moves around in the classroom. RFID (Radio Frequency IDentifier) tags are used to identify and track students. Another tracking camera is mounted in the front of the classroom and it captures students’ attention status by recognizing the ‘blink frequency’ of their eyes. During the class session, instructors can load their pre-prepared PowerPoint and Word documents and write on the whiteboard (even when they are away from the whiteboard). The students can also write individual notes on the instructors’ handwriting window. All these classroom activities are being transmitted through various technologies to students at a distance. They are also recorded and archived for later review. Using this hi-tech environment, the teacher can move freely in the room, use his body language to communicate, and also interact with learners naturally and easily as in a traditional face-to-face classroom. This online college has about 17,000 Students, and 99% of them are working professionals who attend school part time. Therefore, their academic backgrounds, knowledge, and skills vary a great deal. Given such diversity, it is important to provide personalized learning services. The Shanghai system has harnessed data-mining technologies to organize learning communities and provide learning content recommendation based on student profiles (L. P. Shen & Shen, 2005; R. M. Shen, Yang, & Han, 2003). The large number of students in this college and its expansive course delivery systems make it an ideal place to test new and emerging technologies. The goal of this study is to understand how learners’ emotions evolve during learning process, so as to develop learning systems that recognize and respond appropriately to their emotional change. This paper also proposes a prototype of an affective e-Learning model that combines learner’s emotions with the Shanghai e-Learning platform. As Picard (2004) stated, theories of emotional impact on learning need to be tested and further developed. Until today, there is still a lack of comprehensive and empirically validated theories about emotion and learning. In this experiment, we examined several of the existing emotion theories in learning to help construct our affective e- Learning model. We used Russell’s ‘circumplex model’ to describe user’s emotion space. We then used the Kort’s ‘learning spiral model’ as the starting point to explore the affective evolution during learning. Following is the description of these models and our proposal for a prototype of an affective e-Learning model. In our search for an emotion theory we focused on dimensional models instead of cognitive appraisal model for user emotion modeling because they cover the feeling of emotional experience both on a low level and a higher, cognitive level. One well-established dimensional model is psychologist Russell’s circumplex model of affect (Russell, 1980) where emotions are seen as combinations of arousal and valence. In Russell’s model, emotions are distributed in a system of coordinates where the y-axis indicates the degree of arousal and the x-axis measures the valence, from negative to positive emotions. The Russell’s model is widely used in recent researches (Craig et al., 2004; Fagerberg et al., 2004; Kort et al., 2001; Leon et al., 2007; Picard et al., 2001). And most of these just explored from three to eight basic emotions. Though Kort et al. (2001) proposed five sets of about thirty emotions that may be relevant to learning, however, we believe that skilled human tutors and teachers react to assist students based on a few ‘least common set’ of affect as opposed to a large number of complex factors; thus, we carefully select a basic learning emotion set which we deem most important for shaping our affective learning model. The basic set includes the most important and frequently occurred emotions during learning, namely, interest, engagement, confusion, frustration, boredom, hopefulness, satisfaction and disappointment. They might not be placed exactly the same for all people when put in the Russell’s two-dimension emotion space, because this model focuses on subjective experiences. Figure 2 is an example of two-dimensional basic learning emotion space. We anticipate revising this emotion set when the study progresses. Kort, Reilly and Picard (2001) proposed a four quadrant learning spiral model (Figure 3) in which emotions change while the learner moves through quadrants and up the spiral. In quadrant I the learner is experiencing positive affect and constructing knowledge. At this point, the learner is working through the material with ease and has not experienced anything overly puzzling. Once discrepancies start to arise between the information and the learner’s knowledge structure, he/she moves to quadrant II, which consists of constructive learning and negative affect. Here the learner experiences affective states such as confusion. As the learner tries to sort out the puzzle but fails, he might move into quadrant III. This is the quadrant of unlearning and negative affect, when the learner experiences emotions such as frustration. After the misconceptions are discarded, the learner moves into quadrant IV, marked by ‘unlearning’ and positive affect”. While in this quadrant the learner is still not sure exactly how to move forward. However, he/she does acquire new insights and search for new ideas. Once the learner develops new ideas, he/she is propelled back into quadrant I; thus, concluding one cycle around the learning spiral of Kort’s model. As the learner move up the spiral, cycle after cycle, he/she become more competent and acquire more domain knowledge. Kort, Reilly and Picard (2001) also described the empirical research methods to validate this spiral model, and promised a future paper to report the results of the empirical research. There were some efforts of empirical research on this model, e.g. Kort & Reilly (2002) stated that “we are in the process of performing empirical research on this model”, and “ideally, the Learning Companion should observe and try to understand the processes a learner experiences during all of these quadrants; however, this is currently beyond the capabilities of the technology”(Kapoor, Mota, & Picard, 2001) , and “The process of cycling through these emotions during learning is currently being investigated in the present project”(D'Mello et al., 2005). But to our best knowledge, there have been no empirical validation report about this model. We used this ideal learning spiral model as the starting point to explore learner’s emotional evolution during the learning process. For our study, we used Russell’s ‘circumplex model’ to describe learner’s emotions detected from biophysical signals, and used the Kort’s ‘learning spiral model’ as the starting point to explore learners’ emotional evolution during the learning process. Finally, based on our previous work (L. P. Shen, Leon, Callaghan, & Shen, 2007; L. P. Shen & Shen, 2005), we proposed a model of affective learning (Figure 4) focusing on how we could make use of the information when we have got the learner’s emotion states and their evolution. This affective learning model considers the contextual information of the learner and the learning setting, and generates appropriate responses to the learner, based on his/her emotional states, cognitive abilities, and learning goals. This model can also be used to customize the interaction between the learner and learning system, to predict learner responses to system behavior, and to predict learner's future interaction with the learning system. The affective learning model used a combination of (a) a cognitive appraisal approach to affective user modeling (which inferences emotions according to situations experienced by the user as well as the observable behavior of the user) and (b) a physiological approach. Figure 4 shows a high-level description of this affective learning model, which only displays the general factors involved. The upper part of the model was modified OCC (Ortony et al., 1990) cognitive appraisal model for emotions, and the lower part of the model was the physiology recognition of emotions, where these two models converge One of the pattern recognition method-Bayesian Networks (Duda, Hart, & Stork, 2000) were employed to model the relations between the emotional states and their causal variables. This affective model indicated that the user’s emotional states were related to learner profiles (e.g., learning preferences, cognitive skills, knowledge structure), goals of learning, and learner interaction with the learning system. The user’s emotional states would in turn influence the measurements of the available sensors. The advantage of having a model based on a Bayesian Network was that it could leverage any evidence available on the variables related to emotional states to make predictions for any other variable in the model. For instance, the user’s emotional state could be assessed using existing information on learner profile and the learning context, even in absence of reliable sensors. Or, we could assess both emotional state and learner profile from reliable sensors, learner behavior and system reactions. The emotion detection system helped the system to provide timely help or adequate content based on the emotions the learner ...
Similar publications
The concept map (CM) has been adopted to improve teaching effects because it can effectively reduce the bad effects of e-learning, such as disorientation and cognition load, and keep aware of their knowledge learning path. But adaptive learning systems (ALS) which integrated the CM perform well in extracting and managing the tacit knowledge in a st...
Citations
... An accurate understanding of human emotions is beneficial for several applications, such as multimedia analysis, digital entertainment, health monitoring, human-computer interaction, etc (Shen et al., 2009;Beale & Peter, 2008;Qian et al., 2019;D'Mello & Kory, 2015). Compared with traditional emotion recognition, which only uses a unimodal data source, multimodal emotion recognition that exploits and explores different data sources, such as visual, audio, and text, has Communicated by Vittorio Murino. ...
Multimodal emotion recognition identifies human emotions from various data modalities like video, text, and audio. However, we found that this task can be easily affected by noisy information that does not contain useful semantics and may occur at different locations of a multimodal input sequence. To this end, we present a novel paradigm that attempts to extract noise-resistant features in its pipeline and introduces a noise-aware learning scheme to effectively improve the robustness of multimodal emotion understanding against noisy information. Our new pipeline, namely Noise-Resistant Multimodal Transformer (NORM-TR), mainly introduces a Noise-Resistant Generic Feature (NRGF) extractor and a multimodal fusion Transformer for the multimodal emotion recognition task. In particular, we make the NRGF extractor learn to provide a generic and disturbance-insensitive representation so that consistent and meaningful semantics can be obtained. Furthermore, we apply a multimodal fusion Transformer to incorporate Multimodal Features (MFs) of multimodal inputs (serving as the key and value) based on their relations to the NRGF (serving as the query). Therefore, the possible insensitive but useful information of NRGF could be complemented by MFs that contain more details, achieving more accurate emotion understanding while maintaining robustness against noises. To train the NORM-TR properly, our proposed noise-aware learning scheme complements normal emotion recognition losses by enhancing the learning against noises. Our learning scheme explicitly adds noises to either all the modalities or a specific modality at random locations of a multimodal input sequence. We correspondingly introduce two adversarial losses to encourage the NRGF extractor to learn to extract the NRGFs invariant to the added noises, thus facilitating the NORM-TR to achieve more favorable multimodal emotion recognition performance. In practice, extensive experiments can demonstrate the effectiveness of the NORM-TR and the noise-aware learning scheme for dealing with both explicitly added noisy information and the normal multimodal sequence with implicit noises. On several popular multimodal datasets (e.g., MOSI, MOSEI, IEMOCAP, and RML), our NORM-TR achieves state-of-the-art performance and outperforms existing methods by a large margin, which demonstrates that the ability to resist noisy information in multimodal input is important for effective emotion recognition.
... They demonstrated that this process of getting emotional feedback was useful in improving the learning process. Further, it was also proved that emotional awareness strengthens the students' performance (Shen et al. 2009). ...
... For a socio-emotional approach to technology in the educational field, the proposal given by [27] is cited, an e-learning model (online teaching and learning through the Internet and technology) of emotional orientation based on the use of emotional states, contextual environments, learning environments, cognitive skills, and learning goals, so that from this information, we can generate appropriate responses for students, personalising the learning system to improve student interaction [12]. In this sequence, [28] states that "training devices, focused on providing personalised attention to individuals in training, require support from a theory that emerges from the characteristics of a personalised approach", which is different from the traditional approach, going more towards human emotion. ...
This article concerns the analysis and strengthening of children’s emotional self-regulation as a key process in the sustainable and comprehensive educational development of students from 6 to 8 years of age. The objective of the present study was to design a didactic proposal for technological mediation (WhatsApp) that contributes to emotional self-regulation and underpins the sustainable education of children in the context of the prevalence of COVID-19. The research design involved documentation, field, and propositional work. Regarding the documentation design, the content analysis technique of the Institutional Educational Project and the Coexistence Project of an official educational institution located in Bogotá, Colombia, were used. Regarding the field design, the survey technique was applied through a structured questionnaire for populations made up of second grade primary school students, parents, and teachers of the institution. Among the main results, it stands out that the prevalence of COVID-19 and its post-pandemic implications have generated greater use of available technologies, such as the WhatsApp application, evidencing a positive relationship between the level of emotional self-regulation of children and its use as a didactic mediation agent. These findings serve as input for the design of the interactive TICSR-WA proposal.
... Automated emotion recognition typically involves measuring various parameters of the human body or electrical impulses in the nervous system and analyzing their changes. Research on the utilization of emotion recognition technologies based on biophysical signals in the learning process is detailed in [17], while various methods of emotion recognition are presented in [18]. ...
The assessment of affective states in online learning often relies on various devices, such as sensors, which indicate physiological reactions of a person's emotional state. It is important to note that not every university can afford the sensors and devices required for this purpose. Furthermore, not every student may be willing to monitor their emotional state using sensors on a personal computer during online learning. This article focuses on the potential benefits of detecting affective states through self-reported survey responses. It aims to explore ways to improve student‘s learning by identifying and responding to specific emotional states during online learning. This paper presents the identification of emotions using a survey, administered to engineering students during an e-learning course in the adaptive
learning system (ALS). In addition, the authors developed a model of Bayesian Network (BN) to analyze the current emotional state of students in e-learning and propose supportive messages to students in order to improve their emotional state.
... Emotion recognition can be used to adjust learning material and to observe students so as to keep them engaged with the topic. Therefore, emotion recognition software can lead to better learning results [54]. This could also help teachers to adjust their teaching methods using deep emotional feedback that would not be available in the real world. ...
In this paper, we look at the latest developments in the field of online virtual realities, with particular focus on “metaverses”. Within this field, we aim to describe potential interactions with emotion recognition and manipulation technologies. The article points out how these technologies can generate new threats to human and individual rights. Moreover, it covers how already existing uses of these technologies can become increasingly worrying in the context of metaverses. Finally, we look at existing legal and technical protections; we focus on their success and their shortcomings and how they can be improved in order to adapt to these new technologies. Our research question was: how can emotion recognition and manipulation technology be used and/or abused in the context of a metaverse?
... These processes include information processing, communication processing, negotiation processing, decision-making processing, category sorting tasks, and creative problem-solving processes (Isen, 2015). Exemplifying the intricate connection between emotional experiences and cognitive functions, these findings emphasize the pivotal role that emotions play in the learning process, whether in face-to-face settings (Vogel & Schwabe, 2016) or online environments (Shen et al., 2009;Um et al., 2012). ...
Considering the need for pedagogically effective learning activities and materials to support language learning, particularly within teacher-led instruction, it is curious that at present there is no overarching, research-based framework available to educators to draw from when designing and implementing such activities and materials. To address this gap, the authors of this paper have drawn from a host of relevant research pertaining to cognitive neuroscience, educational psychology, and second language acquisition to establish a framework for designing and implementing activities and learning materials capable of facilitating enhanced language learning outcomes within an inclusive classroom. Incorporating ten key considerations – attention and focus, desirable difficulty, depth of processing, deliberate practice, novelty and surprise, wakeful rest, visible learning, meaningful feedback, affective engagement, and strategic choice and use – this versatile framework not only provides teachers with necessary knowledge for designing language learning activities and materials in an engaging and efficacious manner but may also embolden them to do so.
... For example, artificial intelligence, particularly deep learning, has been used to assist students in understanding and categorizing artworks, thereby enriching their educational experiences and enjoyment [18,43,107]. In the domain of online platforms, interactive engagement plays a crucial role in enhancing the appreciation experience [89]. By integrating elements such as virtual reality and conversation agents, online platforms transform passive viewers into active participants [69,85,103,109]. ...
Art appreciation serves as a crucial medium for emotional communication and sociocultural dialogue. In the digital era, fostering deep user engagement on online art appreciation platforms remains a challenge. Leveraging generative AI technologies, we present EyeSee, a system designed to engage users through anthropomorphic characters. We implemented and evaluated three modes (Narrator, Artist, and In-Situ) acting as a third-person narrator, a first-person creator, and first-person created objects, respectively, across two sessions: Narrative and Recommendation. We conducted a within-subject study with 24 participants. In the Narrative session, we found that the In-Situ and Artist modes had higher aesthetic appeal than the Narrator mode, although the Artist mode showed lower perceived usability. Additionally, from the Narrative to Recommendation session, we found that user-perceived relatability and believability within each interaction mode were sustained, but the user-perceived consistency and stereotypicality changed. Our findings suggest novel implications for applying anthropomorphic in-situ narratives to other educational settings.
... A prototype of an e-Learning model which includes the tracking of the affective states of the learners is described in [21]. The model considers the learning goals, the contextual information and cognitive abilities of the learners and provides personalized feedback according to the affective state of the learners. ...
... The learning content was recommended to the student both considering his emotional state and ignoring it. The results indicate that emotion-aware content recommendation requires fewer human interventions in the learning process, 11 interventions in the case of emotion-aware recommendation compared to 21 in the case of non-emotion-aware recommendation [21]. An intelligent tutoring system, called Affective AutoTutor, uses the affective and cognitive states of the students and responds accordingly. ...
... The model was developed in "Learning Companion" project, Affective Computing Group, MIT Media Lab [20] Experimental study carried out in a Shanghai online college: learners' emotions are recognized based on physiological signals. The results show the positive impact of AER on the learning process [21] Controlled experiments performed with Affective AutoTutor: the affect recognition is based on multimodal data (facial, text, body movements). Usage of Affective AutoTutor proves "dramatic improvements in the learning compared to the original AutoTutor system" [22] Deep learning model used to maintain the engagement of learners during learning sessions via technology. ...
In a technologically advanced world, artificial intelligence has impacted all fields of activity. The augmentation of online learning by means of emotion recognition systems raises new challenges in terms of obtaining high-performance systems and in interpreting the results. The paper aims to investigate the usage of automated emotion recognition in learning and to develop a deep learning model based on physiological data to recognize emotions often encountered in classrooms. So, an 1D-CNN model based on physiological data is used to recognize seven emotions: boredom, confusion, frustration, curiosity, excitement, concentration, and anxiety. These emotions are described according to the PAD model and the 5 EEG signals, FP1, AF3, F7, T7, FP2, are taken from the DEAP dataset to train and to test the convolutional neural network model. The high accuracy we obtained (i.e. boredom—99.64%, confusion—99.70%, frustration—99.66%, curiosity—99.80%, excitement—99.91%, concentration—99.70%, anxiety—99.21%) proves that the use of signals obtained via only five channels is sufficient to recognize the presence of emotions. Furthermore, an improved method of analysis based on LIME is proposed and used to obtain reliable explanations for the predictions of our model.
... In the study [11], Shen et al. used physiological data, including Heart Rate (HR), Skin Conductance Response (SCR), Blood Volume Pulse (BVP), and EEG sensors to identify emotions within the context of online learning. Their unique strategy was to use a Support Vector Machine (SVM) promising model specifically for the valence-arousal space. ...
... Commonly, sentiments surrounding education can range from positive to negative. In this context, the emotions affect the motivation and the outcome of the learning process (19)(20)(21). The role of Artificial Intelligence (AI) in sentiment analysis is crucial for aiding doi: 10.25100/iyc.v26i1.13759 ...
Student evaluation of teaching (SET) is an ad-hoc way of assessing teaching effectiveness in higher education institutions. In this paper, we present an approach to analyzing sentiments expressed in SET comments using a large language model (LLM). By employing natural language processing techniques, we extract and analyze sentiments expressed by students when the course has ended, aiming to provide educators and administrators with valuable insights into teaching quality and elements to improve teaching practice. Our study demonstrates the effectiveness of LLMs in sentiment analysis of comments, highlighting their potential to enhance the evaluation process. Our experiments with a crowdsourced tagged dataset show a 93% of accuracy in the classification of feedback messages. We discuss the implications of our findings for educational institutions and propose future directions for research in this domain.