Ming Li’s research while affiliated with Beihang University and other places

With the rapid development of deep learning, Electroencephalograph(EEG) emotion recognition has played a significant role in affective brain-computer interfaces. Many advanced emotion recognition models have achieved excellent results. However, current research is mostly conducted in laboratory settings for emotion induction, which lacks sufficient ecological validity and differs significantly from real-world scenarios. Moreover, emotion recognition models are typically trained and tested on datasets collected in laboratory environments, with little validation of their effectiveness in real-world situations. VR, providing a highly immersive and realistic experience, is an ideal tool for emotional research. In this paper, we collect EEG data from participants while they watched VR videos. We propose a purely Transformer-based method, EmoSTT. We use two separate Transformer modules to comprehensively model the temporal and spatial information of EEG signals. We validate the effectiveness of EmoSTT on a passive paradigm collected in a laboratory environment and an active paradigm emotion dataset collected in a VR environment. Compared with state-of-the-art methods, our method achieves robust emotion classification performance and can be well transferred between different emotion elicitation paradigms.

Working memory is crucial for higher cognitive functions in humans and is a focus in cognitive rehabilitation. Compared to conventional working memory training methods, VR-based training provides a more immersive experience with realistic scenarios, offering enhanced transferability to daily life. However, existing VR-based training methods often focus on basic cognitive tasks, underutilize VR’s realism, and rely heavily on subjective assessment methods. In this paper, we introduce a VR Sandbox for working memory training and evaluation, MEM-Box, which simulates everyday life scenarios and routines and adaptively adjusts task difficulty based on user performance. We conducted a training experiment utilizing the MEM-Box and compared it with a control group undergoing PC-based training. The results of the Stroop test indicate that both groups demonstrated improvements in working memory abilities, with MEM-Box training showing greater efficacy. Physiological data confirmed the effectiveness of the MEM-Box, as we observed lower HRV and SDNN. Furthermore, the results of the frequency-domain analysis indicate higher sympathetic nervous system activity (LFpower and LF/HF) during MEM-Box training, which is related to the higher sense of presence in VR. These metrics pave the way for building adaptive VR systems based on physiological data.

Cognitive control is often perplexing to elucidate and can be easily influenced by emotions. Understanding the individual cognitive control level is crucial for enhancing VR interaction and designing adaptive and self-correcting VR/AR applications. Emotions can reallocate processing resources and influence cognitive control performance. However, current research has primarily emphasized the impact of emotional valence on cognitive control tasks, neglecting emotional arousal. In this study, we comprehensively investigate the influence of emotions on cognitive control based on the arousal-valence model. A total of 26 participants are recruited, inducing emotions through VR videos with high ecological validity and then performing related cognitive control tasks. Leveraging physiological data including EEG, HRV, and EDA, we employ classification techniques such as SVM, KNN, and deep learning to categorize cognitive control levels. The experiment results demonstrate that high-arousal emotions significantly enhance users' cognitive control abilities. Utilizing complementary information among multi-modal physiological signal features, we achieve an accuracy of 84.52 distinguishing between high and low cognitive control. Additionally, time-frequency analysis results confirm the existence of neural patterns related to cognitive control, contributing to a better understanding of the neural mechanisms underlying cognitive control in VR. Our research indicates that physiological signals measured from both the central and autonomic nervous systems can be employed for cognitive control classification, paving the way for novel approaches to improve VR/AR interactions.

The ecological validity of emotion-inducing scenarios is essential for emotion research. In contrast to the classical passive induction paradigm, immersive VR fully engages the psychological and physiological components of the subject, which is considered an ecologically valid paradigm for studying emotion. Several studies investigate the emotional responses to different VR tasks or games using subjective scales. However, little research regards VR as an eliciting material, especially when systematically analyzing emotional processes in VR from a neurophysiological perspective. To fill this gap and scientifically evaluate VR's ability to be used as an active method for emotion elicitation, we investigate the dynamic relationship between explicit information (subjective evaluations) and implicit information (objective neurophysiological data). A total of 28 participants are enlisted to watch eight VR videos while their SAM/IPQ scores and EEG data are recorded simultaneously. In ecologically valid scenarios, the subjective results demonstrate that VR has significant advantages for evoking emotion in arousal-valence. This conclusion is backed by our examination of objective neurophysiological evidence that VR videos effectively induce high-arousal emotions. In addition, we obtain features of critical channels and frequency oscillations associated with emotional valence, thereby validating previous research in more lifelike circumstances. In particular, we discover hemispheric asymmetry in the occipital region under high and low emotional arousal, which adds to our understanding of neural features and the dynamics of emotional arousal. As a result, we successfully integrate EEG and VR to demonstrate that VR is more pragmatic for evoking natural feelings and is beneficial for emotional research. Our research has set a precedent for new methodologies of using VR induction paradigms to acquire a more reliable explanation of affective computing.

This paper assesses the impact of emotional arousal on learning through a virtual reality video of geography immersion learning. Validity was measured with traditional questionnaire data and using electroencephalography (EEG). Twenty-four human subjects were recruited and presented with eight geography immersive learning videos with different affective tendencies. EEG data of the subjects were collected while they were watching the videos. After the video watching, subjects were requested to fill in an emotion scale, a learning motivation scale, and a mind flow experience. The results of the study show that VR video learning materials can well induce the positive and negative emotions of the subjects. Compared with negative emotions, the alpha band power in the frontal lobe of the brain, beta and gamma bands in the temporal lobe region is significantly higher under positive emotions. In addition, the results of the subjective scales indicate that subjects have stronger intrinsic motivations and better flow experiences under positive emotions. However, there was no significant difference for the immersion under positive and negative emotions. Our findings demonstrate the usability of teaching virtual reality situational geography stories and the broad value of using this teaching method for future instruction.