Figure 3 - uploaded by Danny Plass-Oude Bos
Content may be subject to copyright.
Source publication
Making the computer more empathic to the user is one of the aspects of aective computing. With EEG-based emotion recognition, the computer can actually take a look inside the user's head to observe their mental state. This paper describes a research project conducted to rec- ognize emotion from brain signals measured with the BraIn- quiry EEG PET d...
Contexts in source publication
Context 1
... temporal lobe (the side areas covering T3-T6 in Fig- ure 3) is essential for hearing, language and emotion, and also plays an important role in memory. ...
Context 2
... is responsible for cognitive, emotional and motivational pro- cesses. The prefrontal lobe is part of the frontal cortex (top half in Figure 3), which is said to be the emotional control center and to even determine personality . It is involved in, among others, judgement and social behavior. ...
Context 3
... make this textual explanation a little less abstract, refer to Figure 3 for a visual representation. Please note that this is not a normal top view of the head, in which the positions on the circle would be shown on the outer border and the temporal lobe would not be visible. ...
Context 4
... relaxed a player is, is determined by the ratio of the beta and alpha brainwaves as recorded by the EEG. The EEG is measured by three electrodes mounted on the forehead (Figure 3: Fp1, Fp2, and Fpz -inbetween Fp1 and Fp2 on the vertical line). There is no mentioning of reference leads or whether the measurements are bipolar or monopolar [13]. ...
Context 5
... and F4 (Figure 3) are the most used positions for look- ing at this alpha activity, as they are located above the dorsolateral prefrontal cortex [communication with Martijn Arns] [20]. As mentioned in the previous section about emo- tion in the brain, the prefrontal lobe plays a crucial role in emotion regulation and conscious experience. ...
Citations
... In this review, we specifically focus on the analysis and learning methods for covariance-based neuroimaging data, which capture second-order statistics of neuroimaging signals, particularly covariance matrices derived from different neuroimaging modalities. For example, spatial covariance matrices in EEG are frequently used because they capture spatial relationships and synchronization between different regions of the brain, which are essential for tasks such as motor imagery and emotion classification [1], [2]. Functional connectivity matrices derived from fMRI data are used to investigate cognitive-related disorders such as Alzheimer's disease, schizophrenia, and multiple sclerosis [3], [4]. ...
... Research in psychophysiology shows a clear correlation between neural activity in the left and right frontal lobes and specific emotional states. Increased activity in the left frontal region is associated with positive emotions, while increased activity in the right frontal region correlates with negative emotional states [2]. A recent study used SPDNet to capture emotional representations in multiple contexts. ...
Neuroimaging provides a critical framework for characterizing brain activity by quantifying connectivity patterns and functional architecture across modalities. While modern machine learning has significantly advanced our understanding of neural processing mechanisms through these datasets, decoding task-specific signatures must contend with inherent neuroimaging constraints, for example, low signal-to-noise ratios in raw electrophysiological recordings, cross-session non-stationarity, and limited sample sizes. This review focuses on machine learning approaches for covariance-based neuroimaging data, where often symmetric positive definite (SPD) matrices under full-rank conditions encode inter-channel relationships. By equipping the space of SPD matrices with Riemannian metrics (e.g., affine-invariant or log-Euclidean), their space forms a Riemannian manifold enabling geometric analysis. We unify methodologies operating on this manifold under the SPD learning framework, which systematically leverages the SPD manifold's geometry to process covariance features, thereby advancing brain imaging analytics.
... To calculate the full width at half maximum (FWHM), we analyzed the signal contour in the 100-250 Hz range for higher frequencies and 10-35 Hz for lower frequencies. Double support time was determined by the time difference between peaks in the [100,200] Hz and [20,30] Hz bands. This is based on the observation that the higher frequency part ( [100,200] Hz) mainly represents heel strike (double support initiation), and the lower frequency part ( [20,30] Hz) represents toeoff (double support termination). ...
... Double support time was determined by the time difference between peaks in the [100,200] Hz and [20,30] Hz bands. This is based on the observation that the higher frequency part ( [100,200] Hz) mainly represents heel strike (double support initiation), and the lower frequency part ( [20,30] Hz) represents toeoff (double support termination). This is slightly narrower than in the previous calculation because when calculating the double support time, we want to reduce the noisy peaks and keep the main peak, so narrowing down the frequency band selection shows better performance. ...
Emotion recognition is critical for various applications such as early detection of mental health disorders and emotion based smart home systems. Previous studies used various sensing methods for emotion recognition, such as wearable sensors, cameras, and microphones. However, these methods have limitations in long term domestic, including intrusiveness and privacy concerns. To overcome these limitations, this paper introduces a nonintrusive and privacy friendly personalized emotion recognition system, EmotionVibe, which leverages footstep induced floor vibrations for emotion recognition. The main idea of EmotionVibe is that individuals' emotional states influence their gait patterns, subsequently affecting the floor vibrations induced by their footsteps. However, there are two main research challenges: 1) the complex and indirect relationship between human emotions and footstep induced floor vibrations and 2) the large between person variations within the relationship between emotions and gait patterns. To address these challenges, we first empirically characterize this complex relationship and develop an emotion sensitive feature set including gait related and vibration related features from footstep induced floor vibrations. Furthermore, we personalize the emotion recognition system for each user by calculating gait similarities between the target person (i.e., the person whose emotions we aim to recognize) and those in the training dataset and assigning greater weights to training people with similar gait patterns in the loss function. We evaluated our system in a real-world walking experiment with 20 participants, summing up to 37,001 footstep samples. EmotionVibe achieved the mean absolute error (MAE) of 1.11 and 1.07 for valence and arousal score estimations, respectively, reflecting 19.0% and 25.7% error reduction compared to the baseline method.
... His approach, which developed from the study of suggestion and its effect on human behavior, included using auditory and visual variables (such as baroque music, aural messages, and measured breathing in a peaceful environment). These background cognitive and emotional stimuli evoke a physical response that creates harmony between the various body systems: heart rate, breathing, and brain activity in alpha wave frequency [23]. ...
... Furthermore, researchers have recorded the way in which alpha wave activity enables accelerated learning [22]. It was further found that the use of alpha waves improves working memory; decreases the sense of cognitive overload [31,32]; enhances processing skills of semantic memory, and generates an overall state of relaxation and mental alertness [23]. ...
... Measured breathing has been found to induce relaxation by diverting attention from distractions while focusing on the breathing rhythm. Hence, our system sought to focus the learner's attention on measured breathing instead of the effort required to receive and process new words [23,33]. ...
There is a need to find innovative learning methods that enable accelerated learning of a foreign language. This study examined the effect of computer-assisted language learning (CALL) in acquiring a foreign language, which combines cognitive and emotional stimuli in the background. The study explored two factors related to the acquisition of a foreign language: the duration and scope of the learning process and the depth of internalization of the newly acquired language. Another objective was to assess the learning method in two learning environments, 2D and VR, to determine if the learning environment affects the learning results and leads to better vocabulary retention. One hundred native French speakers, with an average age of 47.5, participated in the study and had no prior knowledge of the newly acquired language. We randomly divided the participants into two groups (2D and VR). They studied 550 words in a new language for five days: 30 min each evening and 15 min in the morning. The post-learning test pointed out that both groups improved their vocabulary scores significantly. Approximately one month after the learning experience, we administered a knowledge retention test to 32 participants and found that the level of knowledge had been retained. Finally, background variables (e.g., gender, age, and previous knowledge of the newly acquired language) did not affect the learning results. The findings indicate that CALL, which integrates background cognitive and emotional stimuli in both learning environments, significantly accelerates learning pace, broadens the scope of newly acquired words, and ensures retention. The level of improvement observed in our study is notably higher than that reported in the literature for studies that had previously evaluated CALL and in-class language acquisition.
... In contrast, relatively greater activation of the left frontal area is associated with positive emotions [47]. Accordingly, Valence (Table 1, Formula 2) has been as well studied by subtracting the natural logarithm of the left hemisphere a power (aF3) from the natural logarithm of the right hemisphere a power (aF4) and comparing their difference [23,55,94,97]. ...
... This suggests that the ratio b/a can express a person's arousal state. In Formula 4 listed in Table 1, the arousal is expressed as the ratio between b/a concerning electrodes AF3, AF4, F3, and F4 [97]. Likewise, Arousal (Table 1, Formula 5) can also be expressed in terms of power measured in F3 and F4 [23,95]. ...
This study investigates the use of electroencephalography (EEG) to characterize emotions and provides insights into the consistency between self-reported and machine learning outcomes. Thirty participants engaged in five virtual reality environments designed to elicit specific emotions, while their brain activity was recorded. The participants self-assessed their ground truth emotional state in terms of Arousal and Valence through a Self-Assessment Manikin. Gradient Boosted Decision Tree was adopted as a classification algorithm to test the EEG feasibility in the characterization of emotional states. Distinctive patterns of neural activation corresponding to different levels of Valence and Arousal emerged, and a noteworthy correspondence between the outcomes of the self-assessments and the classifier suggested that EEG-based affective indicators can be successfully applied in emotional characterization, shedding light on the possibility of using them as ground truth measurements. These findings provide compelling evidence for the validity of EEG as a tool for emotion characterization and its contribution to a better understanding of emotional activation.
... 3) Single-modal signals with multi-stimuli. Bos et al. [34] acquired participants' emotional EEG, based on the International Affective Picture System (IAPS) and the International Affective Digital Sounds (IADS) for emotion induction. Raheel et al. [26] utilized traditional multimedia, and a electric heater, and a fan to construct the tactilely enhanced multimedia and collect EEG to classify emotions. ...
Affective data is the basis of emotion recognition, which is mainly acquired through extrinsic elicitation. To investigate the enhancing effects of multi-sensory stimuli on emotion elicitation and emotion recognition, we designed an experimental paradigm involving visual, auditory, and olfactory senses. A multimodal emotional dataset (OVPD-II) that employed the video-only or video-odor patterns as the stimuli materials, and recorded the electroencephalogram (EEG) and electrooculogram (EOG) signals, was created. The feedback results reported by subjects after each trial demonstrated that the video-odor pattern outperformed the video-only pattern in evoking individuals’ emotions. To further validate the efficiency of the video-odor pattern, the transformer was employed to perform the emotion recognition task, where the highest accuracy reached 86.65% (66.12%) for EEG (EOG) modality with the video-odor pattern, which improved by 1.42% (3.43%) compared with the video-only pattern. What’s more, the hybrid fusion (HF) method combined with the transformer and joint training was developed to improve the performance of the emotion recognition task, which achieved classify accuracies of 89.50% and 88.47% for the video-odor and video-only patterns, respectively.
... The Fourier frequency analysis splits the raw EEG signals and removes the detected noises using the bandpass filter [97]. Bos et al. [98] used a bandpass filter made available from EEGLab for Matlab to eliminate noise and artefacts from EEG recordings. The moving average filter pre-processes RESP and EDA signals [99]. ...
Automated Emotion Recognition Systems (ERS) with physiological signals help improve health and decision-making in everyday life. It uses traditional Machine Learning (ML) methods, requiring high-quality learning models for physiological data (sensitive information). However, automated ERS enables data attacks and leaks, significantly losing user privacy and integrity. This privacy problem can be solved using a novel Federated Learning (FL) approach, which enables distributed machine learning model training. This review examines 192 papers focusing on emotion recognition via physiological signals and FL. It is the first review article concerning the privacy of sensitive physiological data for an ERS. The paper reviews the different emotions, benchmark datasets, machine learning, and federated learning approaches for classifying emotions. It proposes a novel multi-modal Federated Learning for Physiological signals based on Emotion Recognition Systems (Fed-PhyERS) architecture, experimenting with the AMIGOS dataset and its applications for a next-generation automated ERS. Based on critical analysis, this paper provides the key takeaways, identifies the limitations, and proposes future research directions to address gaps in previous studies. Moreover, it reviews ethical considerations related to implementing the proposed architecture. This review paper aims to provide readers with a comprehensive insight into the current trends, architectures, and techniques utilized within the field.
... Our results also showed frontal-temporal relevance at 20-24, 24-28, and 28-32 Hz, which is supported by [36], where the high beta band (21)(22)(23)(24)(25)(26)(27)(28)(29) shows the best odor-induced emotion classification performance. Furthermore, the frontal-temporal lobe is found to be related to emotional valence processing [57]- [59]. Our results indicate relevance at the frontal-temporal region across multiple frequency bands. ...
... The plot shows relevance in frontal-temporal channels across a wide range of frequencies, mainly in the beta and gamma bands. The frontal-temporal lobe is reported to be related to emotional valence processing [57]- [59], and sub-bands of the beta and gamma band are having high relevance in the left frontal-temporal region. Additionally, frontal gamma-band is found to be related to emotion processing [54], [55], while the beta band was found to contribute to odor-induced emotion classification [36]. ...
The olfactory system enables humans to smell different odors, which are closely related to emotions. The high temporal resolution and non-invasiveness of Electroencephalogram (EEG) make it suitable to objectively study human preferences for odors. Effectively learning the temporal dynamics and spatial information from EEG is crucial for detecting odor-induced emotional valence. In this paper, we propose a deep learning architecture called Temporal Attention with Spatial Autoencoder Network (TASA) for predicting odor-induced emotions using EEG. TASA consists of a filter-bank layer, a spatial encoder, a time segmentation layer, a Long Short-Term Memory (LSTM) module, a multi-head self-attention (MSA) layer, and a fully connected layer. We improved upon the previous work by utilizing a two-phase learning framework, using the autoencoder module to learn the spatial information among electrodes by reconstructing the given input with a latent representation in the spatial dimension, which aims to minimize information loss compared to spatial filtering with CNN. The second improvement is inspired by the continuous nature of the olfactory process; we propose to use LSTM-MSA in TASA to capture its temporal dynamics by learning the intercorrelation among the time segments of the EEG. TASA is evaluated on an existing olfactory EEG dataset and compared with several existing deep learning architectures to demonstrate its effectiveness in predicting olfactory-triggered emotional responses. Interpretability analyses with DeepLIFT also suggest that TASA learns spatial-spectral features that are relevant to olfactory-induced emotion recognition.
... The recognizability of various emotions depends on how effectively it is possible to map the EEG features to the emotion representation selected. The current auditory braincomputer interface study concludes that it is better to train with visual input to increase or decrease the sensorimotor rhythm amplitude than with auditory feedback [35]. This is not linked to the recognition of feelings, although it is noted in the debate that a less evolved sense of hearing can be present in healthy individuals without eye problems [36]. ...
... Visual stimuli may be easier to recognize from brain impulses than audio stimuli since the visual sense is more developed. Following this logic, a combined attempt to evoke an emotion from both visual and auditory input can offer the optimal atmosphere for recognition of emotions [35]. In this paper the DEAP dataset [37] has been used which is available online. ...
... Later many entropy generalizations were formulated and effectively employed for various EEG-based medical research, including mental illnesses, epilepsy [16][17][18], Alzheimer's [19][20][21], autism, and depression [22,23], among others. Considering these outcomes, entropy measures were employed in studying emotion recognition from EEG signals [24][25][26][27]. ...
Human emotion recognition remains a challenging and prominent issue, situated at the convergence of diverse fields, such as brain–computer interfaces, neuroscience, and psychology. This study utilizes an EEG data set for investigating human emotion, presenting novel findings and a refined approach for EEG-based emotion detection. Tsallis entropy features, computed for q values of 2, 3, and 4, are extracted from signal bands, including theta-θ (4–7 Hz), alpha-α (8–15 Hz), beta-β (16–31 Hz), gamma-γ (32–55 Hz), and the overall frequency range (0–75 Hz). These Tsallis entropy features are employed to train and test a KNN classifier, aiming for accurate identification of two emotional states: positive and negative. In this study, the best average accuracy of 79% and an F-score of 0.81 were achieved in the gamma frequency range for the Tsallis parameter q = 3. In addition, the highest accuracy and F-score of 84% and 0.87 were observed. Notably, superior performance was noted in the anterior and left hemispheres compared to the posterior and right hemispheres in the context of emotion studies. The findings show that the proposed method exhibits enhanced performance, making it a highly competitive alternative to existing techniques. Furthermore, we identify and discuss the shortcomings of the proposed approach, offering valuable insights into potential avenues for improvements.
... The selection of stimulus materials is critical because this step directly impacts the effectiveness of emotion elicitation. Previous studies have employed various types of stimuli to evoke emotions, including music [57], pictures [58], facial expressions [15], and videos [28], [32]. Among all the available stimulus materials, videos have been found to be particularly effective because they provide both visual and auditory stimuli, which can reliably and efficiently elicit emotions. ...
Recognizing emotions from physiological signals is a topic that has garnered widespread interest, and research continues to develop novel techniques for perceiving emotions. However, the emergence of deep learning has highlighted the need for comprehensive and high-quality emotional datasets that enable the accurate decoding of human emotions. To systematically explore human emotions, we develop a multimodal dataset consisting of six basic (happiness, sadness, fear, disgust, surprise, and anger) emotions and the neutral emotion, named SEED-VII. This multimodal dataset includes electroencephalography (EEG) and eye movement signals. The seven emotions in SEED-VII are elicited by 80 different videos and fully investigated with continuous labels that indicate the intensity levels of the corresponding emotions. Additionally, we propose a novel Multimodal Adaptive Emotion Transformer (MAET), that can flexibly process both unimodal and multimodal inputs. Adversarial training is utilized in the MAET to mitigate subject discrepancies, which enhances domain generalization. Our extensive experiments, encompassing both subject-dependent and cross-subject conditions, demonstrate the superior performance of the MAET in terms of handling various inputs. Continuous labels are used to filter the data with high emotional intensity, and this strategy is proven to be effective for attaining improved emotion recognition performance. Furthermore, complementary properties between the EEG signals and eye movements and stable neural patterns of the seven emotions are observed.
