Conference Paper

An Evaluation of EEG-based Metrics for Engagement Assessment of Distance Learners

Authors:
To read the full-text of this research, you can request a copy directly from the authors.

Abstract

Maintaining students' cognitive engagement in educational settings is crucial to their performance, though quantifying this mental state in real-time for distance learners has not been studied extensively in natural distance learning environments. We record electroencephalographic (EEG) data of students watching online lecture videos and use it to predict engagement rated by human annotators. An evaluation of prior EEG-based engagement metrics that utilize power spectral density (PSD) features is presented. We examine the predictive power of various supervised machine learning approaches with both subject-independent and individualized models when using simple PSD feature functions. Our results show that engagement metrics with few power band variables, including those proposed in prior research, do not produce predictions consistent with human observations. We quantify the performance disparity between cross-subject and per-subject models and demonstrate that individual differences in EEG patterns necessitate a more complex metric for educational engagement assessment in natural distance learning environments.

No full-text available

Request Full-text Paper PDF

To read the full-text of this research,
you can request a copy directly from the authors.

... 10 School Student [165]; [166]; [167]; [168]; [169]; [170]; [171]; [172]; [173]; [174]; [175]; [176]; [177]; [178]; [179]; [180]; [181]; [182]; [183]; [184]; [185]; [186]; [187]; [120]; [35]; [188]; [189]; [190]; [191]; [192] 30 University / College Student [193]; [194]; [195]; [196]; [197]; [198]; [199]; [200]; [201]; [202]; [203]; [204]; [205]; [206]; [207]; [48]; [208]; [209]; [115]; [210]; [211]; [212]; [213]; [214]; [215]; [167]; [216]; [217]; [218]; [219]; [220]; [119]; [221]; [222]; [108]; [223]; [224]; [225]; [226]; [227]; [228]; [176]; [229]; [230]; [231]; [232]; [233]; [234]; [235]; [236]; [237]; [238]; [239]; [240]; [241]; [242]; [243]; [244]; [245]; [246]; [247]; [248]; [249]; [250]; [251]; [139]; [135]; [112]; [138]; [252]; [253]; [254]; [255]; [256]; [257] [258]; [259]; [136]; [260]; [137]; [261]; [262]; [263]; [264]; [265]; [266]; [267]); [268]; [114]; [269]; [270]; [117]; [271]; [116]; [272]; [273]; [141]; [274]; [143]; [275]; [276]; [144]; [277]; [278]; [279]; [280]; [281]; [282]; [283]; [284]; [285]; [145]; [286]; [287]; [288]; [289]; [290]; [291]; [292]; [293]; [294]; [295]; [296]; [297]; [298]; [299]; [300]; [301]; [302]; [192] ; [303]; [304]. ...
... School Student [310]; [311]; [190]. [118]; [124]; [165]; [166]; [167]; [168]; [169]; [170]; [171]; [193]; G [194]; [195]; [196]; [197]; [198]; [199]; [200]; [201]; [202]; [203]; [204]; [205]; [206]; [207]; [48]; [208]; [209]; [115]; [210]; [211]; [212]; [213]; [214]; [215]; [305]; [105]; [128]; [129]; [130]; [306]; [307]; [242]; [216]; [217]; [218]; [219]; [220]; [119]; [221]; [222]; [172]; [173]; [174]; [160]; [108]; [223]; [175]; [224]; [225]; [226]; [227]; [228]; [176]; [110]; [229]; [230]; [231]; [232]; [177]; [125]; [233]; [234]; [235]; [236]; [237]; [238]; [239]; [240]; [241]; [107]; [243]; [244]; [245]; [246]; [247]; [248]; [249]; [250]; [178]; [179]; [180]; [161]; [162] ; [132]; [308]; [113]; [111]; [310]; [311]; [309]; [290]; [292]; [294]; [295]; [296]; [297]; [298]; [188]; [299]; [163]; [300]; [301]; [302]; [190]; [192] ; [303]; [164]. ...
Article
Full-text available
Brain–computer interface (BCI) technology has the potential to positively contribute to the educational learning environment, which faces many challenges and shortcomings. Cognitive and affective BCIs can offer a deep understanding of brain mechanisms, which may improve learning strategies and increase brain-based skills. They can offer a better empirical foundation for teaching–learning methodologies, including adjusting learning content based on brain workload, measuring student interest of a topic, or even helping students focus on specific tasks. The latest findings from emerging BCI technology, neuroscience, cognitive sciences, and psychology could be used in learning and teaching strategies to improve student abilities in education. This study investigates and analyzes the research on BCI patterns and its implementation for enhancing cognitive capabilities of students. The results showed that there is insufficient literature on BCI that addresses students with disabilities in the learning process. Further, our analysis revealed a bias toward the significance of cognitive process factors compared with other influential factors, such as the learning environment and emotions that influence learning. Finally, we concluded that BCI technology could improve students’ learning and cognitive skills—when consistently associated with the different pedagogical teaching–learning strategies—for better academic achievement.
... On the other hand, the rapid development in sensory technology has enabled researchers and practitioners to push the boundaries of learning engagement detection and its analysis by investigating various machine-readable signals or behaviors, such as electroencephalogram (EEG) signal, physiological signal, electrodermal activity, facial expression, gaze, keystroke and mouse movement [17]. In order to obtain high quality data generated from the learning activities, intrusive or wearable devices such as electrode headset [18], wristband [19], or costly sophisticated eye tracker [20] have to be adopted. The adoption of these devices may cause inconvenience or discomfort to learners during data acquisition process. ...
Article
Full-text available
“Lack of supervision” is a particularly challenging problem in E-learning or distance learning environments. A wide range of research efforts and technologies have been explored to alleviate its impact by monitoring students’ engagement, such as emotion or learning behaviors. However, the current research still lacks multi-dimensional computational measures for analyzing learner’s engagement from the interactions that occur in digital learning environment. In this paper, we propose an integrated framework to identify learning engagement from three facets: affect, behavior and cognitive state, which are conveyed by learner’s facial expressions, eye movement behaviors and the overall performance during short video learning session. To recognize the three states of learners, three channel data is recorded: 1) video/image sequence captured by camera; 2) eye movement information from a non-intrusive and cost-effective eye tracker; and 3) click stream data from mouse. Based on these modalities, a multi-channel data fusion strategy is designed that concatenates time series features of three channels in the same time segment to predict course learning performance. We also presented a new method to make the self-reported annotations more reliable without using external observers’ verification. To validate the approach and methods, 46 participants were invited to attend a representative on-line course that consists of short videos in our designed learning environment. The results demonstrated the effectiveness of the proposed framework and methods in monitoring learning engagement. More importantly, a prototype system was developed to detect learner’s emotional and eye behavioral engagement in real-time as well as predict the learning performance of learners after they had completed each short video course.
Conference Paper
Full-text available
Massive open online courses (MOOCs) have received wide publicity and many institutions have invested considerable effort in developing, promoting and delivering such courses. However, there are still many unresolved questions relating to MOOCs and their effectiveness. One of the major recurring issues raised in both academic literature and the popular press is the consistently high dropout rate of MOOC learners. Although many thousands of participants enrol on these courses, the completion rate for most courses is below 13%. This paper investigates MOOC attrition from several different perspectives. Firstly, we review existing literature relating to MOOC dropout rates, bringing together existing findings on completion rates and analyses of several specific courses, which identify factors that correlate to likelihood of dropout. We provide a meta-analysis of the basic figures on overall dropout rates previously collected to identify relationships between course factors and dropout rates. In addition, the literature is reviewed from a qualitative perspective drawing together findings on the reasons for dropout and methods suggested for resolving or reducing the dropout rate. Secondly, using themes emerging from the initial investigation, we provide a preliminary analysis of data gathered from a Computing MOOC run by the University of Warwick, UK and presented using a Moodle platform. Different aspects of students' demographic data are examined to see if relationships to persistence exist. An important feature of this course is that it has been run in two different parallel modes (" traditional " MOOC mode with peer support, and " supported " mode with real time, tutored programming labs). This allows direct comparison between the dropout figures for the two different modes. Qualitative information from student evaluations is also considered. Finally, we discuss our findings relating MOOC dropout rates, considering what factors are within the control of a MOOC provider and suggesting the most promising avenues for improvement. Our results indicate that many participants who may be classed as dropouts (for example, because they do not complete the necessary components to gain a certificate) are still participating in the course in their own preferred way (either at a slower pace or with selective engagement). This suggests that the structure of " a course " may not be helpful to all participants and supporting different patterns of engagement and presentation of material may be beneficial.
Article
Full-text available
Magnetoencephalography and electroencephalography (M/EEG) measure the weak electromagnetic signals generated by neuronal activity in the brain. Using these signals to characterize and locate neural activation in the brain is a challenge that requires expertise in physics, signal processing, statistics, and numerical methods. As part of the MNE software suite, MNE-Python is an open-source software package that addresses this challenge by providing state-of-the-art algorithms implemented in Python that cover multiple methods of data preprocessing, source localization, statistical analysis, and estimation of functional connectivity between distributed brain regions. All algorithms and utility functions are implemented in a consistent manner with well-documented interfaces, enabling users to create M/EEG data analysis pipelines by writing Python scripts. Moreover, MNE-Python is tightly integrated with the core Python libraries for scientific comptutation (NumPy, SciPy) and visualization (matplotlib and Mayavi), as well as the greater neuroimaging ecosystem in Python via the Nibabel package. The code is provided under the new BSD license allowing code reuse, even in commercial products. Although MNE-Python has only been under heavy development for a couple of years, it has rapidly evolved with expanded analysis capabilities and pedagogical tutorials because multiple labs have collaborated during code development to help share best practices. MNE-Python also gives easy access to preprocessed datasets, helping users to get started quickly and facilitating reproducibility of methods by other researchers. Full documentation, including dozens of examples, is available at http://martinos.org/mne.
Article
Full-text available
Two experiments are presented examining adaptive and adaptable methods for invoking automation. Empirical investigations of adaptive automation have focused on methods used to invoke automation or on automation-related performance implications. However, no research has addressed whether performance benefits associated with brain-based systems exceed those in which users have control over task allocations. Participants performed monitoring and resource management tasks as well as a tracking task that shifted between automatic and manual modes. In the first experiment, participants worked with an adaptive system that used their electroencephalographic signals to switch the tracking task between automatic and manual modes. Participants were also divided between high- and low-reliability conditions for the system-monitoring task as well as high- and low-complacency potential. For the second experiment, participants operated an adaptable system that gave them manual control over task allocations. Results indicated increased situation awareness (SA) of gauge instrument settings for individuals high in complacency potential using the adaptive system. In addition, participants who had control over automation performed more poorly on the resource management task and reported higher levels of workload. A comparison between systems also revealed enhanced SA of gauge instrument settings and decreased workload in the adaptive condition. The present results suggest that brain-based adaptive automation systems may enhance perceptual level SA while reducing mental workload relative to systems requiring user-initiated control. Potential applications include automated systems for which operator monitoring performance and high-workload conditions are of concern.
Article
Full-text available
The ability to continuously and unobtrusively monitor levels of task engagement and mental workload in an operational environment could be useful in identifying more accurate and efficient methods for humans to interact with technology. This information could also be used to optimize the design of safer, more efficient work environments that increase motivation and productivity. The present study explored the feasibility of monitoring electroencephalo-graphic (EEG) indices of engagement and workload acquired unobtrusively and quantified during performance of cognitive tests. EEG was acquired from 80 healthy participants with a wireless sensor headset (F3-F4,C3-C4,Cz-POz,F3-Cz,Fz-C3,Fz-POz) during tasks including: multi-level forward/backward-digit-span, grid-recall, trails, mental-addition, 20-min 3-Choice Vigilance, and image-learning and memory tests. EEG metrics for engagement and workload were calculated for each 1 -s of EEG. Across participants, engagement but not workload decreased over the 20-min vigilance test. Engagement and workload were significantly increased during the encoding period of verbal and image-learning and memory tests when compared with the recognition/ recall period. Workload but not engagement increased linearly as level of difficulty increased in forward and backward-digit-span, grid-recall, and mental-addition tests. EEG measures correlated with both subjective and objective performance metrics. These data in combination with previous studies suggest that EEG engagement reflects information-gathering, visual processing, and allocation of attention. EEG workload increases with increasing working memory load and during problem solving, integration of information, analytical reasoning, and may be more reflective of executive functions. Inspection of EEG on a second-by-second timescale revealed associations between workload and engagement levels when aligned with specific task events providing preliminary evidence that second-by-second classifications reflect parameters of task performance.
Article
Advances in affective computing technologies have made it possible for researchers to investigate brain function while users interact in virtual environments. Progress in sensors and algorithms for off-the-shelf EEG systems has made it possible for gaming researchers to perform real-time estimation of human cognitive and affective states using EEG. In this study our aim was to coordinate " Task Engagement " data with " Arousal-Valence " data. The resulting coordinate was designed for application to expressive transformations to video game play in real time by tuning different performance parameters in an Engagement-Arousal rule system. Results revealed that the engagement index (Beta / (Alpha + Theta)) was capable of differentiating high intensity game events (Player Death) from general game play. Given that higher levels of engagement during death events may reflect increase in autonomic response, we also measured arousal by using (BetaF3 + BetaF4) / (AlphaF3 + AlphaF4) and valence using (AlphaF4 / BetaF4)-(AlphaF3 / BetaF3). Results revealed that arousal increases and valence decreases during high intensity game events (Player Death) when compared to lower intensity game events (General Game Play). Given our desire to establish " Task Engagement " data with " Arousal-Valence " coordinates for a flow model, we divided the data into quartiles, which allowed us to establish upper and lower thresholds to indicate when the player has left a state of flow. Our aim was to use an off-the-shelf EEG system to establish " Task Engagement " and " Arousal-Valence " coordinates during video game play that can be used for a flow model. It is believed that this model will allow for future use of the Emotiv for assessing the cognitive and emotional processing of the player.
A bio-cybernetic, closed-loop system was validated for use in an adaptive automation environment. Subjects were asked to perform either a single task or multiple tasks from the Multi-Attribute Task Battery. EEG was continuously sampled while they performed the task(s) and an EEG index was derived (20 Beta/Alpha + Theta). The system switched between manual and automatic modes according to the level of operator engagement based upon the EEG index. The NASA-TLX was administered after each trial. The results of the study demonstrated that it was possible to moderate an operator's level of engagement through a closed-loop system driven by the operator's EEG. In addition, the system was sensitive to increases in task load. These findings show promise for designing adaptive automation technology around psychophysiological input.
Article
Student engagement is a key concept in contemporary education, where it is valued as a goal in its own right. In this paper we explore approaches for automatic recognition of engagement from students’ facial expressions. We studied whether human observers can reliably judge engagement from the face; analyzed the signals observers use to make these judgments; and automated the process using machine learning. We found that human observers reliably agree when discriminating low versus high degrees of engagement (Cohen’s $kappa = 0.96$). When fine discrimination is required (four distinct levels) the reliability decreases, but is still quite high ($kappa = 0.56$). Furthermore, we found that engagement labels of 10-second video clips can be reliably predicted from the average labels of their constituent frames (Pearson $r=0.85$), suggesting that static expressions contain the bulk of the information used by observers. We used machine learning to develop automatic engagement detectors and found that for binary classification (e.g., high engagement versus low engagement), automated engagement detectors perform with comparable accuracy to humans. Finally, we show that both human and automatic engagement judgments correlate with task performance. In our experiment, student post-test performance was predicted with comparable accuracy from engagement labels ($r=0.47$) as from pre-test scores ($r=0.44$).
Article
This text is the second edition of this book. It expands the widely acclaimed 1981 book, filling more gaps between EEG and the physical sciences. EEG opens a "window on the mind" by finding new connections between psychology and physiology. Topics include synaptic sources, electrode placement, choice of reference, volume conduction, power and coherence, projection of scalp potentials to dura surface, dynamic signatures of conscious experience, and neural networks immersed in global fields of synaptic action.
Article
A biocybernetic system has been developed as a method to evaluate automated flight deck concepts for compatibility with human capabilities. A biocybernetic loop is formed by adjusting the mode of operation of a task set (e.g., manual/automated mix) based on electroencephalographic (EEG) signals reflecting an operator's engagement in the task set. A critical issue for the loop operation is the selection of features of the EEG to provide an index of engagement upon which to base decisions to adjust task mode. Subjects were run in the closed-loop feedback configuration under four candidate and three experimental control definitions of an engagement index. The temporal patterning of system mode switching was observed for both positive and negative feedback of the index. The indices were judged on the basis of their relative strength in exhibiting expected feedback control system phenomena (stable operation under negative feedback and unstable operation under positive feedback). Of the candidate indices evaluated in this study, an index constructed according to the formula, beta power/(alpha power + theta power), reflected task engagement best.
Article
Changes in cortical activity during working memory tasks were examined with electroencephalograms (EEGs) sampled from 115 channels and spatially sharpened with magnetic resonance imaging (MRI)-based finite element deblurring. Eight subjects performed tasks requiring comparison of each stimulus to a preceding one on verbal or spatial attributes. A frontal midline theta rhythm increased in magnitude with increased memory load. Dipole models localized this signal to the region of the anterior cingulate cortex. A slow (low-frequency), parietocentral, alpha signal decreased with increased working memory load. These signals were insensitive to the type of stimulus attribute being processed. A faster (higher-frequency), occipitoparietal, alpha signal was relatively attenuated in the spatial version of the task, especially over the posterior right hemisphere. Theta and alpha signals increased, and overt performance improved, after practice on the tasks. Increases in theta with both increased task difficulty and with practice suggests that focusing attention required more effort after an extended test session. Decreased alpha in the difficult tasks indicates that this signal is inversely related to the amount of cortical resources allocated to task performance. Practice-related increases in alpha suggest that fewer cortical resources are required after skill development. These results serve: (i) to dissociate the effects of task difficulty and practice; (ii) to differentiate the involvement of posterior cortex in spatial versus verbal tasks; (iii) to localize frontal midline theta to the anteromedial cortex; and (iv) to demonstrate the feasibility of using anatomical MRIs to remove the blurring effect of the skull and scalp from the ongoing EEG. The results are discussed with respect to those obtained in a prior study of transient evoked potentials during working memory.
Article
The necessity of psychophysiological research in ergonomics has gradually been recognized in Japan. In this paper, frontal midline theta rhythm (Fm-theta) and eyeblinking are recommended as tools in this field, especially for assessing workers' attention concentration, mental workload, fatigue, and interest during VDT work at the workplace and playing video games at home. In experiment 1, Fm-theta and eyeblink rates were measured in 10 Japanese abacus experts (Group E) and 10 normal students (Group C) during a visual search task with VDT. Memory load affected all measures. The amount of Fm-theta appeared more in Group E than Group C, but blink rate was lower in Group E than in Group C. As abacus experts have such highly developed skills in concentration, the result indicates that the amount of Fm-theta would be a good index of attention concentration in VDT workers. The second experiment was done with 10 school-aged children as subjects during three visual tasks: video game, mental test and animation. Amounts of Fm-theta and the degree of blink inhibition were maximum while playing the video game, which all subjects reported they most preferred, and minimum while watching animation, which eight subjects reported to be most boring. An interesting task would seem to provoke Fm-theta and inhibit eyeblink activity. From these two experiments, Fm-theta and eyeblink rate would appear to be good indices of attention concentration and task pleasantness of a mental task using VDT.
Article
The present study was designed to determine whether a biocybernetic, adaptive system could enhance vigilance performance. Participants were asked to monitor the repetitive presentation of white bars on a computer screen for occasional increases in length. An index of task engagement was derived from participants' electroencephalographic (EEG) activity and was used to change the presentation rate of events among 3 values (6, 20, and 60 events/min). Under a negative feedback contingency, event rates increased if the engagement index decreased and, conversely, decreased if the index increased. Under positive feedback, the opposite contingency existed. Each experimental participant had a yoked control partner who received the same pattern of changes in event rates irrespective of his or her EEG activity. The results showed that better vigilance performance was obtained under negative feedback and that the performance of the yoked participants was similar to that of their experimental partners. These findings suggest that it may be possible to improve monitoring performance on critical activities such as air traffic control and radar and sonar operation through a pattern of event rate changes that do not rely on an operator's overt behavior.
Article
The present study examined the effects of an electroencephalographic- (EEG-) based system for adaptive automation on tracking performance and workload. In addition, event-related potentials (ERPs) to a secondary task were derived to determine whether they would provide an additional degree of workload specificity. Participants were run in an adaptive automation condition, in which the system switched between manual and automatic task modes based on the value of each individual's own EEG engagement index; a yoked control condition; or another control group, in which task mode switches followed a random pattern. Adaptive automation improved performance and resulted in lower levels of workload. Further, the P300 component of the ERP paralleled the sensitivity to task demands of the performance and subjective measures across conditions. These results indicate that it is possible to improve performance with a psychophysiological adaptive automation system and that ERPs may provide an alternative means for distinguishing among levels of cognitive task demand in such systems. Actual or potential applications of this research include improved methods for assessing operator workload and performance.
Article
The performance of an adaptive automation system was evaluated using a cognitive vigilance task. Participants responded to the presence of a green "K" in an array of two, five, or nine distractor stimuli during a 40-min vigil. The array with the target stimulus was presented once each minute. Participants EEG was recorded and an engagement index (EI = 20 x beta/(alpha + theta)) was derived. In the negative feedback condition, increases in the EI caused the number of stimuli in the array to decrease while decreases in the EI caused the number of stimuli to increase. For the positive feedback condition, increases in the index caused an increase in the array size (AS) while decreases caused a decrease in the array size. Each experimental participant had a yoked control partner who received the same pattern of changes in array irrespective of their engagement index. A vigilance decrement was seen only for the positive feedback, experimental group.
Article
We describe a set of computational tools able to estimate cortical activity and connectivity from high-resolution EEG and fMRI recordings in humans. These methods comprise the estimation of cortical activity using realistic geometry head volume conductor models and distributed cortical source models, followed by the evaluation of cortical connectivity between regions of interest coincident with the Brodmann areas via the use of Partial Directed Coherence. Connectivity patterns estimated on the cortical surface in different frequency bands are then imaged and interpreted with measures based on graph theory. These computational tools were applied on a set of EEG and fMRI data from a Stroop task to demonstrate the potential of the proposed approach. The present findings suggest that the methodology is able to identify differences in functional connectivity patterns elicited by different experimental tasks or conditions.
EEG based cognitive workload classification during NASA MATB-II multitasking
  • Sushil Chandra
  • Greeshma Verma
  • Alok Sharma
  • Devendra Mittal
  • Jha
Sushil Chandra, Kundan Lal Verma, Greeshma Sharma, Alok Mittal, and Devendra Jha, "EEG based cognitive workload classification during NASA MATB-II multitasking," International Journal of Cognitive Research in Science, Engineering and Education (IJCRSEE), vol. 3, no. 1, pp. 35-41, 2015.