No full-text available
To read the full-text of this research,
you can request a copy directly from the authors.
Psychophysiological Sensing and State Classification for Attention Management in Commercial Aviation
... Various features, such as statistical [16,22,34] and power spectral density features [16,18,[21][22][23][24][25]28,34,35], have been extracted from pilots' EEG recordings in earlier research in order to classify pilots' mental states. For example, Wu et al. [28] used the power spectrum curve area representation of the decomposed delta, theta, alpha, and beta brain waves obtained using wavelet packet transform as features to perform the classification. ...
... For the detection of various pilot mental states, previous studies implemented various ML [18,[22][23][24][25][26][27]34,35,40,41] and DL [16,18,25,26,28,35,42,43] algorithms. For instance, Han et al. [25] proposed a detection system based on multimodal physiological signals and a multimodal deep learning (MDL) network, consisting of convolutional neural network (CNN) and long short-term memory (LSTM) algorithms, to detect pilot's mental states, namely distraction, workload, fatigue, and normal. ...
... For the detection of various pilot mental states, previous studies implemented various ML [18,[22][23][24][25][26][27]34,35,40,41] and DL [16,18,25,26,28,35,42,43] algorithms. For instance, Han et al. [25] proposed a detection system based on multimodal physiological signals and a multimodal deep learning (MDL) network, consisting of convolutional neural network (CNN) and long short-term memory (LSTM) algorithms, to detect pilot's mental states, namely distraction, workload, fatigue, and normal. ...
The safety of flight operations depends on the cognitive abilities of pilots. In recent years, there has been growing concern about potential accidents caused by the declining mental states of pilots. We have developed a novel multimodal approach for mental state detection in pilots using electroencephalography (EEG) signals. Our approach includes an advanced automated preprocessing pipeline to remove artefacts from the EEG data, a feature extraction method based on Riemannian geometry analysis of the cleaned EEG data, and a hybrid ensemble learning technique that combines the results of several machine learning classifiers. The proposed approach provides improved accuracy compared to existing methods, achieving an accuracy of 86% when tested on cleaned EEG data. The EEG dataset was collected from 18 pilots who participated in flight experiments and publicly released at NASA’s open portal. This study presents a reliable and efficient solution for detecting mental states in pilots and highlights the potential of EEG signals and ensemble learning algorithms in developing cognitive cockpit systems. The use of an automated preprocessing pipeline, feature extraction method based on Riemannian geometry analysis, and hybrid ensemble learning technique set this work apart from previous efforts in the field and demonstrates the innovative nature of the proposed approach.
... [1]. Advanced Air Mobility (AAM) builds upon the UAM concept by incorporating use cases not specific to operations in urban environments [2]. A significant economic barrier on the introduction of these concepts is the cost of an onboard human vehicle operator. ...
... Multistate classifiers implemented using machine learning and deep learning techniques. Multi-state prediction using this method can identify non-nominal attentional states at rates >80% [2,18]. ...
... Another method has also been presented [29], using heart rate variability, finger plethysmogram amplitude, and perspiration behavior to assess workload. Other methods should be explored, such as multi-modal classifications using galvanic skin response and pre-processed electroencephalography, or measures of autonomous nervous system responses [2][3][4][5][6] to detect an overloaded operator toward the allocation of functions. ...
... The EEG data was sourced from the AHPLS dataset, a rich and diverse dataset that has been used in several recent studies to understand and model human cognitive states [16][17][18][19][20][21]. The AHPLS dataset is unique in its inclusion of data from pilots under various mental states, namely channelised attention (CA), diverted attention (DA), startle/surprise (SS), and normal/no event (NE) states. ...
... Several studies have explored the detection of specific mental states using EEG data. For instance, Harrivel et al. [17,18] recorded brain signals (i.e., EEG) and non-brain signals (i.e., ECG, R, and galvanic skin response (GSR)), capturing the attention-related pilot performance limiting states, including CA, DA, SS, and NE. The authors employed various ML techniques to perform binary and multiclass classification tasks in two different studies. ...
Predicting pilots’ mental states is a critical challenge in aviation safety and performance, with electroencephalogram data offering a promising avenue for detection. However, the interpretability of machine learning and deep learning models, which are often used for such tasks, remains a significant issue. This study aims to address these challenges by developing an interpretable model to detect four mental states—channelised attention, diverted attention, startle/surprise, and normal state—in pilots using EEG data. The methodology involves training a convolutional neural network on power spectral density features of EEG data from 17 pilots. The model’s interpretability is enhanced via the use of SHapley Additive exPlanations values, which identify the top 10 most influential features for each mental state. The results demonstrate high performance in all metrics, with an average accuracy of 96%, a precision of 96%, a recall of 94%, and an F1 score of 95%. An examination of the effects of mental states on EEG frequency bands further elucidates the neural mechanisms underlying these states. The innovative nature of this study lies in its combination of high-performance model development, improved interpretability, and in-depth analysis of the neural correlates of mental states. This approach not only addresses the critical need for effective and interpretable mental state detection in aviation but also contributes to our understanding of the neural underpinnings of these states. This study thus represents a significant advancement in the field of EEG-based mental state detection.
... Therefore, the field of aviation could benefit from continuous monitoring and detection of pilot's workload. Many studies reported on the classification of workload (Blanco et al., 2016;Dehais et al., 2019a;Durantin et al., 2016;Gateau et al., 2015Harrivel et al., 2016;Kakkos et al., 2019;Saproo et al., 2016;Verdière et al., 2018; see Table 4). Notably, fNIRS is used more widely in the classification of workload compared to the classification of other cognitive concepts. ...
... These attempts range from classifying mental states such as drowsiness and high workload, to cognitive phenomena such as inattentional deafness, emotions and anticipation of visual cues. Most of the classifiers discussed have performed above chance level, and in some cases, pilot's mental state predictions can be improved with the use of multimodal features, such as GSR (Harrivel et al., 2016;Roza and Postolache, 2019;Wright and McGown, 2001; and measurements of eye behavior (Diaz-Piedra et al., 2019;Guragain et al., 2019;Neri et al., 2002;Saproo et al., 2016;Wang et al., 2019a;Wright and McGown, 2001). Thus, it can be concluded that classifiers in general can provide objective measures for mental states and cognition in aviation. ...
This paper systematically reviews 20 years of publications (N = 54) on aviation and neurophysiology. The main goal is to provide an account of neurophysiological changes associated with flight training with the aim of identifying neurometrics indicative of pilot's flight training level and task relevant mental states, as well as to capture the current state-of-art of (neuro)ergonomic design and practice in flight training. We identified multiple candidate neurometrics of training progress and workload, such as frontal theta power, the EEG Engagement Index and the Cognitive Stability Index. Furthermore, we discovered that several types of classifiers could be used to accurately detect mental states, such as the detection of drowsiness and mental fatigue. The paper advances practical guidelines on terminology usage, simulator fidelity, and multimodality, as well as future research ideas including the potential of Virtual Reality flight simulations for training, and a brain-computer interface for flight training.
... The United States Air Force Research Laboratory has used psychophysiological features to predict functional state during multiple flight tasks, including the Multi-Attribute Task Battery (MATB) and an air traffic control task [Christensen et al. 2012;Wilson and Russell 2003a,b]. Crew state monitoring using multi-modal physiological sensing has also been tested at NASA Langley Research Center in flight simulation cockpits [Harrivel et al. 2016[Harrivel et al. , 2017. Specifically, the crew state monitoring team was able to achieve an average multi-state prediction accuracy of 88.6% using electroencephalography (EEG), galvanic skin response (GSR), and heart rate variability (HRV) with a subject-dependent model. ...
... The MATB is a well-validated tool complete with system monitoring, tracking, resource management, and communications tasks that can be adjusted in frequency and difficulty to simulate high and low workload flight events [Comstock and Arnegard 1992]. Both the PST and the MATB have been used in previous psychophysiological monitoring studies [Das et al. 2017;Harrivel et al. 2016;Saha et al. 2017;Wilson and Russell 2003b]. As mentioned previously, subjective assessments, namely the NASA Task Load Index (TLX), and performance metrics of speed and accuracy from the PST and MATB trials will be used to validate the high and low workload labels. ...
As next-generation space exploration missions necessitate increasingly autonomous systems, there is a critical need to better detect and anticipate crewmember interactions with these systems. The success of present and future autonomous technology in exploration spaceflight is ultimately dependent upon safe and efficient interaction with the human operator. Optimal interaction is particularly important for surface missions during highly coordinated extravehicular activity (EVA), which consists of high physical and cognitive demands with limited ground support. Crew functional state may be affected by a number of variables including workload, stress, and motivation. Real-time assessments of crew state that do not require a crewmember's time and attention to complete will be especially important to assess operational performance and behavioral health during flight. In response to the need for objective, passive assessment of crew state, the aim of this work is to develop an accurate and precise prediction model of human functional state for surface EVA using multi-modal psychophysiological sensing. The psychophysiological monitoring approach relies on extracting a set of features from physiological signals and using these features to classify an operator's cognitive state. This work aims to compile a non-invasive sensor suite to collect physiological data in real-time. Training data during cognitive and more complex functional tasks will be used to develop a classifier to discriminate high and low cognitive workload crew states. The classifier will then be tested in an operationally relevant EVA simulation to predict cognitive workload over time. Once a crew state is determined, further research into specific countermeasures, such as decision support systems, would be necessary to optimize the automation and improve crew state and operational performance.
... CAST identified distraction resulting from attention-related human performance limiting states (AHPLS) as a contributing factor to loss of airplane state awareness (ASA) in a set of 18 commercial aviation accidents and incidents [2]. The CSM team conducted a series of research studies utilizing a suite of sensors to detect and identify the psychophysiological signatures of suboptimal mental states (channelized attention, diverted attention, startle/surprise, and confirmation bias) while pilots engaged in a state-targeted high-fidelity air traffic flight simulation [3,4,5]. This work demonstrated the ability to produce real-time classification of targeted states using multiple modalities collected with convenient sensor technologies during scenarios designed to induce AHPLS. ...
... In anticipated future flight operations with commercial aircraft and emergent UAM vehicles as described in this paper, the skill level of the human operators is expected to vary greatly, thus underlining the importance of monitoring highly variable human performance. To collect objective data regarding human experience and performance within these new systems, the TUCaNs will include CSM systems [3,4] based on psychophysiological monitoring of the human operators to enable coordinated stimulus presentation and mental state prediction. ...
... These two states are characterized by the disengagement of the executive network, underpinned by the deactivation of the dorsolateral prefrontal cortex (Durantin et al., 2014;Harrivel, Weissman, Noll & Peltier, 2013). Secondly, attentional over-engagement, also referred to attentional tunneling (Wickens, 2005) and "channelized attention" (Harrivel, et al., 2016), is defined as "the allocation of attention to a particular channel of information, diagnostic hypothesis or task goal, for a duration that is longer than optimal, given the expected cost of neglecting events on other channels, failing to consider other hypotheses, or failing to perform other tasks". Some authors postulate that this impaired attentional state results from a disengagement deficit of the orientation network underpinned by the thalamus (LaBerge et al., 1992). ...
... Here the adaptation involves an attention management training approach to complement the usual observations of airline training instructor pilots by informing them, in the training context, of the occurrence of attention-related human performance limiting states (AHPLS) experienced by their trainees. Classifier models are trained to recognize trainee state during simulated flight scenarios based on patterns of the physiological signals measured during benchmark tasks (Harrivel, et al., 2016). Machine learning models' real time determinations of the cognitive states induced by the scenario tasks are displayed as gauges embedded in a mosaic of windows that also displays real time images of the scenario tasks that the trainee is performing (e.g., scene camera, simulator displays, animation of simulator controls), and this mosaic 1 is video recorded . ...
Human operators interacting with machines or computers continually adapt to the needs of the system ideally resulting in optimal performance. In some cases, however, deteriorated performance is an outcome. Adaptation to the situation is a strength expected of the human operator which is often accomplished by the human through self-regulation of mental state. Adaptation is at the core of the human operator’s activity, and research has demonstrated that the implementation of a feedback loop can enhance this natural skill to improve training and human/machine interaction. Biocybernetic adaptation involves a “loop upon a loop,” which may be visualized as a superimposed loop which senses a physiological signal and influences the operator’s task at some point. Biocybernetic adaptation in, for example, physiologically adaptive automation employs the “steering” sense of “cybernetic,” and serves a transitory adaptive purpose – to better serve the human operator by more fully representing their responses to the system. The adaptation process usually makes use of an assessment of transient cognitive state to steer a functional aspect of a system that is external to the operator’s physiology from which the state assessment is derived. Therefore, the objective of this paper is to detail the structure of biocybernetic systems regarding the level of engagement of interest for adaptive systems, their processing pipeline, and the adaptation strategies employed for training purposes, in an effort to pave the way towards machine awareness of human state for self-regulation and improved operational performance.
... The initial results quantified the ability to discriminate between cognitive states as induced by benchmark tasks. 3 In the current study, the same benchmark tasks were used with new pilot participants to train classifier models which are then used to predict the cognitive state of those participants during flight simulation scenarios. The initial focus is on the states of Channelized Attention and Startle/Surprise. ...
... Use of benchmark tasks was modeled after the methods of Hirshfield, et al. 16 The AHPLS to be predicted and the selected benchmark tasks are listed in Table 1 and were described previously. 3 These tasks are used to induce AHPLS under controlled conditions for 6 minutes each, and were chosen for their high likelihood to induce these experiences in isolation and with the full knowledge of the participant (except for the startle task and the high versus low workload condition). Many of these tasks have been employed in previous task-oriented research. ...
The Commercial Aviation Safety Team found the majority of recent international commercial aviation accidents attributable to loss of control inflight involved flight crew loss of airplane state awareness (ASA), and distraction was involved in all of them. Research on attention-related human performance limiting states (AHPLS) such as channelized attention, diverted attention, startle/surprise, and confirmation bias, has been recommended in a Safety Enhancement (SE) entitled “Training for Attention Management.” To accomplish the detection of such cognitive and psychophysiological states, a broad suite of sensors was implemented to simultaneously measure their physiological markers during a high fidelity flight simulation human subject study. Twenty-four pilot participants were asked to wear the sensors while they performed benchmark tasks and motion-based flight scenarios designed to induce AHPLS. Pattern classification was employed to predict the occurrence of AHPLS during flight simulation also designed to induce those states. Classifier training data were collected during performance of the benchmark tasks. Multimodal classification was performed, using pre-processed electroencephalography, galvanic skin response, electrocardiogram, and respiration signals as input features. A combination of one, some or all modalities were used. Extreme gradient boosting, random forest and two support vector machine classifiers were implemented. The best accuracy for each modality-classifier combination is reported. Results using a select set of features and using the full set of available features are presented. Further, results are presented for training one classifier with the combined features and for training multiple classifiers with features from each modality separately. Using the select set of features and combined training, multistate prediction accuracy averaged 0.64 +/- 0.14 across thirteen participants and was significantly higher than that for the separate training case. These results support the goal of demonstrating simultaneous real-time classification of multiple states using multiple sensing modalities in high fidelity flight simulators. This detection is intended to support and inform training methods under development to mitigate the loss of ASA and thus reduce accidents and incidents.
*editorial comment added 3/28/2023: This work was done considering professional pilots in the context of commercial aviation, and does not make claims regarding undergraduate motivation.*
... Here we list some common examples in the literature. Classification has been applied to detect failure in prognostics [32], analyze pilot cognitive and psychophysiological states [33], and predict aerospace structure defect [34]. Clustering has been widely used to characterize and predict traffic flow patterns [35], extract representative flight trajectories [36], reduce big data from real-world operations [9], and study the patterns of air transportation emissions [37]. ...
Metric learning is the process of learning a tailored distance metric for a particular task. This advanced subfield of machine learning is useful to any machine learning or data mining task that relies on the computation of distances or similarities over objects. In recently years, machine learning techniques have been extensively used in aviation and aerospace engineering to make predictions, extract patterns, discover knowledge, etc. Nevertheless, metric learning, an element that can advance the performance of complex machine learning tasks, has so far been hardly utilized in relevant literature. In this study, we apply classic metric learning formulations with novel components on aviation environmental impact modeling. Through a weakly-supervised metric learning task, we achieve significant improvement in the newly emerged problem of aircraft characterization and segmentation for environmental impacts. The result will enable the more efficient and accurate modeling of aircraft environmental impacts, a focal topic in sustainable aviation. This work is also a demonstration that shows the potential and value of metric learning in a wide variety of similar studies in the transportation domain.
... Based on the insights gained from previous work, we will include eye tracking to record pilots' eye movements, as well as their heart rate (HR), base/resting pulse (BP), and endodermal activity (EDA). The combination of these measures allows the framework to model the pilot's psychophysiological state, which has been indicated by previous work [36]. Furthermore, we will record the aircraft state and the pilot's inputs, which provide the framework with context like the type of situation and reaction of the pilot, specifically during an aircraft failure [37]. ...
Eye tracking has a longstanding history in aviation research. Amongst others it has been employed to bring pilots back "in the loop", i.e., create a better awareness of the flight situation. Interestingly, there exists only little research in this context that evaluates the application of machine learning algorithms to model pilots' understanding of the aircraft's state and their situation awareness. Machine learning models could be trained to differentiate between normal and abnormal patterns with regard to pilots' eye movements, control inputs, and data from other psychophysiological sensors, such as heart rate or blood pressure. Moreover, when the system recognizes an abnormal pattern, it could provide situation specific assistance to bring pilots back in the loop. This paper discusses when pilots benefit from such a pilot-aware system, and explores the technical and user oriented requirements for implementing this system.
... Furthermore, learning-based classifiers have also been used to detect high and low anxiety in drivers from ECG and accelerometer data (Dobbins and Fairclough, 2018), or to create a virtual driving platform to maximize engagement in people with autism spectrum disorder (Bian et al., 2019). Applications for airplane pilots used classifiers to identify features from EEG and skin response signals that can model the users in scenarios of attention-related human performance limiting states (Harrivel et al., 2016) or to find relationships of cardiovascular features with psychophysiological stress while performing piloting maneuvers (Hanakova et al., 2017). To the best of our knowledge, this is the first project that aims at characterizing psychophysiological responses of police officers on duty for designing biocybernetic loops in VR firearms training. ...
Crucial elements for police firearms training include mastering very specific psychophysiological responses associated with controlled breathing while shooting. Under high-stress situations, the shooter is affected by responses of the sympathetic nervous system that can impact respiration. This research focuses on how frontal oscillatory brainwaves and cardiovascular responses of trained police officers (N = 10) are affected during a virtual reality (VR) firearms training routine. We present data from an experimental study wherein shooters were interacting in a VR-based training simulator designed to elicit psychophysiological changes under easy, moderate and frustrating difficulties. Outcome measures in this experiment include electroencephalographic and heart rate variability (HRV) parameters, as well as performance metrics from the VR simulator. Results revealed that specific frontal areas of the brain elicited different responses during resting states when compared with active shooting in the VR simulator. Moreover, sympathetic signatures were found in the HRV parameters (both time and frequency) reflecting similar differences. Based on the experimental findings, we propose a psychophysiological model to aid the design of a biocybernetic adaptation layer that creates real-time modulations in simulation difficulty based on targeted physiological responses.
... The general aim of pilot training is to (i) teach the correct understanding and handling of an aircraft, (ii) continuously increase the level of automation as to (iii) enable correct decision-making and task execution even under high pressure. To avoid attentional and procedural errors, such as loss of control in flight and loss of airplane state awareness [12]- [14], pilot actions are generally fixed in rigorous, pre-defined processes. While this approach successfully reduces procedural errors, there is no adaption to individual capabilities in training, putting invested hours on the same level as developed skills. ...
... Finally, other classification methods should be explored, addressing cross-time and cross-task generalizability of the classifier models. Regardless of these potential improvements, we note that the present techniques are ready for investigations in which hemodynamic measures synergistically complement EEG and other psychophysiological measures in multimodal classification schemes [94]. ...
Brain activity can predict a person’s level of engagement in an attentional task. However, estimates of brain activity are often confounded by measurement artifacts and systemic physiological noise. The optimal method for filtering this noise – thereby increasing such state prediction accuracy – remains unclear. To investigate this, we asked study participants to perform an attentional task while we monitored their brain activity with functional near infrared spectroscopy (fNIRS). We observed higher state prediction accuracy when noise in the fNIRS hemoglobin [Hb] signals was filtered with a non-stationary (adaptive) model as compared to static regression (84% ± 6% versus 72% ± 15%).
Detecting the cognitive states of pilots from their electroencephalogram signals is a challenging task and has different applications. At present there are a number of methods available for the cognitive states prediction. However imbalance in data is ultimately the challenge impacting the analysis performance. Here we use condensed nearest neighbors under-sampling technique to recover the class balance of training data before implementing k-nearest neighbors for classification of the electroencephalogram signals. The underlying cognitive states of pilots can be detected from integration of these techniques that include also data pre-processing and dimensionality reduction. Results of experiments for a benchmark database are reported with improved performance.
In the aviation industry, safety remains vital, often compromised by pilot errors attributed to factors such as workload, fatigue, stress, and emotional disturbances. To address these challenges, recent research has increasingly leveraged psychophysiological data and machine learning techniques, offering the potential to enhance safety by understanding pilot behavior. This systematic literature review rigorously follows a widely accepted methodology, scrutinizing 80 peer-reviewed studies out of 3352 studies from five key electronic databases. The paper focuses on behavioral aspects, data types, preprocessing techniques, machine learning models, and performance metrics used in existing studies. It reveals that the majority of research disproportionately concentrates on workload and fatigue, leaving behavioral aspects like emotional responses and attention dynamics less explored. Machine learning models such as tree-based and support vector machines are most commonly employed, but the utilization of advanced techniques like deep learning remains limited. Traditional preprocessing techniques dominate the landscape, urging the need for advanced methods. Data imbalance and its impact on model performance is identified as a critical, under-researched area. The review uncovers significant methodological gaps, including the unexplored influence of preprocessing on model efficacy, lack of diversification in data collection environments, and limited focus on model explainability. The paper concludes by advocating for targeted future research to address these gaps, thereby promoting both methodological innovation and a more comprehensive understanding of pilot behavior.
The fundamental approach to improve pilots’ situation awareness (SA) would be to reorganize and restructure the presentation of information to fit pilot's cognitive model on the flight deck. This would facilitate pilots’ perception, understanding, and projection, hence making it easier to find the relevant targets. Sixty pilots (30 B‐737 pilots and 30 B‐777 pilots) participated in this study to investigate pilots’ SA while interacting with digital displays and moving pointed needle displays on cabin pressurization system. The results have shown significant differences on pilots’ perception, understanding, and overall SA between digital display and pointed display on the flight deck. Pilots significantly preferred the digital design cabin pressurization system, which is consistent with the proximity compatibility principle, and the position of the display on the center instrument panel is easily accessible to both pilots and does not require large head movements. There are some recommendations on the cabin pressurization design including the size of outflow valve position indicator, which should be significantly increased to provided saliency of information; color coding should be used on cabin altitude and differential pressure indicator to mark critical cabin altitude; and standard operating procedures shall include cabin altitude and differential pressure reading by pilot monitoring. The final and completed solution to the issues on the cabin pressurization system is to redesign the scattered pointed displays as integrated digital displays to fit the human‐centered principle.
This paper presents an adaptive neuro/fuzzy system which can be trained to detect the current human emotions from a set of measured responses. Six models are built using different types of input/output membership functions and trained by different kinds of input arrays. The models are compared based on their ability to train with lowest error values. Many factors impact the error values such as input/output membership functions, the training data arrays, and the number of epochs required to train the model.
The concept of task engagement is associated with effortful striving to reach a desired goal. This dimension is fundamental for software designed to elicit high quality performance. This paper will review the concept of task engagement, both in the psychological literature and with respect to affective computing approaches, such as biofeedback and the definition of 'flow' states. This paper will briefly describe a series of laboratory experiments designed to explore measures of task engagement based on EEG and cardiovascular measures. These experiments employed a number of manipulations to influence task engagement, e.g. performance feedback, task difficulty and financial incentives. Results demonstrated the sensitivity of EEG measures to cognitive sources of engagement (e.g. mental workload) whilst cardiovascular variables tended to respond to the motivation to achieve. We use these findings to explore how real-time monitoring of engagement may generate adaptive dynamics for software design using a computer game as an exemplar system.
The ability to distinguish between high and low levels of task engagement in the real world is important for detecting and preventing performance decrements during safety-critical operational tasks. We therefore investigated whether functional Near Infrared Spectroscopy (fNIRS), a portable brain neuroimaging technique, can be used to distinguish between high and low levels of task engagement during the performance of a selective attention task. A group of participants performed the multi-source interference task (MSIT) while we recorded brain activity with fNIRS from two brain regions. One was a key region of the “task-positive” network, which is associated with relatively high levels of task engagement. The second was a key region of the “task-negative” network, which is associated with relatively low levels of task engagement (e.g., resting and not performing a task). Using activity in these regions as inputs to a multivariate pattern classifier, we were able to predict above chance levels whether participants were engaged in performing the MSIT or resting. We were also able to replicate prior findings from functional magnetic resonance imaging (fMRI) indicating that activity in task-positive and task-negative regions is negatively correlated during task performance. Finally, data from a companion fMRI study verified our assumptions about the sources of brain activity in the fNIRS experiment and established an upper bound on classification accuracy in our task. Together, our findings suggest that fNIRS could prove quite useful for monitoring cognitive state in real-world settings.
Theories of attention, too often generated from artificial laboratory experiments, may have limited validity when attention in the natural world is considered. For instance, for more than two decades, conceptualizations of “reflexive” and “volitional” shifts of spatial attention have been grounded in methodologies that do not recognize or utilize the basic fact that people routinely use the eyes of other people as rich and complex attentional cues. This fact was confirmed by our novel discovery that eyes will trigger a reflexive shift of attention even when they are presented centrally and are known to be spatially nonpredictive. This exploration of real-world attention also led to our finding that, contrary to popular wisdom, arrows, like eyes, are capable of producing reflexive shifts of attention—a discovery that brings into question much of the existing attention research. We argue that research needs to be grounded in the real world and not in experimental paradigms. It is time for cognitive psychology to reaffirm the difficult task of studying attention in a manner that has relevance to real-life situations.
Physiological computing represents a mode of human–computer interaction where the computer monitors, analyzes and responds to the user’s psychophysiological activity in real-time. Within the field, autonomic nervous system responses have been studied extensively since they can be measured quickly and unobtrusively. However, despite a vast body of literature available on the subject, there is still no universally accepted set of rules that would translate physiological data to psychological states. This paper surveys the work performed on data fusion and system adaptation using autonomic nervous system responses in psychophysiology and physiological computing during the last ten years. First, five prerequisites for data fusion are examined: psychological model selection, training set preparation, feature extraction, normalization and dimension reduction. Then, different methods for either classification or estimation of psychological states from the extracted features are presented and compared. Finally, implementations of system adaptation are reviewed: changing the system that the user is interacting with in response to cognitive or affective information inferred from autonomic nervous system responses. The paper is aimed primarily at psychologists and computer scientists who have already recorded autonomic nervous system responses and now need to create algorithms to determine the subject’s psychological state.
The acoustic startle reflex (ASR), a defensive response, is a contraction of the skeletal and facial muscles in response to an abrupt, intense (>80 db) auditory stimulus, which has been extensively studied in rats and humans. Prepulse inhibition (PPI) of ASR is the normal suppression of the startle reflex when an intense stimulus is preceded by a weak non-starting pre-stimulus. PPI, a measure of sensory motor gating, is impaired in various neuropsychiatric disorders, including schizophrenia, and is modulated by cognitive and emotional contexts such as fear and attention. We have modeled the fear modulation of PPI of ASR based on its anatomical substrates and taking into account data from behaving rats and humans. The model replicates the principal features of both phenomena and predicts underlying neural mechanisms. In addition, the model yields testable predictions.
A well designed user interface (UI) should be transparent, allowing users to focus their mental workload on the task at hand. We hypothesize that the overall mental workload required to perform a task using a computer system is composed of a portion attributable to the difficulty of the underlying task plus a portion attributable to the complexity of operating the user interface. In this regard, we follow Shneiderman's theory of syntactic and semantic components of a UI. We present an experiment protocol that can be used to measure the workload experienced by users in their various cognitive resources while working with a computer. We then describe an experiment where we used the protocol to quantify the syntactic workload of two user interfaces. We use functional near infrared spectroscopy, a new brain imaging technology that is beginning to be used in HCI. We also discuss extensions of our techniques to adaptive interfaces.
The ability to continuously and unobtrusively monitor levels of task engagement and mental workload in an operational environment could be useful in identifying more accurate and efficient methods for humans to interact with technology. This information could also be used to optimize the design of safer, more efficient work environments that increase motivation and productivity.
The present study explored the feasibility of monitoring electroencephalo-graphic (EEG) indices of engagement and workload acquired unobtrusively and quantified during performance of cognitive tests. EEG was acquired from 80 healthy participants with a wireless sensor headset (F3-F4,C3-C4,Cz-POz,F3-Cz,Fz-C3,Fz-POz) during tasks including: multi-level forward/backward-digit-span, grid-recall, trails, mental-addition, 20-min 3-Choice Vigilance, and image-learning and memory tests. EEG metrics for engagement and workload were calculated for each 1 -s of EEG.
Across participants, engagement but not workload decreased over the 20-min vigilance test. Engagement and workload were significantly increased during the encoding period of verbal and image-learning and memory tests when compared with the recognition/ recall period. Workload but not engagement increased linearly as level of difficulty increased in forward and backward-digit-span, grid-recall, and mental-addition tests. EEG measures correlated with both subjective and objective performance metrics.
These data in combination with previous studies suggest that EEG engagement reflects information-gathering, visual processing, and allocation of attention. EEG workload increases with increasing working memory load and during problem solving, integration of information, analytical reasoning, and may be more reflective of executive functions. Inspection of EEG on a second-by-second timescale revealed associations between workload and engagement levels when aligned with specific task events providing preliminary evidence that second-by-second classifications reflect parameters of task performance.
Drivers' fatigue has been implicated as a causal factor in many accidents. The development of human cognitive state monitoring system for the drivers to prevent accidents behind the steering wheel has become a major focus in the field of safety driving. It requires a technique that can continuously monitor and estimate the alertness level of drivers. The difficulties in developing such a system are lack of significant index for detecting drowsiness and the interference of the complicated noise in a realistic and dynamic driving environment. An adaptive alertness estimation methodology based on electroencephalogram, power spectrum analysis, independent component analysis (ICA), and fuzzy neural network (FNNs) models is proposed in this paper for continuously monitoring driver's drowsiness level with concurrent changes in the alertness level. A novel adaptive feature selection mechanism is developed for automatically selecting effective frequency bands of ICA components for realizing an on-line alertness monitoring system based on the correlation analysis between the time-frequency power spectra of ICA components and the driving errors defined as the deviation between the center of the vehicle and the cruising lane in the virtual-reality driving environment. The mechanism also provides effective and efficient features that can be fed into ICA-mixture-model-based self-constructing FNN to indirectly estimate driver's drowsiness level expressed by approximately and predicting the driving error
Deep neural nets with a large number of parameters are very powerful machine learning systems. However, overfitting is a serious problem in such networks. Large networks are also slow to use, making it difficult to deal with overfitting by combining the predictions of many different large neural nets at test time. Dropout is a technique for addressing this problem. The key idea is to randomly drop units (along with their connections) from the neural network during training. This prevents units from co-adapting too much. During training, dropout samples from an exponential number of different "thinned" networks. At test time, it is easy to approximate the effect of averaging the predictions of all these thinned networks by simply using a single unthinned network that has smaller weights. This significantly reduces overfitting and gives major improvements over other regularization methods. We show that dropout improves the performance of neural networks on supervised learning tasks in vision, speech recognition, document classification and computational biology, obtaining state-of-the-art results on many benchmark data sets. © 2014 Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever and Ruslan Salakhutdinov.
Function estimation/approximation is viewed from the perspective of numerical optimization iti function space, rather than parameter space. A connection is made between stagewise additive expansions and steepest-descent minimization. A general gradient descent "boosting" paradigm is developed for additive expansions based on any fitting criterion. Specific algorithms are presented for least-squares, least absolute deviation, and Huber-M loss functions for regression, and multiclass logistic likelihood for classification. Special enhancements are derived for the particular case where the individual additive components are regression trees, and tools for interpreting such "TreeBoost" models are presented. Gradient boosting of regression trees produces competitives highly robust, interpretable procedures for both regression and classification, especially appropriate for mining less than clean data. Connections between this approach and the boosting methods of Freund and Shapire and Friedman, Hastie and Tibshirani are discussed.
The biocybernetic loop describes the data processing protocol at the heart of all physiological computing systems. The loop also encompasses the goals of the system design with respect to the anticipated impact of the adaptation on user behaviour. There are numerous challenges facing the designer of a biocybernetic loop in terms of measurement, data processing and adaptive design. These challenges are multidisciplinary in nature spanning psychology and computer science. This paper is concerned with the design process of the biocybernetic loop. A number of criteria for an effective loop are described followed by a six-stage design cycle. The challenges faced by the designer at each stage of the design process are exemplified with reference to a case study where EEG data were used to adapt a computer game.
In this paper, we analyze the ECG and EEG signals by using a set of fuzzy systems developed for signal processing and try to see if one can quantify the mental workload based on the trend of EEG signal changes and the variation of pulse width from ECG signal. The signal data were taken from nuclear power plant operators while they perform the turbine operations. First, we apply a fuzzy system to extract the heart beat intervals from the ECG and a smoothing algorithm for the variation of the resulting heart beat intervals to obtain a trend variable. Next, we apply the same smoothing algorithm to the α-band frequencies and to the θ-band frequencies to get their trend curves. Finally, the three trend variables; the variation of the heart beat intervals, the α-band power, and the θ-band power are combined by a pair of fuzzy systems to estimate the mental workload during nuclear power plant operations. The results of applying our algorithm to three different data sets are included, along with a comparison between these results and the results obtained by applying a linear combination of the three variables. Compared results show that the `mental workload' computed by fuzzy systems with nonlinear rules reflects the changes more clearly than the ones computed by linear functions.
The results of a multi-year research program to identify the factors associated with variations in subjective workload within and between different types of tasks are reviewed. Subjective evaluations of 10 workload-related factors were obtained from 16 different experiments. The experimental tasks included simple cognitive and manual control tasks, complex laboratory and supervisory control tasks, and aircraft simulation. Task-, behavior-, and subject-related correlates of subjective workload experiences varied as a function of difficulty manipulations within experiments, different sources of workload between experiments, and individual differences in workload definition. A multi-dimensional rating scale is proposed in which information about the magnitude and sources of six workload-related factors are combined to derive a sensitive and reliable estimate of workload.
Questionnaire and interview assessment can provide reliable data on attitudes and self-perceptions on emotion, and experimental laboratory assessment can examine functional relations between stimuli and reactions under controlled conditions. On the other hand, ambulatory assessment is less constrained and provides naturalistic data on emotion in daily life, with the potential to (1) assure external validity of laboratory findings, (2) provide normative data on prevalence, quality and intensity of real-life emotion and associated processes, (3) characterize previously unidentified emotional phenomena, and (4) model real-life stimuli for representative laboratory research design. Technological innovations now allow for detailed ambulatory study of emotion across domains of subjective experience, overt behavior and physiology. However, methodological challenges abound that may compromise attempts to characterize biobehavioral aspects of emotion in the real world. For example, emotional effects can be masked by social engagement, mental and physical workloads, as well as by food intake and circadian and quasi-random variation in metabolic activity. The complexity of data streams and multitude of factors that influence them require a high degree of context specification for meaningful data interpretation. We consider possible solutions to typical and often overlooked issues related to ambulatory emotion research, including aspects of study design decisions, recording devices and channels, electronic diary implementation, and data analysis.
A biocybernetic system has been developed as a method to evaluate automated flight deck concepts for compatibility with human capabilities. A biocybernetic loop is formed by adjusting the mode of operation of a task set (e.g., manual/automated mix) based on electroencephalographic (EEG) signals reflecting an operator's engagement in the task set. A critical issue for the loop operation is the selection of features of the EEG to provide an index of engagement upon which to base decisions to adjust task mode. Subjects were run in the closed-loop feedback configuration under four candidate and three experimental control definitions of an engagement index. The temporal patterning of system mode switching was observed for both positive and negative feedback of the index. The indices were judged on the basis of their relative strength in exhibiting expected feedback control system phenomena (stable operation under negative feedback and unstable operation under positive feedback). Of the candidate indices evaluated in this study, an index constructed according to the formula, beta power/(alpha power + theta power), reflected task engagement best.
The functional state of the human operator is critical to optimal system performance. Degraded states of operator functioning can lead to errors and overall suboptimal system performance. Accurate assessment of operator functional state is crucial to the successful implementation of an adaptive aiding system. One method of determining operators' functional state is by monitoring their physiology. In the present study, artificial neural networks using physiological signals were used to continuously monitor, in real time, the functional state of 7 participants while they performed the Multi-Attribute Task Battery with two levels of task difficulty. Six channels of brain electrical activity and eye, heart and respiration measures were evaluated on line. The accuracy of the classifier was determined to test its utility as an on-line measure of operator state. The mean classification accuracies were 85%, 82%, and 86% for the baseline, low task difficulty, and high task difficulty conditions, respectively. The high levels of accuracy suggest that these procedures can be used to provide accurate estimates of operator functional state that can be used to provide adaptive aiding. The relative contribution of each of the 43 psychophysiological features was also determined. Actual or potential applications of this research include test and evaluation and adaptive aiding implementation.
We investigated whether the 0.1-Hz component of heart rate variability (HRV) allows one to discriminate among levels of mental work stress induced by different types of tasks (diagnosticity) as well as among those induced by different levels of difficulty (sensitivity). Our 14 participants were presented 14 tasks of the Advisory Group for Aerospace Research and Development Standardized Tests for Research with Environmental Stressors battery in a repeated-measures design. Sufficient sensitivity was obtained for a discrimination between work and rest, but we found no support for a more fine-grained sensitivity. Concerning diagnosticity, only the grammatical reasoning task could be discriminated from all other tasks, indicating for this task a level of mental strain comparable to rest, which was in contrast with the results both for perceived difficulty and performance. We propose that HRV is an indicator for time pressure or emotional strain, not for mental workload, given that it seems to allow discrimination between tasks with and without pacing. Application of this research argues against using HRV as a measure of mental and especially cognitive workload, particularly where system safety or occupational risks may be at stake (e.g., when evaluating operator tasks or interface design in control room operations).
Increases in attentional effort are defined as the motivated activation of attentional systems in response to detrimental challenges on attentional performance, such as the presentation of distractors, prolonged time-on-task, changing target stimulus characteristics and stimulus presentation parameters, circadian phase shifts, stress or sickness. Increases in attentional effort are motivated by the expected performance outcome; in the absence of such motivation, attentional performance continues to decline or may cease altogether. The beneficial effects of increased attentional effort are due in part to the activation of top-down mechanisms that act to optimize input detection and processing, thereby stabilizing or recovering attentional performance in response to challenges. Following a description of the psychological construct "attentional effort", evidence is reviewed indicating that increases in the activity of cortical cholinergic inputs represent a major component of the neuronal circuitry mediating increases in attentional effort. A neuronal model describes how error detection and reward loss, indicating declining performance, are integrated with motivational mechanisms on the basis of neuronal circuits between prefrontal/anterior cingulate and mesolimbic regions. The cortical cholinergic input system is activated by projections of mesolimbic structures to the basal forebrain cholinergic system. In prefrontal regions, increases in cholinergic activity are hypothesized to contribute to the activation of the anterior attention system and associated executive functions, particularly the top-down optimization of input processing in sensory regions. Moreover, and influenced in part by prefrontal projections to the basal forebrain, increases in cholinergic activity in sensory and other posterior cortical regions contribute directly to the modification of receptive field properties or the suppression of contextual information and, therefore, to the mediation of top-down effects. The definition of attentional effort as a cognitive incentive, and the description of a neuronal circuitry model that integrates brain systems involved in performance monitoring, the processing of incentives, activation of attention systems and modulation of input functions, suggest that 'attentional effort' represents a viable construct for cognitive neuroscience research.
It is well known that microgravity results in various physiological alterations, for example, head-ward fluid shifts which can impede physiological adaptation. Other factors that may affect crew operational efficiency include disruption of sleep-wake cycles, high workload, isolation, confinement, stress, and fatigue. From an operational perspective, it is difficult to predict which individuals will be most or least affected in this unique environment given that most astronauts are first-time flyers. During future lunar and Mars missions space crews will include both men and women of multi-national origins, different professional backgrounds, and various states of physical condition. Therefore, new methods or technologies are needed to monitor and predict astronaut performance and health, and to evaluate the effects of various countermeasures on crew during long-duration missions. Herein we describe the development and validation of a new methodology for assessing the deleterious effects of spaceflight on crew health and performance. We reviewed several studies conducted in both laboratory and operational environments with men and women ranging in age between 18 to 50 yr. The studies included the following: soldiers performing command and control functions during mobile operations in enclosed armored vehicles; subjects participating in laboratory tests of an anti-motion sickness medication; subjects exposed to chronic hypergravity aboard a centrifuge; and subject responses to 36-h of sleep deprivation. Physiological measurements, performance metrics, and subjective self-reports were collected in each study. The results demonstrate that multivariate converging indicators provide a significantly more reliable method for assessing environmental effects on performance and health than any single indicator.
We have investigated the quantitative effects of a number of common elements of QRS detection rules using the MIT/BIH arrhythmia database. A previously developed linear and nonlinear filtering scheme was used to provide input to the QRS detector decision section. We used the filtering to preprocess the database. This yielded a set of event vectors produced from QRS complexes and noise. After this preprocessing, we tested different decision rules on the event vectors. This step was carried out at processing speeds up to 100 times faster than real time. The role of the decision rule section is to discriminate the QRS events from the noise events. We started by optimizing a simple decision rule. Then we developed a progressively more complex decision process for QRS detection by adding new detection rules. We implemented and tested a final real-time QRS detection algorithm, using the optimized decision rule process. The resulting QRS detection algorithm has a sensitivity of 99.69 percent and positive predictivity of 99.77 percent when evaluated with the MIT/BIH arrhythmia database.
Improving Engagement Assessment by Model Individualization and Deep Learning
- F Li
Li, F., "Improving Engagement Assessment by Model Individualization and Deep Learning," Dissertation, Old Dominion
University, 2015.
A Systematic Approach for Real-Time Operator Functional State Assessment
- G Zhang
- W Wang
- A Pepe
- R Xu
Zhang, G., Wang, W., Pepe, A., and Xu, R., "A Systematic Approach for Real-Time Operator Functional State
Assessment," Proceedings of the MODSIM World conference, Hampton, VA., Oct.13-15, 2010.
The Multi-Attribute Task Battery II (MATB-II) Software for Human Performance and Workload Research: A User's Guide
- Y Santiago-Espada
- R R Myer
- K A Latorella
- J R Comstock
Santiago-Espada, Y., Myer, R. R., Latorella, K. A., & Comstock, J. R., "The Multi-Attribute Task Battery II (MATB-II)
Software for Human Performance and Workload Research: A User's Guide," NASA, Langley Research Center, Hampton:
NASA/TM-2011-217164, L-20031, NF1676L-12800, 2011.