Article

The effects of dynamic workload and experience on commercially available EEG cognitive state metrics in a high-fidelity air traffic control environment

Authors:
To read the full-text of this research, you can request a copy directly from the authors.

Abstract

The current study evaluated the validity of commercially available electroencephalography (EEG) cognitive state metrics of workload and engagement in differentially experienced air traffic control (ATC) students. EEG and pupil diameter recordings were collected from 47 ATC students (27 more experienced and 20 less experienced) during a high-fidelity, variable workload approach-control scenario. Scenario workload was manipulated by increasing the number of aircraft released and the presence of a divided attention task. Results showed that scenario performance significantly degraded with increased aircraft and the presence of the divided attention task. No scenario performance differences were found between experience groups. The EEG engagement metric significantly differed between experience groups, with less experienced controllers exhibiting higher engagement than more experienced controllers. The EEG workload metric and pupil diameter were sensitive to workload manipulations but did not differentiate experience groups. Commercially available EEG cognitive state metrics may be a viable tool for enhancing ATC training.

No full-text available

Request Full-text Paper PDF

To read the full-text of this research,
you can request a copy directly from the authors.

... color or symbol coding influence pupil response. Similarly, Bernhardt et al. (2019) suggest that pupil diameter is likely to be sensitive towards increasing visual load. Influences of polarity due to screen illumination on pupil dilation were found by Dobres et al. (2017). ...
... For this reason, we conducted a second experiment focusing on screen polarity and presentation of information in two trials. For instance, Bernhardt et al. (2019) suggest that pupil diameter is likely to be sensitive towards increasing visual load. We hypothesize that screen polarity influences pupil diameter but not the scaled ICA (H03). ...
... Factor levels of information were: none (1), fixation cross (2), low density (3), high density (4), centred circle (5), and moving circle (6). Density was tested with a small number of nine centred dots (3 × 3 rows) versus a high number of dots (9 × 13 rows) spread over the entire screen, according to findings of Bernhardt et al. (2019). Regarding movement, a stationary filled circle was compared to a circle moving with constant speed. ...
Article
Full-text available
Various researchers have proposed pupillometric indicators to assess a person's cognitive strain. However, to distinguish the variation of pupil light response from psychosensory pupil response in experimental field conditions is a challenge. The Index of Cognitive Activity (ICA) addresses this problem by wavelet separation. This research investigates the ICA's sensitivity for multiple level task-evoked cognitive activity and visual influences concerning informational work tasks. Objective and subjective measures assessed cognitive strain of participants (N = 22) during various tasks. In a first experiment, mental arithmetic tasks were used to induce different levels of cognitive activity. In a second experiment, influences of screen polarity and presentation of information were investigated (N = 18). The results indicate that eye metrics are rarely sensitive to slight variations in task difficulty. Moreover, the ICA is likely to be sensitive towards constant screen illumination and shows tendencies regarding changes in displayed information. Possible ramifications for the objective assessment of cognitive strain are discussed.
... Twelve studies did not report the gender of the participants. Two studies investigated the differences between expertise of the participants (Bernhardt et al. 2019;Hyun et al. 2006). ...
... Therefore, he suggested that more research is needed before blink rate could be used to measure mental workload. Sao Paulo, Brazil, April 5 -8, 2021 Pupil diameter is correlated to workload Rodriguez et al.(2015) and found to be larger in higher workload levels (Bernhardt et al. 2019;Ahlstrom and Friedman 2006;Martin et al. 2011;Truschzinski 2017). Pupil diameter differed significantly between scenarios Martin et al. (2011);Truschzinski (2017) but not between expertise (Bernhardt et al. 2019). ...
... Sao Paulo, Brazil, April 5 -8, 2021 Pupil diameter is correlated to workload Rodriguez et al.(2015) and found to be larger in higher workload levels (Bernhardt et al. 2019;Ahlstrom and Friedman 2006;Martin et al. 2011;Truschzinski 2017). Pupil diameter differed significantly between scenarios Martin et al. (2011);Truschzinski (2017) but not between expertise (Bernhardt et al. 2019). Pupil diameter was more sensitive to workload from visual rather than EEG workload metric (Bernhardt et al. 2019). ...
Conference Paper
Full-text available
Air Traffic Control (ATC) is a complex cognitive task. The complexity may cause the workload of air traffic controllers (ATCs) to increase. Increased workload could degrade operator performance, which further affect safety and functioning of the system. Numerous studies in evaluating mental workload in various tasks used physiological and biochemical measures, including studies in ATCs. The use of physiological measures in those studies provides unique information of the operator's condition. This systematic review summarizes literatures on the measurement of the mental workload in ATCs using physiological and biochemical measures. The conducted systematic search come up with thirty-four studies to include for analysis. Physiological measures are categorized into cardiovascular, ocular, brain, respiration, and skin measures. Biochemical measures consisted of cortisol and salivary immunoglobulin (sIgA). The literature review covers various studies either in simulation environment or real working environment, all with different conditions and task scenarios. Even though this review specifically focuses on mental workload of ATCs, in general, the result of physiological measures still differs between broad studies. The difference might be influenced by the study task loads, difficulty levels, and the participants' characteristics.
... As an ergonomic concept, MWL (also referred to as cognitive workload) is "the dynamic relationships between the resources that are needed to carry out a task and the ability of the operator to adequately supply those resources" [3]. MWL can also be defined as "the amount of cognitive resources being expended at a given point of time" [4]. ...
... A number of studies have provided certain features of MWL variations in the frontal [13][14][15] and parietal [16,17] regions. Moreover, changes in task demand have been presented to yield changes in spectral band frequencies in diverse domains including ergonomics in general [17][18][19] or ATC in particular [3,20,21]. For example, decreased alpha activity was observed with increasing cognitive demands in simulated ATC task by Brookings et al. [20]. ...
... Circadian rhythms are of extreme significance when addressing on-the-job safety and may be utilized to reduce work-related risks [25]. However, evaluation of human operator's cognitive state in real-world environments during the day has been overlooked in the previous literature [3,26,27]. ...
Article
Full-text available
The aim of this study was to investigate the effects of mental workload (MWL) and time of day on cognitive performance and electroencephalographic (EEG) parameters of air traffic controllers. EEG signals recorded while 20 professional air traffic controllers performed cognitive tasks [A-X Continuous Performance Test (AX-CPT) and 3-back working memory task] after they were exposed to two levels of task difficulty (high and low MWL) in the morning and afternoon. Significant decreases in cognitive performance were found when the levels of task difficulty increased in both tasks. The results confirmed the sensitivity of the theta and beta activities to levels of task difficulty in the 3-back task, while they were not affected in the AX-CPT. Theta and beta activities were influenced by time of day in the AX-CPT. The findings provide guidance for application of changes in EEG parameters when MWL level is manipulated during the day that could be implemented in future for the development of real-time monitoring systems to improve aviation safety.
... This makes it an ideal media to induce, study and understand Affective and Cognitive States (ACS). Affective States (AS) refer to states such as emotions, mood, and feelings [132], and Cognitive States (CS) refer to states like cognitive load or mental workload, which influence how information is processed (e.g., reasoning, deliberation, planning) [22]. While there are some distinctions in their definitions and models, AS and CS are interwoven [78]. ...
... They can be manipulated, for instance, by stories, narrative content, and preferences, and they influence decision making. On the other hand, CS derive from cognitive resources, cognitive skills, and influence how information is processed [22]. They can be manipulated, for instance, by the intrinsic difficulty of a task, the number of distractors, and the instructions presentation format. ...
Article
In Virtual Reality (VR), users can be immersed in emotionally intense and cognitively engaging experiences. Yet, despite strong interest from scholars and a large amount of work associating VR and Affective and Cognitive States (ACS), there is a clear lack of structured and systematic form in which this research can be classified. We define "Affective and Cognitive VR" to relate to works which (1) induce ACS, (2) recognize ACS, or (3) exploit ACS by adapting virtual environments based on ACS measures. This survey clarifies the different models of ACS, presents the methods for measuring them with their respective advantages and drawbacks in VR, and showcases Affective and Cognitive VR studies done in an immersive virtual environment (IVE) in a non-clinical context. Our article covers the main research lines in Affective and Cognitive VR. We provide a comprehensive list of references with the analysis of 63 research articles and summarize future works directions.
... Similarly, it was often associated to other states or notions like cognitive load, mental effort, and cognitive performances. We can separate these notions into two categories: Cognitive States (CS), which influence or are consequences of how information is processed (e.g., reasoning, deliberation, planning) [28] such as MW or cognitive performances, and Affective States (AS), which refer to states like emotions, mood, feelings, and personality traits [258]. While there are some distinctions in their definitions and models, CS and AS have various similarities as multicomponent constructs [160,215,258] and in the methods to measure them [204]. ...
Thesis
Full-text available
Malgré l’émergence rapide des systèmes automatisés et le développement de l’informatique affective, très peu d’études considèrent la charge mentale de travail dans la conception de scénarios de formation en RV. Cette thèse a pour objectif de contribuer au développement des systèmes adaptatifs en RV, basés sur la charge mentale de travail des utilisateurs. Nous proposons 3 axes de recherches : induction, reconnaissance, et exploitation de la charge mentale de travail en RV, ainsi qu'une définition de la « Réalité Virtuelle Affective et Cognitive ». Dans un premier temps, nous étudierons l’impact du port de casque de RV sur l’effort mental des utilisateurs. De plus, l’influence potentielle de la marche et de l’effet d’accommodation en RV seront analysés. Puis, nous proposerons une approche méthodologique pour introduire l’évaluation de la charge mentale de travail dans la conception de scénarios de formation en RV. Cette méthodologie permettra notamment de moduler le niveau de charge mentale de travail des utilisateurs au cours du temps. Des études utilisateurs seront menées dans un simulateur de vol en RV afin d'évaluer cette approche. Finalement, nous proposerons une solution tout-en-un afin d'estimer la charge mentale de travail des utilisateurs en temps-réel en utilisant des capteurs intégrés aux casques de RV. Cette configuration sera comparée aux systèmes plus répandus dans le commerce vis-à-vis des performances de prédiction. Les influence du types de mesures, des capteurs, et des méthodes de normalisation des signaux seront également analysées.
... As with other comprehension tasks, the goal of understanding static or dynamic images and enable learning is equivalent to an internal cognitive mental model, i.e., an image visualization medium that accurately represents an entity (an object or event depicted by a dynamic or static image). In cognitive processes, the process of selecting, organizing, and integrating representative information from different forms and constructing continuous mental cognitive models often imposes a high cognitive load on the learners [37]. The related cognitive load theory also provides additional concepts for understanding and analyzing cognitive learning using images. ...
Article
Full-text available
This paper discusses the influence of dynamic images and traditional static images on user perception in web interface visualizations of products. First, thirty graduate students in industrial design participated in an eye-tracking experiment, performing visual imagery (VI) tasks of the product images with different presentation formats and durations. The results of eye movement experiment show that the visual cognitive effect was better for the dynamic images than the static image, and the efficiency of visual search was improved. However, the emotional experience of viewing dynamic images was substantially affected by the presentation time. Secondly, there were significant differences in the cognitive level and emotional experience of the users between the dynamic images with different presentation times. The optimal perception experience was observed at a presentation time of 9000 ms, indicating that the subjective responses of the users’ questionnaire survey did not represent the actual cognitive needs of the users. This study provides a scientific basis for product designers to achieve an improved browsing experience of their products.
... Due to its properties, the method is expected to support solving such important problems as participants' comparison (Stanton, 2005), reference point absence, different workload definition, tasks similarity (Wickens, 2002), quantitative MWL definition (Wang et al., 2020), measurement continuity (Miller, 2001), metric sensitivity (Bernhardt et al., 2019), the "Multiple resource the-ory" further research (Bommer, 2016), dynamic MWL assessment (Iqbal et al., 2020;Kostenko et al., 2016), and workload redline determination (De Waard, 1996). It also contributes to continuous MWL profiles' construction as an alternative to analytical methods (Rusnock & Borghetti, 2018) and should be tested for such an urgent topic as MWL prediction (Bommer, 2016;Kirwan et al., 1997;Vidulich et al., 1991;Wickens, 2002). ...
Article
Full-text available
Mental workload is a well-known concept with a long development history. It can be used to examine students’ attitudes at the end of the educational process and compare them in groups or separately. However, building a continuous workload profile across the range of task complexity increase is still an urgent issue. All four groups of methods used to define mental workload have such flaws for the workload profile construction process as significant time requirements, single value processing and post-processing of the received results. Only one of them can be used without modifications to construct the operator’s attitude chart (profile) regarding the workload range and it doesn’t operate with more reliable absolute values. To resolve this problem, a special workload assessment grid was developed, considering the advantages of a subjective group of methods and seven core characteristics. The reasoning for grid axes choice, threshold values, and question formulation were provided. Statistics were calculated for the full sample, different grades, and educational institutions. Comparison of the received responses with referential values, cross-comparison between institutions and different grades were performed. The results contribute to such important aspects of workload, as redlines, workload profiling, and operator’s comparison.
... An interesting EEG study conducted under real flight conditions [9] exposed that a reduction of alpha and theta frequencies bands can reveal the neural signature of intentional deafness (for example, inadvertent lack of auditory stimulation). A recent study [4] exploring air traffic controllers has calculated an EEG-based workload index, which can distinguish the task workload requirements of analyzing forebrain functions, however, they did not explore alpha and theta involvement with workload changes. ...
Article
Full-text available
Increasing workload is a central notion in human factors research that can decrease the performance and yield accidents. Thus, it is crucial to understand the impact of different internal operator’s factors including eye movements, memory and audio-visual integration. Here, we explored the relationship between cognitive workload (low vs. high) and eye movements (saccades, fixations and smooth pursuit). The task difficulty was induced by auditory noise, arithmetical count and working memory load. We estimated cognitive workload using EOG and EEG-based mental state monitoring. One novelty consists in recording the EOG around the ears (alternative EOG) and around the eyes (conventional EOG). The number of blinks and saccades amplitude increased along with the difficulty increase (p ≤ 0.05). We found significant correlations between EOG and EEG (theta/alpha ratio) and between conventional and alternative EOG signal. The increase in cognitive load may disturb the coding and maintenance of related visual information. Alternative EOG metrics could be a valuable tool for detecting workload.
... Recent studies exploring Air Traffic Controllers (Bernhardt et al., 2019) have computed an EEG-based workload index that could differentiate between task workload requirements exploring front-parietal brain function. Yet, EEG exploration has achieved traction in aviation and space operations, current studies face challenges related to the intrusive and bulky nature of the equipment (Caldwell et al., 2002), the discomfort of long preparation time, and dependence on gel electrodes (e.g., wet electrodes). ...
Article
Full-text available
Electro-encephalography (EEG) and electro-oculography (EOG) are methods of electrophysiological monitoring that have potentially fruitful applications in neuroscience, clinical exploration, the aeronautical industry, and other sectors. These methods are often the most straightforward way of evaluating brain oscillations and eye movements, as they use standard laboratory or mobile techniques. This review describes the potential of EEG and EOG systems and the application of these methods in aeronautics. For example, EEG and EOG signals can be used to design brain-computer interfaces (BCI) and to interpret brain activity, such as monitoring the mental state of a pilot in determining their workload. The main objectives of this review are to, (i) offer an in-depth review of literature on the basics of EEG and EOG and their application in aeronautics; (ii) to explore the methodology and trends of research in combined EEG-EOG studies over the last decade; and (iii) to provide methodological guidelines for beginners and experts when applying these methods in environments outside the laboratory, with a particular focus on human factors and aeronautics. The study used databases from scientific, clinical, and neural engineering fields. The review first introduces the characteristics and the application of both EEG and EOG in aeronautics, undertaking a large review of relevant literature, from early to more recent studies. We then built a novel taxonomy model that includes 150 combined EEG-EOG papers published in peer-reviewed scientific journals and conferences from January 2010 to March 2020. Several data elements were reviewed for each study (e.g., pre-processing, extracted features and performance metrics), which were then examined to uncover trends in aeronautics and summarize interesting methods from this important body of literature. Finally, the review considers the advantages and limitations of these methods as well as future challenges.
... This study included high engagement (Berka et al., 2007). These commercially available metrics have been shown to be valuable in both marketing and human performance areas (Bernhardt et al., 2019). ...
Article
Full-text available
Neuroforecasting predicts population-wide choices based on neural data of individuals and can be used for example in neuromarketing to estimate campaign successes. To deliver true value, the brain activity metrics should deliver predictive value above and beyond traditional stated preferences. Evidence from movie trailer research has proposed neural synchrony, which compares the similarity of brain responses across participants and has shown to be a promising tool in neuroforecasting for movie popularity. The music industry might also benefit from these increasingly accurate success predictors, but only one study has been forecasting music popularity, using fMRI measures. Current research validates the strength of neural synchrony as a predictive measure for popularity of music, making use of electroencephalogram (EEG) to capture moment-to-moment neural similarity between respondents while they listen to music. Neural synchrony is demonstrated to be a significant predictor for public appreciation on Spotify three weeks and ten months after the release of the albums, especially when combined with the release of a single. On an individual level, other brain measures were shown to relate to individual subjective likeability ratings, including Frontal Alpha Asymmetry and engagement when combined with the factors artist and single release. Our results show the predictive value of brain activity measures outperforms stated preferences. Especially, neural synchrony carries high predictive value for the popularity on Spotify, providing the music industry with an essential asset for efficient decision making and investments, in addition to other practical implications that include neuromarketing and advertising industries.
... Its purpose is not only to study the structure and function of the brain, which is the domain of neuroscience but also to do so in the context of individual perceptions and behaviors in the workplaces, homes, transportation, and other everyday environments. Neuroergonomics involves interpreting real-life situations and activities performed in natural real-world settings, studying the neural basis of cognitive and motor functions by measuring electromagnetic or hemodynamics activity in the brain (Bernhardt et al., 2019;Nuamah et al., 2020;Parasuraman and Rizzo, 2008). ...
Article
This study aims to evaluate the effect of workstation type on the neural and vascular networks of the prefrontal cortex (PFC) underlying the cognitive activity involved during mental stress. Workstation design has been reported to affect the physical and mental health of employees. However, while the functional effects of ergonomic workstations have been documented, there is little research on the influence of workstation design on the executive function of the brain. In this study, 23 healthy volunteers in ergonomic and non-ergonomic workstations completed the Montreal imaging stress task, while their brain activity was recorded using the synchronized measurement of electroencephalography and functional near-infrared spectroscopy. The results revealed desynchronization in alpha rhythms and oxygenated hemoglobin, as well as decreased functional connectivity in the PFC networks at the non-ergonomic workstations. Additionally, a significant increase in salivary alpha-amylase activity was observed in all participants at the non-ergonomic workstations, confirming the presence of induced stress. These findings suggest that workstation design can significantly impact cognitive functioning and human capabilities at work. Therefore, the use of functional neuroimaging in workplace design can provide critical information on the causes of workplace-related stress.
... Although EEG based methods have been used in domains such as air traffic control (e.g. Aricò et al., 2019;Bernhardt et al., 2019;Radüntz et al., 2021), aviation (e.g. Blanco et al., 2018;Hebbar et al., 2021;Lee et al., 2020), construction (Chen et al., , 2017, manufacturing (Argyle et al., 2021), power plant operation (Reinerman-Jones et al., 2016), remote operation (Durantin et al., 2014;Rojas et al., 2020), ship handling and software development (Fritz et al., 2014), field studies have mostly been limited to the field of aviation (Dehais et al., 2019;Noel et al., 2005;Wilson, 2002). ...
Article
Full-text available
Although the objective assessment of mental workload has been a focus of human factors research, few studies have investigated stakeholders' attitudes towards its implementation in real workplaces. The present study addresses this research gap by surveying N = 702 managers in three European countries (Germany, United Kingdom, Spain) about their expectations and concerns regarding sensor-based monitoring of employee mental workload. The data confirm the relevance of expectations regarding improvements of workplace design and employee well-being, as well as concerns about restrictions of employees' privacy and sovereignty, for the implementation of workload monitoring. Furthermore, Bayesian regression models show that the examined expectations have a substantial positive association with managers’ willingness to support workload monitoring in their company. Privacy concerns are identified as a significant barrier to the acceptance of workload monitoring, both in terms of their prevalence among managers and their strong negative relationship with monitoring support.
... Another study identified that neurophysiological measures are more sensitive, in comparison to eye-tracking metrics, when discriminating the impact that task difficulty and complexity have in a high-fidelity driving scenario (Flumeri et al. 2018). In contrast, Bernhardt et al. (2019), reached the conclusion that EEG workload and pupil dilation are not correlated to each other, a finding also shared by Matthews et al. (2015), and hypothesized that it may be that pupil dilation may not be a direct measure of cognitive workload. Regardless, fNIRS and eye-tracking are portable, safe, affordable and unobtrusive systems, which can present a potential multi-sensor approach to a multidimensional evaluation of an ATC specialist's cognitive workload. ...
Conference Paper
Full-text available
Air Traffic Control (ATC) specialists work in an environment where the proficient interaction between humans and computer systems is crucial to provide a safe and efficient flow of traffic. The complexity of this system may increase due to planned changes in operator roles and a projected rise in traffic volume. This increase, over an already highly complex system, will exacerbate the mental workload placed on the operator. The emergence of wearable sensors that measure physiological signals enables us to monitor the mental workload in real time without interfering in operational activity. However, the use of a single sensor approach may not provide a comprehensive assessment of cognitive workload while executing a complex task. Therefore, this study implemented a multimodal approach by using two sensors, namely functional near infrared spectroscopy (fNIRS) and eye-tracking, to evaluate the cognitive workload changes experienced by an ATC specialist. Three retired tower controllers with over 20 years of experience, underwent three sessions of experimentation where each individual fulfilled one of the following roles - observer, a Local controller or a Ground controller. During each iteration, the fNIRS and eye tracking sensors were attached to the Local controller while they commanded aircraft through verbal clearances. The task difficulty and complexity were quantified by the number of aircraft and clearances given, respectively. The number of aircraft displayed on the screen increased across time and was positively correlated with oxygenation measures assessed by the fNIRS signals of both the right and left prefrontal cortex. On the other hand, the number of fixations was positively correlated with the number of clearances. These results suggest fNIRS and eye-tracking measures are sensitive to changes in cognitive workload, and indicates that they may be amenable to complement each other for the assessment of the multidimensionality of cognitive workload induced by task difficulty and complexity.
... The cognitive workload involves studying the dynamic relationships between the resources necessary to accomplish a task and the ability of the brain to adequately supply those resources [1], [2]. It is defined as the total mental activity imposed on a subject's cognitive system during a particular period of work. ...
Article
Full-text available
Mental workload has been widely estimated based on electroencephalography (EEG) in the frequency domain. However, simple frequency features are not entirely accurate indicators of the cognitive load because surface EEG signals are weak, nonstationary and randomness. We hypothesize that graph methods, which analyse the relationship between each point and other points of the EEG signals, may provide a more precise identification of the mental load. To investigate this hypothesis, we aim to identify the optimum graph features from 14 channel EEG recordings (sampling rate=128 Hz) in order to detect the high cognitive load related to multitasking. Three graph features: mean degree d, clustering coefficient c, and degree distribution p(k), are extracted from 48 subjects EEG records. Each experimental subject conducts two tasks: without tasks and with a simultaneous capacity task, respectively. After the experiment is completed, the feeling of the subject with the cognitive load tags in three types: low load, medium load, and heavy load. The optimal features of these three levels of the subject sensation and two types of cognitive load in different tasks are selected on the basis of statistical analysis. Then all graph features are forwarded into a support vector machine (SVM) and a decision tree to conduct objective scoring classification and a three subjective rating classification, respectively. Based on the present results,channels O2, T8, FC6, F8, and AF4 are considered optimal for a more efficiently estimation of the cognitive load. c associated with F8 and T8 during low cognitive load is significantly lower than those associated with high cognitive load (p<0.001). Using three graph features, the accuracy of identifying two types of mental load is 89.6%. Current findings suggest that the mental workload associated with multi-tasks can be accurately assessed using the graph approaches to EEG data.
... The number of blinks also reduced considerably with the increasing workload in both tasks. Pupil size is a reliable measure of workload (Beatty, 1982, Mandrick et al., 2016 as it dilates with increasing workload (Batmaz and Ozturk, 2008, Kosch et al., 2018, Truschzinski et al., 2018, Bernhardt et al., 2019, Kearney et al., 2019, Marinescu et al., 2018, Wanyan et al., 2014. Recarte et al., 2008 show that blink inhibition happens in higher workload conditions and so, the blink rate is inversely correlated with the attentional levels and workload experienced by the operator (Veltman and Gaillard, 1996, Brookings et al., 1996, Wilson, 2002, Borghini et al., 2014, Widyanti et al., 2017, Wanyan et al., 2018. ...
Preprint
Full-text available
Objective We have designed tracking and collision prediction tasks to elucidate the differences in the physiological response to the workload variations in basic ATC tasks to untangle the impact of workload variations experienced by operators working in a complex ATC environment. Background Even though several factors influence the complexity of ATC tasks, keeping track of the aircraft and preventing collision are the most crucial. Methods Physiological measures, such as electroencephalogram (EEG), eye activity, and heart rate variability (HRV) data, were recorded from 24 participants performing tracking and collision prediction tasks with three levels of difficulty. Results The neurometrics of workload variations in the tracking and collision prediction tasks were markedly distinct, indicating that neurometrics can provide insights on the type of mental workload. The pupil size, number of blinks and HRV metric, root mean square of successive difference (RMSSD), varied significantly with the mental workload in both these tasks in a similar manner. Conclusion Our findings indicate that variations in task load are sensitively reflected in physiological signals, such as EEG, eye activity and HRV, in these basic ATC-related tasks. Application These findings have applicability to the design of future mental workload adaptive systems that integrate neurometrics in deciding not just ‘when’ but also ‘what’ to adapt. Our study provides compelling evidence in the viability of developing intelligent closed-loop mental workload adaptive systems that ensure efficiency and safety in ATC and beyond. Précis This article identifies the physiological correlates of mental workload variation in basic ATC tasks. The findings assert that neurometrics can provide more information on the task that contributes to the workload, which can aid in the design of intelligent mental workload adaptive system.
... Various studies have been conducted recently to address the future ATCO simulation training issues regarding the implementation of new learning tools and technologies (Upgrove and Jafer (8,9) , Chayya et al. (10) , Coyne (11) ) and their impact on trainees' learning skills such as cross-task cue utilisation and situational awareness (Falkland and Wiggins (12) ), as well as workload and engagement metrics (Bernhardt et al. (13) ). These studies provide important insights into the development of ATCO simulation training content within the framework of regulations. ...
Article
Full-text available
Air traffic controller training is highly regulated but lacks prescribed common assessment criteria and methods to evaluate trainees at the level of basic training and consideration of how trainees in fluence flight efficiency.We investigated whether there is a correlation between two parameters, viz. the trainees’ assessment score and fuel consumption, obtained and calculated after real-time human-in-the-loop radar simulations within the ATCOSIMA project. Although basic training assessment standards emphasise safety indicators, it was expected that trainees with higher assessment scores would achieve better flight efficiency, i.e. less fuel consumption. However, the results showed that trainees’ assessment scores and fuel consumption did not correlate in the expected way, leading to several conclusions.
Article
Full-text available
In measurement and evaluation for a cognitive performance carried out on various tasks can use objective and subjective measurement tools. This study aims to review research on measuring instruments and provide its potential to be used in research related to cognitive ergonomics. The method that is used in this study is a review of article literature on studies that are subjective and objective measurement tools. the results of this study indicate that in each study usually does not only use one measuring instrument, to validate measurements, but another measurement tool is also used. And in research usually uses subjective and objective measurement tools for the same task.
Preprint
Full-text available
In this paper, an intelligent algorithm for modeling and simulating the Air Traffic Control (ATC) is presented. Reducing the workload of ATC operators by using computers instead of humans, the algorithm decreases the human faults. In this regard, two methods, including Naïve Bayes classifier (NB) and neural network (NN), are employed to make the machines intelligent. Having used a huge number of data from radio signal strengths at determined positions, least-square and backpropagation algorithms are utilized to train and evaluation test of NB classifier and neural network, respectively. The experiments compare between these two methods; however, both of them show satisfactory results in the determination of the agents' positions.
Conference Paper
Full-text available
In this paper, an intelligent algorithm for modeling and simulating the Air Traffic Control (ATC) is presented. Reducing the workload of ATC operators by using computers instead of humans, the algorithm decreases the human faults. In this regard, two methods, including Naïve Bayes classifier (NB) and neural network (NN), are employed to make the machines intelligent. Having used a huge number of data from radio signal strengths at determined positions, least-square and backpropagation algorithms are utilized to train and evaluation test of NB classifier and neural network, respectively. The experiments compare between these two methods; however, both of them show satisfactory results in the determination of the agents' positions.
Chapter
This research analyzed neuroimaging techniques for measuring cognitive load in multimedia research using a systematic literature review on all related papers published until April 2020. The most striking observation to emerge from the analysis is that electroencephalography, functional magnetic resonance imaging, functional near-infrared spectroscopy, and transcranial doppler ultrasonography have been the most preferred neuroimaging tools utilized in cognitive load in multimedia learning research. Forty articles were reviewed based on the techniques that should be known in the field of neuroimaging to study cognitive load in multimedia learning, the analysis methods for neuroimaging, the results related to cognitive load in multimedia research. The study's findings were evaluated, and many discrepancies in cognitive load research related to multimedia learning were discovered.
Chapter
Current organizational and manufacturing processes imply high mental workload demands that can be evaluated through the construct of MWL (mental workload). This term is often used in new manufacturing and organizational environments, which have replaced physical tasks with cognitive activities involving a high MWL. By overusing the attentional resources given to the tasks, such work environments are placing high cognitive loads on operators, thus affecting their performance and causing them to experience mental fatigue. A formal evaluation of MWL offers the opportunity to prevent mental disorders and maintain mental health. On the other hand, the lack of evaluation and proper management of MWL in the industry can result in errors that create economic costs, accidents, injuries, or even deadly events. Finally, MWL assessment and management can be a human-oriented strategy designed to improve and sustain the future of an organization. Industries must find competitive advantages in sustainable processes from the economic, environmental, and social view. In this sense, this chapter aims to present a literature review to provide a comprehensive literature analysis of MWL evaluation and management for sustainable processes in the manufacturing industry, from a social perspective.
Chapter
Mental or cognitive workload is an essential factor in Air Traffic Control (ATC). Besides the task demand, there are various types of individual factors that affect the workload felt by the Air Traffic Controller (ATCo). Therefore, this study aims to examine the relationship between individual factors and perceived mental workload using NASA-TLX. A total of 256 questionnaires were successfully collected from Air Traffic Controllers in the busiest airport in Indonesia. All observed groups indicated high perceived workload scores, with F-Test of One-way ANOVA used to confirm the relationship between individual factors and mental workload. The result showed that the workload is significantly related to the gender specification and working experience of ATCos. Fewer working years and female gender tended to have lower perceived workload than other groups. On the other hand, the difference in unit tasks (ACC and APP) and age failed to differ the perceived workload significantly. The result confirms that the mental workload of ATCo is high, and affected by some individual factors.
Article
Full-text available
Safety-critical systems like air traffic control (ATC) are usually less automated than might be expected by the public, so human intelligence will remain at the core in the decision-making (DM) process. Meanwhile, human factors (HFs) need to be fully considered in the DM process, which can design the ATC system to be more intelligent and more adaptive to the behaviour of the user. However, the existing DM research lacks the systematic methods that fully consider human performance in a smart manner. This study proposed a human-centred adaptive DM methodology that combines subjective and objective measurements made by functional near-infrared spectroscopy (fNIRS) via intelligent automation (IA). Moreover, this paper also described a case study of radar display map operation, including descriptive and optimised maps, to illustrate the proposed approach and verify its feasibility and effectiveness. The results were determined by jointly considering the user-generated and system-generated data and suggested that the proposed approach could capture subjective and objective data, take into consideration the HFs information to provide real-time online feedback and adjust the decision support system to HFs. It is hoped that this study can promote the methodology of human-centred subjective and objective data-driven applications in the future ATC environment adaptive decision research.
Article
This research empirically evaluates the introduction of speech to existing keyboard and mouse input modalities in an application used to control aircraft in a simulated, complex and dynamic environment. Task performance and task performance degradation are assessed for three levels of workload. Previous studies have evaluated task performance using these modalities however, only a couple have evaluated task performance under varying workload. Even though speech is a common addition to modern control interfaces, the effect of varying workload on this combination of control modalities has not yet been reported. Thirty-six participants commanded simulated aircraft through generated obstacle courses to reach a Combat Air Patrol (CAP) point while also responding to a secondary task. There were nine conditions that varied the control modality (Keyboard and Mouse (KM), Voice (V), and Keyboard, Mouse and Voice (KMV)), and workload by varying the number of aircraft being controlled (low, medium and high). Results showed that KM outperformed KMV and V for the low and medium workload levels. However, task performance with KMV was found to degrade the least as workload increased. KMV and KM were found to enable significantly more correct responses to the secondary task which was delivered aurally. Participants reported a preference for the combined modalities (KMV), self-assessing that KMV most reduced their workload. This research suggests that the addition of a speech interface to existing keyboard and mouse modalities, for control of aircraft in a simulation, may help manage cognitive load and may assist in controlling more aircraft under higher workloads.
Chapter
At present, electroencephalogram (EEG) has been widely used in the classification of mental workload. But most of the EEG acquisition devices used in the research a use a large number of electrodes. However, this brings high hardware costs, limited portability and discomfort to the wearer. In addition, most of the channels have information redundancy and noise interference, which have a negative impact on the subsequent mental workload classification. Therefore, it is necessary to use fewer channels to accurately identify the mental load of the operator. Focusing on the above problems, a method of channel selection based on Davies–Bouldin Index (DBI) for visual manipulation tasks is proposed in this paper, it selects effective channels by analyzing the differences between the features of low and high workload data.
Article
This study provides a systematic synthesis of empirical research on mental workload (MWL) in air traffic control (ATC). MWL is a key concept in research on innovative technologies, because the assessment of MWL is crucial to the evaluation of such technologies. Our specific focus was on physiological measures of MWL. The used search strategy identified 39 peer-reviewed publications that analyzed ATC tasks, examined different levels of difficulty of the ATC task, and considered at least one physiological measure of MWL. Positive relations between measures of MWL and task difficulty were observed most frequently, indicating that the measures indeed allowed the assessment of MWL. The most commonly used physiological measures were brain measures (EEG and fNIR) and heart rate measures. The review revealed a need for more precise descriptions of crucial experimental parameters in order to permit a transition of the field toward more interactive and dynamic types of analysis.
Article
The positivity effect among older adults is a tendency to process more positive and/or less negative emotional stimuli compared to younger adults, with unknown upper age boundaries. Cognitive and emotional working memory were assessed in young-old adults (60–75) and very old adults (VOAs; 80+) to determine whether emotional working memory declines similar to the age-related decline of cognitive working memory. The moderating role of valence on the link between age and emotional working memory was examined to identify change in positivity effect with advanced age. Electroencephalography (EEG) markers of cognitive workload and engagement were obtained to test the theory of cognitive resource allocation in older adults’ emotional stimuli processing. EEG recordings were collected during cognitive memory task and emotional working memory tasks that required rating emotional intensity of images pairs. Results indicate a positivity effect among VOAs that does not require additional cognitive effort and is not likely to diminish with age.
Preprint
Full-text available
Pupillometry is a promising method for assessing mental workload and could be helpful in the optimization of systems that involve human-computer interaction. The present study focuses on replicating the studies by Ahern (1978) and Klingner (2010), which found that for three levels of difficulty of mental multiplications, the more difficult multiplications yielded larger dilations of the pupil. Using a remote eye tracker, our research expands upon these two previous studies by statistically testing for each 1.5 s interval of the calculation period (1) the mean absolute pupil diameter (MPD), (2) the mean pupil diameter change (MPDC) with respect to the pupil diameter during the pre-stimulus accommodation period, and (3) the mean pupil diameter change rate (MPDCR). An additional novelty of our research is that we compared the pupil diameter measure with a self-report measure of workload, the NASA Task Load Index (NASA-TLX), and with the mean blink rate (MBR). The results showed that the findings of Ahern and Klingner were replicated, and that the MPD and MPDC discriminated just as well between the lowest and highest difficulty levels as did the NASA-TLX. The MBR, on the other hand, did not interpretably differentiate between the difficulty levels. Moderate to strong correlations were found between the MPDC and the proportion of incorrect responses, indicating that the MPDC was higher for participants with a poorer performance. For practical applications, validity could be improved by combining pupillometry with other physiological techniques.
Article
Full-text available
This research investigated controller' situation awareness by comparing COOPANS's acoustic alerts with newly designed semantic alerts. The results demonstrate that ATCOs' visual scan patterns had significant differences between acoustic and semantic designs. ATCOs established different eye movement patterns on fixations number, fixation duration and saccade velocity. Effective decision support systems require human-centered design with effective stimuli to direct ATCO's attention to critical events. It is necessary to provide ATCOs with specific alerting information to reflect the nature of the critical situation in order to minimise the side effects of startle and inattentional deafness. Consequently, the design of a semantic alert can significantly reduce ATCOs' response time, therefore providing valuable extra time in a time-limited situation to formulate and execute resolution strategies in critical air safety events. The findings of this research indicate that the context-specified design of semantic alerts could improve ATCO's situational awareness and significantly reduce response time in the event of Short Term Conflict Alert (STCA) activation which alerts to two aircraft having less than the required lateral or vertical separation. Practitioner Summary: Eye movements are closely linked with visual attention and can be analysed to explore shifting attention whilst performing monitoring tasks. This research has found that context-specific designed semantic alerts facilitated improved ATCO cognitive processing by integrating visual and auditory resources. Semantic designs have been demonstrated to be superior to acoustic design by directing the operator's attention more quickly to critical situations. Abbreviations: APW: area proximity warning; ASRS: aviation safety reporting system; ATC: air traffic control; ATCO: air traffic controller; ATM: air traffic management; COOPANS: cooperation between air navigation service providers; HCI: human-computer interaction; IAA: irish aviation authority; MSAW: minimum safe altitude warning; MTCD: medium-term conflict detection; SA: situation awareness; STCA: short term conflict alert; TP: trajectory prediction.
Article
Full-text available
Electroencephalograph (EEG) has been increasingly studied to identify distinct mental factors when persons perform cognitively demanding tasks. However, most of these studies examined EEG correlates at channel domain, which suffers the limitation that EEG signals are the mixture of multiple underlying neuronal sources due to the volume conduction effect. Moreover, few studies have been conducted in real-world tasks. To precisely probe EEG correlates with specific neural substrates to mental factors in real-world tasks, the present study examined EEG correlates to three mental factors, i.e., mental fatigue [also known as time-on-task (TOT) effect], workload and effort, in EEG component signals, which were obtained using an independent component analysis (ICA) on high-density EEG data. EEG data were recorded when subjects performed a realistically simulated air traffic control (ATC) task for 2 h. Five EEG independent component (IC) signals that were associated with specific neural substrates (i.e., the frontal, central medial, motor, parietal, occipital areas) were identified. Their spectral powers at their corresponding dominant bands, i.e., the theta power of the frontal IC and the alpha power of the other four ICs, were detected to be correlated to mental workload and effort levels, measured by behavioral metrics. Meanwhile, a linear regression analysis indicated that spectral powers at five ICs significantly increased with TOT. These findings indicated that different levels of mental factors can be sensitively reflected in EEG signals associated with various brain functions, including visual perception, cognitive processing, and motor outputs, in real-world tasks. These results can potentially aid in the development of efficient operational interfaces to ensure productivity and safety in ATC and beyond.
Article
Full-text available
The assessment of mental workload could be helpful to road safety especially if developments of vehicle automation will increasingly place drivers into roles of supervisory control. With the rapidly decreasing size and increasing resolution of cameras as well as exponential computational power gains, remote eye measurements are growing in popularity as non-obtrusive and non-distracting tools for assessing driver workload. This review summarizes literature on the relation between eye measurement parameters and drivers’ mental workload. Various eye activity measures including blinks, fixations, and saccades have previously researched and confirmed as useful estimates of a driver's mental workload. Additionally, recent studies in pupillometry have shown promise for real-time prediction and assessment of driver mental workload after effects of illumination are accounted for. Specifically, workload increases were found to be indicated by increases in blink latency, PERCLOS, fixation duration, pupil dilation, and ICA; by decreases in blink duration and gaze variability; and with mixed results regarding blink rate. Given such a range of measures available, we recommend using multiple assessment methods to increase validity and robustness in driver assessment.
Conference Paper
Full-text available
The present study examined whether personality characteristics and general intelligence predict multitasking performance. The Multi-Attribute Task Battery-II was used to assess multitasking performance. Personality factors included the Big Five, openness, conscientiousness, extraversion, agreeableness, and neuroticism. The results indicated scores on general intelligence predict performance on the tracking task of the Multi-Attribute Task Battery-II, where higher scores of general intelligence predicted improved tracking performance. Additionally, conscientiousness and neuroticism were found to predict worsened performance on the resource management task of the Multi-Attribute Task Battery-II. Furthermore, agreeableness was found to predict perceived workload on the mental demand subscale of the Workload Rating Scale.
Conference Paper
Full-text available
The present study examined the sensitivity of several candidate metrics of real-time workload within the spatial component of an unmanned aerial vehicle (UAV) task. Advanced Brain Monitoring's (ABM) wireless B-Alert system was used to collect participant's EEG workload and engagement data. Eye tracking data was also collected. The UAV simulation required participants to report heading information of moving vehicles, as seen from the UAV. There were four blocks of difficulty, over which a significant performance decrement was shown. Additionally, participants rated their workload significantly higher and pupil diameter significantly increased across blocks of increasing difficulty, as well as within each block during periods of highest mental demand. ABM's workload and engagement metrics however did not show a significant change over or within blocks. The results showed that pupil diameter shows promise as a correlate of mental workload.
Article
Full-text available
A study was run to test the sensitivity of multiple workload indices to the differing cognitive demands of four military monitoring task scenarios and to investigate relationships between indices. Various psychophysiological indices of mental workload exhibit sensitivity to task factors. However, the psychometric properties of multiple indices, including the extent to which they intercorrelate, have not been adequately investigated. One hundred fifty participants performed in four task scenarios based on a simulation of unmanned ground vehicle operation. Scenarios required threat detection and/or change detection. Both single- and dual-task scenarios were used. Workload metrics for each scenario were derived from the electroencephalogram (EEG), electrocardiogram, transcranial Doppler sonography, functional near infrared, and eye tracking. Subjective workload was also assessed. Several metrics showed sensitivity to the differing demands of the four scenarios. Eye fixation duration and the Task Load Index metric derived from EEG were diagnostic of single-versus dual-task performance. Several other metrics differentiated the two single tasks but were less effective in differentiating single- from dual-task performance. Psychometric analyses confirmed the reliability of individual metrics but failed to identify any general workload factor. An analysis of difference scores between low- and high-workload conditions suggested an effort factor defined by heart rate variability and frontal cortex oxygenation. General workload is not well defined psychometrically, although various individual metrics may satisfy conventional criteria for workload assessment. Practitioners should exercise caution in using multiple metrics that may not correspond well, especially at the level of the individual operator.
Article
Full-text available
According to Cognitive Load Theory (CLT), one of the crucial factors for successful learning is the type and amount of working-memory load (WML) learners experience while studying instructional materials. Optimal learning conditions are characterized by providing challenges for learners without inducing cognitive over- or underload. Thus, presenting instruction in a way that WML is constantly held within an optimal range with regard to learners' working-memory capacity might be a good method to provide these optimal conditions. The current paper elaborates how digital learning environments, which achieve this goal can be developed by combining approaches from Cognitive Psychology, Neuroscience, and Computer Science. One of the biggest obstacles that needs to be overcome is the lack of an unobtrusive method of continuously assessing learners' WML in real-time. We propose to solve this problem by applying passive Brain-Computer Interface (BCI) approaches to realistic learning scenarios in digital environments. In this paper we discuss the methodological and theoretical prospects and pitfalls of this approach based on results from the literature and from our own research. We present a strategy on how several inherent challenges of applying BCIs to WML and learning can be met by refining the psychological constructs behind WML, by exploring their neural signatures, by using these insights for sophisticated task designs, and by optimizing algorithms for analyzing electroencephalography (EEG) data. Based on this strategy we applied machine-learning algorithms for cross-task classifications of different levels of WML to tasks that involve studying realistic instructional materials. We obtained very promising results that yield several recommendations for future work.
Article
Full-text available
Objective: To demonstrate that psychophysiology may have applications for objective assessment of expertise development in deadly force judgment and decision making (DFJDM). Background: Modern training techniques focus on improving decision-making skills with participative assessment between trainees and subject matter experts primarily through subjective observation. OBJECTIVE metrics need to be developed. The current proof of concept study explored the potential for psychophysiological metrics in deadly force judgment contexts. Method: Twenty-four participants (novice, expert) were recruited. All wore a wireless Electroencephalography (EEG) device to collect psychophysiological data during high-fidelity simulated deadly force judgment and decision-making simulations using a modified Glock firearm. Participants were exposed to 27 video scenarios, one-third of which would have justified use of deadly force. Pass/fail was determined by whether the participant used deadly force appropriately. Results: Experts had a significantly higher pass rate compared to novices (p < 0.05). Multiple metrics were shown to distinguish novices from experts. Hierarchical regression analyses indicate that psychophysiological variables are able to explain 72% of the variability in expert performance, but only 37% in novices. Discriminant function analysis (DFA) using psychophysiological metrics was able to discern between experts and novices with 72.6% accuracy. Conclusion: While limited due to small sample size, the results suggest that psychophysiology may be developed for use as an objective measure of expertise in DFDJM. Specifically, discriminant function measures may have the potential to objectively identify expert skill acquisition. Application: Psychophysiological metrics may create a performance model with the potential to optimize simulator-based DFJDM training. These performance models could be used for trainee feedback, and/or by the instructor to assess performance objectively.
Article
Full-text available
Electroencephalography (EEG) holds promise as a neuroimaging technology that can be used to understand how the human brain functions in real-world, operational settings while individuals move freely in perceptually-rich environments. In recent years, several EEG systems have been developed that aim to increase the usability of the neuroimaging technology in real-world settings. Here, the usability of three wireless EEG systems from different companies are compared to a conventional wired EEG system, BioSemi's ActiveTwo, which serves as an established laboratory-grade 'gold standard' baseline. The wireless systems compared include Advanced Brain Monitoring's B-Alert X10, Emotiv Systems' EPOC and the 2009 version of QUASAR's Dry Sensor Interface 10-20. The design of each wireless system is discussed in relation to its impact on the system's usability as a potential real-world neuroimaging system. Evaluations are based on having participants complete a series of cognitive tasks while wearing each of the EEG acquisition systems. This report focuses on the system design, usability factors and participant comfort issues that arise during the experimental sessions. In particular, the EEG systems are assessed on five design elements: adaptability of the system for differing head sizes, subject comfort and preference, variance in scalp locations for the recording electrodes, stability of the electrical connection between the scalp and electrode, and timing integration between the EEG system, the stimulus presentation computer and other external events.
Article
Full-text available
Longer lasting performance in cognitively demanding tasks leads to an exhaustion of cognitive resources and to a state commonly described as mental fatigue. More specifically, the allocation and focusing of attention become less efficient with time on task. Additionally, the selection of even simple responses becomes more error prone. With respect to the recorded EEG, mental fatigue has been reported to be associated with an increase in frontal theta and frontal and occipital alpha activity. The present study focused on the time course of changes in behavior and in the EEG to characterize fatigue-related processes. Participants performed a spatial stimulus-response-compatibility task in eight blocks for an overall duration of four hours. Error rates increased continuously with time on task. Total alpha power was larger at the end compared to the beginning of the experiment. However, alpha power increased rapidly and reached its maximal amplitude already after one hour, whereas frontal theta showed a continuous increase with time on task, possibly related to increased effort to keep the performance level high. Time frequency analyses revealed power changes in the theta band induced by task relevant information that might be assigned to a drain of executive control capacities. Thus, frontal theta turned out to be a reliable marker of distinct changes in cognitive processing with increasing fatigue.
Article
Full-text available
The aim of this study was to identify the cognitive factors that predictability and adaptability during multitasking with a flight simulator. Multitasking has become increasingly prevalent as most professions require individuals to perform multiple tasks simultaneously. Considerable research has been undertaken to identify the characteristics of people (i.e., individual differences) that predict multitasking ability. Although working memory is a reliable predictor of general multitasking ability (i.e., performance in normal conditions), there is the question of whether different cognitive faculties are needed to rapidly respond to changing task demands (adaptability). Participants first completed a battery of cognitive individual differences tests followed by multitasking sessions with a flight simulator. After a baseline condition, difficulty of the flight simulator was incrementally increased via four experimental manipulations, and performance metrics were collected to assess multitasking ability and adaptability. Scholastic aptitude and working memory predicted general multitasking ability (i.e., performance at baseline difficulty), but spatial manipulation (in conjunction with working memory) was a major predictor of adaptability (performance in difficult conditions after accounting for baseline performance). Multitasking ability and adaptability may be overlapping but separate constructs that draw on overlapping (but not identical) sets of cognitive abilities. The results of this study are applicable to practitioners and researchers in human factors to assess multitasking performance in real-world contexts and with realistic task constraints. We also present a framework for conceptualizing multitasking adaptability on the basis of five adaptability profiles derived from performance on tasks with consistent versus increased difficulty.
Article
Full-text available
Fatigue is a well known stressor in aviation operations and its interaction with mental workload needs to be understood. Performance, psychophysiological, and subjective measures were collected during performance of three tasks of increasing complexity. A psychomotor vigilance task, multi-attribute task battery and an uninhabited air vehicle task were performed five times during one night's sleep loss. EEG, ECG and pupil area were recorded during task performance. Performance decrements were found at the next to last and/or last testing session. The EEG showed concomitant changes. The degree of impairment was at least partially dependent on the task being performed and the performance variable assessed.
Article
Full-text available
Mental imagery is known to play a significant role in the skilled performance of complex cognitive tasks, yet is mostly overlooked in the field of air traffic control—a task that is reliant on what controllers term “the picture.” This article explores 3 strands of imagery research: the similarities between imagery and perception, individual differences in imagery, and skill learning and imagery. The research reported is discussed in terms of fundamental implications for air traffic control, implications for the measurement of imagery, implications for training, and implications for technology design.
Article
Full-text available
This chapter argues that fatigue is a problem of the management of control rather than of energy. Thorndike (1900) interpreted fatigue as a problem of doing the right thing, rather than of doing too much. Bartley and Chute (1947) considered fatigue a result of conflict between competing behavioral tendencies—between doing and not doing, between doing one thing and doing another. The idea that the resolution of conflict is an effective basis for the control of action is a familiar one (Berlyne, 1960; Botvinick, Braver, Barch, Carter, & Cohen, 2001; Norman & Shallice, 1986), with cognitive control acting to maintain selected tasks and prevent disruption by competing activities. Fatigue is interpreted here as an adaptive state, serving to maintain effective overall (system-wide) management of goals. In this conceptualization, the subjective experience of fatigue arises through conflict between current and competing goals, or action tendencies. In effect, it is assumed to have a signal value for motivational control, providing a mechanism for resolving conflicts between current goals and other desired courses of action. This approach is developed in the rest of the chapter by considering the boundary conditions for the experience and impact of fatigue, especially in relation to work. The focus is necessarily broader than fatigue itself, as fatigue is considered to be one aspect of the general control system that manages goal activity in the service of motivational requirements. A fundamental assumption of traditional theory is that fatigue is caused by work, but the nature of the work may be important: Does it matter how much effort is applied by the performer or how much control he or she has over what is done? A second assumption is that fatigue causes decrements in the performance of tasks (as a result of prolonged work without rest), but this is not always found to be the case. Researchers consider the nature of performance decrement, and the reasons why task goals may fail, both with the need to maintain them over time and, in the broader context, under the impact of stress and high workload. This leads to considering the general model of motivational control in task performance, in which goal management is conceived of as an essentially compensatory process (Hockey, 1997, 2005). At least for highly motivated tasks, primary task goals appear to be stabilized by increased regulatory control, with associated costs. Over the final part of the chapter, the compensatory control model is adapted and extended to focus on the specific problem of cognitive fatigue and its implications for patterns of performance decrement. In this model, fatigue is interpreted as an adaptive state that signals a growing conflict in control activity between what is being done and what else might be done, between old goals and new goals, and between duties and desires. (PsycINFO Database Record (c) 2012 APA, all rights reserved)
Article
Full-text available
This paper describes the origins and hisotry of multiple resource theory in accounting for difference in dual task interference. One particular application of the theory, the 4-dimensional multiple resources model, is described in detail, positing that there will be greater interference between two tasks to the extent that they share stages (perceptua/cognitive vs response) sensory modalities (auditory vs visual), codes (visual vs spatial) and channels of visual information (focal vs ambient). A computational rendering of this model is then presented. Examples are given of how the model predicts interference differences in operational environments. Finally, three challenges to the model are outlined regarding task demand coding, task allocation and visual resource competition.
Article
Full-text available
Abstract Cognitive load theory has been designed to provide guidelines intended to assist in the presentation of information in a manner that encourages learner activities that optimize intellectual performance. The theory assumes a limited capacity working memory that includes partially independent subcomponents to deal with auditory/verbal material and visual/2- or 3-dimensional information as well as an effectively unlimited long-term memory, holding schemas that vary in their degree of automation. These structures and functions of human cognitive architecture have been used to design a variety of novel instructional procedures based on the assumption that working memory load should be reduced and schema construction encouraged. This paper reviews the theory and the instructional designs generated by it.
Article
Full-text available
Modern work requires cognitively demanding multitasking and the need for sustained vigilance, which may result in work-related stress and may increase the possibility of human error. Objective methods for estimating cognitive overload and mental fatigue of the brain on-line, during work performance, are needed. We present a two-channel electroencephalography (EEG)-based index, theta Fz/alpha Pz ratio, potentially implementable into a compact wearable device. The index reacts to both acute external and cumulative internal load. The index increased with the number of tasks to be performed concurrently (p = 0.004) and with increased time awake, both after normal sleep (p = 0.002) and sleep restriction (p = 0.004). Moreover, the increase of the index was more pronounced in the afternoon after sleep restriction (p = 0.006). As a measure of brain state and its dynamics, the index can be considered equivalent to the heartbeat, an indicator of the cardiovascular state, thus inspiring the name "brainbeat".
Article
Full-text available
This study examined the relation between sleep quality and cognitive performance in older adults, controlling for common medical comorbidities. Participants were community volunteers who, while not selected on the basis of their sleep, did report substantial variability in sleep quality. Good and poor sleepers differed on tests of working memory, attentional set shifting, and abstract problem solving but not on processing speed, inhibitory function, or episodic memory. Poor sleep was also associated with increased depressive symptomatology but only for functional symptoms (e.g., decreased concentration) and not for mood (e.g., sadness). The relationships between sleep quality and cognition were not explained by confound factors such as cerebrovascular disease, depression, or medication usage. Sleep problems may contribute to performance variability between elderly individuals but only in certain cognitive domains.
Article
Air traffic controllers are required to perform complex tasks which require attention and high precision. This study investigates how the difficulty of such tasks influences emotional states, cognitive workload and task performance. We use quantitative and qualitative measurements, including the recording of pupil dilation and changes in affect using questionnaires. Participants were required to perform a number of air traffic control tasks using the immersive human accessible Virtual Reality space in the "eXperience Induction Machine". Based on the data collected, we developed and validated a model which integrates personality, workload and affective theories. Our results indicate that the difficulty of an air traffic control task has a direct influence on cognitive workload as well as on the self-reported mood; whereas both mood and workload seem to change independently. In addition, we show that personality, in particular neuroticism, affects both mood and performance of the participants.
Article
This article provides the reader a focused and organised review of the research progresses on neurophysiological indicators, also called “neurometrics”, to show how neurometrics could effectively address some of the most important Human Factors (HFs) needs in the Air Traffic Management (ATM) field. The state of the art on the most involved HFs and related cognitive processes (e.g. mental workload, cognitive training) is presented together with examples of possible applications in the current and future ATM scenarios, in order to better understand and highlight the available opportunities of such neuroscientific applications. Furthermore, the paper will discuss the potential enhancement that further research and development activities could bring to the efficiency and safety of the ATM service.
Article
reviews major categories of empirical workload measurement techniques and provides guidelines for the choice of appropriate assessment procedures for particular applications sensitivity / diagnosticity rating scales / psychometric techniques (PsycINFO Database Record (c) 2012 APA, all rights reserved)
Article
In our anxiogenic and stressful world, the maintenance of an optimal cognitive performance is a constant challenge. It is particularly true in complex working environments (e.g. flight deck, air traffic control tower), where individuals have sometimes to cope with a high mental workload and stressful situations. Several models (i.e. processing efficiency theory, cognitive-energetical framework) have attempted to provide a conceptual basis on how human performance is modulated by high workload and stress/anxiety. These models predict that stress can reduce human cognitive efficiency, even in the absence of a visible impact on the task performance. Performance may be protected under stress thanks to compensatory effort, but only at the expense of a cognitive cost. Yet, the psychophysiological cost of this regulation remains unclear. We designed two experiments involving pupil diameter, cardiovascular and prefrontal oxygenation measurements. Participants performed the Toulouse N-back Task that intensively engaged both working memory and mental calculation processes under the threat (or not) of unpredictable aversive sounds. The results revealed that higher task difficulty (higher n level) degraded the performance and induced an increased tonic pupil diameter, heart rate and activity in the lateral prefrontal cortex, and a decreased phasic pupil response and heart rate variability. Importantly, the condition of stress did not impact the performance, but at the expense of a psychophysiological cost as demonstrated by lower phasic pupil response, and greater heart rate and prefrontal activity. Prefrontal cortex seems to be a central region for mitigating the influence of stress because it subserves crucial functions (e.g. inhibition, working memory) that can promote the engagement of coping strategies. Overall, findings confirmed the psychophysiological cost of both mental effort and stress. Stress likely triggered increased motivation and the recruitment of additional cognitive resources that minimize its aversive effects on task performance (effectiveness), but these compensatory efforts consumed resources that caused a loss of cognitive efficiency (ratio between performance effectiveness and mental effort).
Article
The aim of this study was to evaluate the sensitivity of a range of EEG indices to time-on-task effects and to a workload manipulation (cueing), during performance of a resource-limited vigilance task. Effects of task period and cueing on performance and subjective state response were consistent with previous vigilance studies and with resource theory. Two EEG indices-the Task Load Index (TLI) and global lower frequency (LF) alpha power-showed effects of task period and cueing similar to those seen with correct detections. Across four successive task periods, the TLI declined and LF alpha power increased. Cueing increased TLI and decreased LF alpha. Other indices-the Engagement Index (EI), frontal theta and upper frequency (UF) alpha failed to show these effects. However, EI and frontal theta were sensitive to interactive effects of task period and cueing, which may correspond to a stronger anxiety response to the uncued task.
Article
This effort investigated the ability of a neurophysiological measure to detect changes in workload during a task which is sensitive to cognitive function. A growing collection of research suggests that physiological measures such as EEG can be used to inform the adaptation of systems. However, it has been proposed that such measures often provide a gross interpretation of cognitive workload during complex tasks and are not sensitive to differences in specific cognitive function. To understand the utility of neurophysiological measures for human-machine interaction, we must know if these measures are sensitive to tasks which are sensitive to changes in cognitive function. To begin to answer this question, we investigated the sensitivity of Advanced Brain Monitoring’s EEG-based measures to changes in workload experienced during a Stroop task. Results indicated that ABM’s workload measure can detect changes associated with the attentional demands and cognitive processes linked to the ability to inhibit word naming during tasks involving semantic interference. This indicates that changes in workload associated with the ability to inhibit competing cognitive processes can be identified using neurophysiological workload measures.
Article
Perhaps the most basic issue in the study of cognitive workload is the problem of how to actually measure it. The electroencephalogram (EEG) continues to be the clinical method of choice for monitoring brain function in assessing sleep disorders, level of anaesthesia and epilepsy. This preference reflects the EEG’s high sensitivity to variations in alertness and attention, the unimposing conditions under which it can be recorded, and the low cost of the technology it requires. These characteristics also suggest that EEG-based monitoring methods might provide a useful tool in ergonomics. This paper reviews a long-term programme of research aimed at developing cognitive workload monitoring methods based on EEG measures. This research programme began with basic studies of the way neuroelectric signals change in response to highly controlled variations in task demands. The results yielded from such studies provided a basis on which to develop appropriate signal processing methodologies to automatically differentiate mental effort-related changes in brain activity from artifactual contaminants and for gauging relative magnitudes of mental effort in different task conditions. These methods were then evaluated in the context of more naturalistic computerbased work. The results obtained from these studies provide initial evidence for the scientific and technical feasibility of using EEG-based methods for monitoring cognitive load during human–computer interaction.
Article
Performance on measures of working memory (WM) capacity predicts performance on a wide range of real-world cognitive tasks. I review the idea that WM capacity (a) is separable from short-term memory, (b) is an important component of general fluid intelligence, and (c) represents a domain-free limitation in ability to control attention. Studies show that individual differences in WM capacity are reflected in performance on antisaccade, Stroop, and dichotic-listening tasks. WM capacity, or executive attention, is most important under conditions in which interference leads to retrieval of response tendencies that conflict with the current task.
Article
To better understand the mechanisms by which working memory training can augment human performance we continuously monitored trainees with near infrared spectroscopy (NIRS) while they performed a dual verbal-spatial working memory task. Linear mixed effects models were used to model the changes in cerebral hemodynamic response as a result of time spent training working memory. Nonlinear increases in left dorsolateral prefrontal cortex (DLPFC) and right ventrolateral prefrontal cortex (VLPFC) were observed with increased exposure to working memory training. Adaptive and yoked training groups also showed differential effects in rostral prefrontal cortex with increased exposure to working memory training. There was also a significant negative relationship between verbal working memory performance and bilateral VLPFC activation. These results are interpreted in terms of decreased proactive interference, increased neural efficiency, reduced mental workload for stimulus processing, and increased working memory capacity with training.
Article
BORGHINI et al. Measuring neurophysiological signals in aircraft pilots and car drivers for the assessment of mental workload, fatigue and drowsiness NEUROSCI BIOBEHAV REV 21(1) XXX-XXX, 2012. This paper reviews published papers related to neurophysiological measurements (electroencephalography: EEG, electrooculography EOG; heart rate: HR) in pilots/drivers during their driving tasks. The aim is to summarise the main neurophysiological findings related to the measurements of pilot/driver's brain activity during drive performance and how particular aspects of this brain activity could be connected with the important concepts of "mental workload", "mental fatigue" or "situational awareness". Review of the literature suggest that exists a coherent sequence of changes for EEG, EOG and HR variables during the transition from normal drive, high mental workload and eventually mental fatigue and drowsiness. In particular, increased EEG power in theta band and a decrease in alpha band occurred in high mental workload. Successively, increased EEG power in theta as well as delta and alpha bands characterize the transition between mental workload and mental fatigue. Drowsiness is also characterized by increased blink rate and decreased HR values. The detection of such mental states is actually performed "off-line" with accuracy around 90% but not on-line. A discussion on the possible future applications of findings provided by these neurophysiological measurements in order to improve the safety of the vehicles will be also presented.
Article
This paper reports three studies on the application of ambulatory monitoring in air traffic control (ATC). The aim of the first study was to explore a set of psychophysiological measures with respect to ATC workload sensitivity and feasibility at the workplace. Nearly all physiological measures showed the expected changes during work. Significant positive correlations were found between cardiovascular responses and the number of aircraft under control, especially heavy, fast, climbing, and descending aircraft. The following en-route (Study 2) and tower (Study 3) simulations identified the relative impact of air traffic features. Heart rate, systolic blood pressure, self-reported concentration, and upset were significantly higher in the simulations with 12 aircraft continuously under control compared to only 6. A high versus low number of potential conflicts between aircraft in the en-route setting (Study 2) also caused significant increases of heart rate, systolic blood pressure, self-reported concentration, and upset. On the basis of these results, a new workload model for air traffic controllers was suggested and implemented. (PsycINFO Database Record (c) 2012 APA, all rights reserved)
Article
mental workload is proposed to be a multidimensional construct that can be largely explained by three component factors: time load, mental effort load, and psychological stress load in this paper, we describe a subjective scaling approach, the Subjective Workload Assessment Technique (SWAT), that captures this multidimensional nature of the mental workload we describe the SWAT procedure as a two-phased method that includes (a) a scale development phase based on conjoint measurement and nonmetric scaling, and (b) an event scoring phase the development of SWAT and its measurement foundations are discussed recent research illustrating SWAT's widespread utility and its sensitivity as a measure of perceived mental workload is summarized (PsycINFO Database Record (c) 2012 APA, all rights reserved)
Article
This paper presents a cognitive-energetical framework for the analysis of effects of stress and high workload on human performance. Following Kahneman's (1973) model, regulation of goals and actions is assumed to require the operation of a compensatory control mechanism, which allocates resources dynamically. A two-level compensatory control model provides the basis for a mechanism of resource allocation through an effort monitor, sensitive to changes in the level of regulatory activity, coupled with a supervisory controller which can implement different modes of performance-cost trade-off. Performance may be protected under stress by the recruitment of further resources, but only at the expense of increased subjective effort, and behavioural and physiological costs. Alternatively, stability can be achieved by reducing performance goals, without further costs. Predictions about patterns of latent decrement under performance protection are evaluated in relation to the human performance literature. Even where no primary task decrements may be detected, performance may show disruption of subsidiary activities or the use of less efficient strategies, as well as increased psychophysiological activation, strain, and fatigue after-effects. Finally, the paper discusses implications of the model for the assessment of work strain, with a focus on individual-level patterns of regulatory activity and coping.
Article
In this study, a combined measure was developed based on various physiological indices in order to evaluate the mental workload during a dual task. To determine the mental effort required for each task, three physiological signals were recorded while ten subjects performed different versions of a dual task composed of tracking and mental arithmetic. These signals were the electroencephalogram (EEG), electrooculogram (EOG), and electrocardiogram (ECG), which were transformed into the suppression of alpha rhythm, eye blink interval, and heart rate variability (HRV), respectively. The alpha suppression provided proper information to infer the efforts for the arithmetic task, but not for the tracking task. Conversely, the blink interval and HRV permitted detailed inferences over the workload of the tracking task, but not for the arithmetic task. These results can be explained in terms of the multiple resources model of workload. The processing indexed by the alpha suppression is inferred to be different from that indexed by the blink interval or HRV. Finally, the physiological measures were combined into a single measure using different weight coefficients. The newly developed measure systematically increased with the difficulty of each task and significantly distinguished between the different versions of each task.Relevance to industryA combined measure of mental workload that has the ability to evaluate operators’ mental effort in a multitask condition would be valuable in a natural working environment, because most such work is composed of multiple tasks. In this paper, an approach is described that developed a combined measure of mental workload based on three physiological indices.
Article
The integration of brain monitoring into the man-machine interface holds great promise for real-time assessment of operator status and intelligent allocation of tasks between machines and humans. This article presents an integrated hardware and software solution for acquisition and real-time analysis of the electroencephalogram (EEG) to monitor indexes of alertness, cognition, and memory. Three experimental paradigms were evaluated in a total of 45 participants to identify EEG indexes associated with changes in cognitive workload: the Warship Commander Task (WCT), a simulated navy command and control environment that allowed workload levels to be systematically manipulated; a cognitive task with three levels of difficulty and consistent sensory inputs and motor outputs; and a multisession image learning and recognition memory test. Across tasks and participants, specific changes in the EEG were identified that were reliably associated with levels of cognitive workload. The EEG indexes were also shown to change as a function of training on the WCT and the learning and memory task. Future applications of the system to augment cognition in military and industrial environments are discussed.
Chapter
The results of a multi-year research program to identify the factors associated with variations in subjective workload within and between different types of tasks are reviewed. Subjective evaluations of 10 workload-related factors were obtained from 16 different experiments. The experimental tasks included simple cognitive and manual control tasks, complex laboratory and supervisory control tasks, and aircraft simulation. Task-, behavior-, and subject-related correlates of subjective workload experiences varied as a function of difficulty manipulations within experiments, different sources of workload between experiments, and individual differences in workload definition. A multi-dimensional rating scale is proposed in which information about the magnitude and sources of six workload-related factors are combined to derive a sensitive and reliable estimate of workload.
Article
I present an account of the origins and development of the multicomponent approach to working memory, making a distinction between the overall theoretical framework, which has remained relatively stable, and the attempts to build more specific models within this framework. I follow this with a brief discussion of alternative models and their relationship to the framework. I conclude with speculations on further developments and a comment on the value of attempting to apply models and theories beyond the laboratory studies on which they are typically based.
Article
An accurate measure of mental workload in human operators is a critical element of monitoring and adaptive aiding systems that are designed to improve the efficiency and safety of human-machine systems during critical tasks. Functional near infrared (fNIR) spectroscopy is a field-deployable non-invasive optical brain monitoring technology that provides a measure of cerebral hemodynamics within the prefrontal cortex in response to sensory, motor, or cognitive activation. In this paper, we provide evidence from two studies that fNIR can be used in ecologically valid environments to assess the: 1) mental workload of operators performing standardized (n-back) and complex cognitive tasks (air traffic control--ATC), and 2) development of expertise during practice of complex cognitive and visuomotor tasks (piloting unmanned air vehicles--UAV). Results indicate that fNIR measures are sensitive to mental task load and practice level, and provide evidence of the fNIR deployment in the field for its ability to monitor hemodynamic changes that are associated with relative cognitive workload changes of operators. The methods reported here provide guidance for the development of strategic requirements necessary for the design of complex human-machine interface systems and assist with assessments of human operator performance criteria.
Article
A great deal of research over the last century has focused on drowsiness/alertness detection, as fatigue-related physical and cognitive impairments pose a serious risk to public health and safety. Available drowsiness/alertness detection solutions are unsatisfactory for a number of reasons: (1) lack of generalizability, (2) failure to address individual variability in generalized models, and/or (3) lack of a portable, un-tethered application. The current study aimed to address these issues, and determine if an individualized electroencephalography (EEG) based algorithm could be defined to track performance decrements associated with sleep loss, as this is the first step in developing a field deployable drowsiness/alertness detection system. The results indicated that an EEG-based algorithm, individualized using a series of brief "identification" tasks, was able to effectively track performance decrements associated with sleep deprivation. Future development will address the need for the algorithm to predict performance decrements due to sleep loss, and provide field applicability.
Article
Complex systems are vulnerable to unpredictable breakdowns in operator performance. Although primary task goals are typically protected by compensatory effort, such protection may break down under fatigue and high strain. Detection of strain states would enable prediction of increased operational risk through adaptive automation, triggering a switch of control from human to computer. A simulated process control task was used to identify markers of strain under a cyclic loading procedure, which forced performance breakdown through stepwise changes in control load. Four trained participants provided data on control performance and a range of candidate psychophysiological markers of strain (two EEG power ratios and HRV). Within-individual analyses showed the strongest sensitivity for 'task load index' (TLI), an EEG measure based on executive control activity in frontal brain areas, though all measures were sensitive for some participants. The implications of such findings for the development of a closed loop system for adaptive automation are discussed.
Article
A new paper-and-pencil test of spatial visualization was constructed from the figures used in the chronometric study of Shepard and Metzler (1971). In large samples, the new test displayed substantial internal consistency (Kuder-Richardson 20 = .88), a test-retest reliability (.83), and consistent sex differences over the entire range of ages investigated. Correlations with other measures indicated strong association with tests of spatial visualization and virtually no association with tests of verbal ability.
Article
Despite the prevalence of sleep complaints among psychiatric patients, few questionnaires have been specifically designed to measure sleep quality in clinical populations. The Pittsburgh Sleep Quality Index (PSQI) is a self-rated questionnaire which assesses sleep quality and disturbances over a 1-month time interval. Nineteen individual items generate seven "component" scores: subjective sleep quality, sleep latency, sleep duration, habitual sleep efficiency, sleep disturbances, use of sleeping medication, and daytime dysfunction. The sum of scores for these seven components yields one global score. Clinical and clinimetric properties of the PSQI were assessed over an 18-month period with "good" sleepers (healthy subjects, n = 52) and "poor" sleepers (depressed patients, n = 54; sleep-disorder patients, n = 62). Acceptable measures of internal homogeneity, consistency (test-retest reliability), and validity were obtained. A global PSQI score greater than 5 yielded a diagnostic sensitivity of 89.6% and specificity of 86.5% (kappa = 0.75, p less than 0.001) in distinguishing good and poor sleepers. The clinimetric and clinical properties of the PSQI suggest its utility both in psychiatric clinical practice and research activities.
Article
A physiological measure of processing load or "mental effort" required to perform a cognitive task should accurately reflect within-task, between-task, and between-individual variations in processing demands. The present article reviews all available experimental data on task-evoked pupillary response. It is concluded that the task-evoked pupillary response fulfills the criteria. Alternative explanations are considered and rejected. Implications for neurophysiological and cognitive theories of processing resources are discussed. (47 ref)
Article
A system was evaluated for use in adaptive automation using two experiments with electroencephalogram (EEG) indices based on the beta, alpha, and theta bandwidths. Subjects performed a compensatory tracking task while their EEG was recorded and converted to one of three engagement indices: beta/(alpha + theta), beta/alpha, or 1/alpha. In experiment one, the tracking task was switched between manual and automatic modes depending on whether the subject's engagement index was increasing or decreasing under a positive or negative feedback condition. Subjects were run for three consecutive 16-min trials. In experiment two, the task was switched depending on whether the absolute level of the engagement index for the subject was above or below baseline levels. It was hypothesized that negative feedback would produce more switches between manual and automatic modes, and that the beta/(alpha + theta) index would be most effective. The results confirmed these hypotheses. Tracking performance was better under negative feedback in both experiments; also, the use of absolute levels of engagement in experiment two resulted in better performance. There were no systematic changes in these effects over three 16-min trials. The implications for the use of such systems for adaptive automation are discussed.
Article
In 1974, Baddeley and Hitch proposed a three-component model of working memory. Over the years, this has been successful in giving an integrated account not only of data from normal adults, but also neuropsychological, developmental and neuroimaging data. There are, however, a number of phenomena that are not readily captured by the original model. These are outlined here and a fourth component to the model, the episodic buffer, is proposed. It comprises a limited capacity system that provides temporary storage of information held in a multimodal code, which is capable of binding information from the subsidiary systems, and from long-term memory, into a unitary episodic representation. Conscious awareness is assumed to be the principal mode of retrieval from the buffer. The revised model differs from the old principally in focussing attention on the processes of integrating information, rather than on the isolation of the subsystems. In doing so, it provides a better basis for tackling the more complex aspects of executive control in working memory.