No full-text available
To read the full-text of this research,
you can request a copy directly from the authors.
DARPA's Augmented Cognition Program-tomorrow's human computer interaction from vision to reality: building cognitively aware computational systems
Abstract
The Augmented Cognition (AugCog) program will extend, by an order-of-magnitude or more, the information management capacity of the "human-computer" combination by developing and demonstrating enhancements to human cognitive ability in diverse and stressful operational environments. Specifically, this program will develop the technologies needed to measure and track a subject's cognitive state in real-time. These measurements will then be used to augment the user's environment, and tailor that environment to a particular user's state. The technologies under development in AugCog have the potential to enhance operational capability, support reduction in the numbers of persons required to perform current functions, and improve human performance in cognitively challenging environments. In FY 2002, the AugCog program is developing robust, noninvasive, real-time, cognitive state detection technology for measuring the cognitive processing state of the user. In FY 2003, AugCog will be developing and testing integrated multi-sensor interface technologies that will permit human state manipulation. These efforts represent a new paradigm for human-computational systems interfaces.
... The goal of AugCog is to enhance end user cognitive capacity and capability in support of human task performance via automated adaption of technical system function and information presentation in a closed loop system [19,20]. Three principle AugCog research areas are: Cognitive State Assessment (CSA) enabled by sensor-based capture of cognitive or functional state; Mitigation Strategies (MS) that respond to cognitive state through closed-loop system adjustments; and Robust Controllers (RC) that allow systems to function with resilience under diverse operating conditions [20,21]. ...
... AugCog will enable computational systems to adapt to the user, rather than forcing the user to adapt to the computational systems. In this way the AugCog program moves beyond the traditional approach to redesigning human-computer interfaces -which often fail to take the state of the user into account [19]. ...
... However, the health domain represents a context with stakeholders who may have conflicting goals (e.g.: patients, family members, and health care providers with different roles). As a result, there may be fewer parallels between the health domain of patients in everyday life, the military domain where AugCog research originated [19,21], and nuclear power plant control rooms where AugCog approaches have translated [20]. In particular, "operators" in military and control room environments possess specialized training, skills, and protocols designed to facilitate achievement of organizational objectives whereas patients in everyday life may not. ...
In this paper, we review current smart watch research in the health domain to inform an Augmented Cognition (AugCog) research agenda for health-related decision making and patient self-management. We connect this AugCog research agenda to prior Clinical Decision Support (CDS), workflow, and informatics research efforts using Persons Living With HIV (PLWH) and Chronic Obstructive Pulmonary Disorder (COPD) patients as examples to illustrate potential research directions.
... Towards achieving this vision, sophisticated new features and systems are designed or will be implemented for future air-traffic management. Brain based measures of operator's cognitive workload could help assess utility of interfaces in human machine systems operating in complex environments and provide objective measures in addition to behavioral performance and subjective self-reported feedback [2][3][4]. The use of human brain imaging sensors and their capacity to integrate with behavioral and physiological measures, position them to play a key role in the development and operation of futuristic 'brain-in-theloop' systems. ...
... The general model: (1) where X is the oxygenation value and Y is the normalized output response. and are scalar model parameters (of a participant) estimated by using two (nbackoxygenation, nback-condition) coordinates, where nback-condition is either 0, 1, 2 or 3. Using minimum and maximum oxygenation (and respective condition), parameters can be solved from the following equation set: (2) Finally, using this model, ATC oxygenation values for all 6 (2 communication x 3 task difficulty) conditions of the participants were transformed. ...
Deployment of portable neuroimaging technologies to operating settings could help assess cognitive states of personnel assigned
to perform critical tasks and thus help improve efficiency and safety of human machine systems. Functional Near Infrared Spectroscopy
(fNIR) is an emerging noninvasive brain imaging technology that relies on optical techniques to detect brain hemodynamics
within the prefrontal cortex in response to sensory, motor, or cognitive activation. Collaborating with the FAA William J.
Hughes Technical Center, fNIR has been used to monitor twenty four certified professional controllers as they manage realistic
Air Traffic Control (ATC) scenarios under typical and emergent conditions. We have implemented a normalization procedure to
estimate cognitive workload levels from fNIR signals during ATC by developing linear regression models that were informed
by the respective participants’ prior n-back data. This normalization can account for oxygenation variance due to inter-personal
physiological differences. Results indicate that fNIR is sensitive task loads during ATC.
... Augmented cognition [27,27] is encephalography (EEG)-based adaptive allocation. EEG represents the signals occurring because of brain activities that are triggered by brain functions. ...
... Augmented cognition [27,27] is encephalography (EEG)-based adaptive allocation. EEG represents the signals occurring because of brain activities that are triggered by brain functions. ...
Artificial intelligence (AI) is finding more uses in the human society resulting in a need to scrutinise the relationship between humans and AI. Technology itself has advanced from the mere encoding of human knowledge into a machine to designing machines that “know how” to autonomously acquire the knowledge they need, learn from it and act independently in the environment. Fortunately, this need is not new; it has scientific grounds that could be traced back to the inception of computers. This paper uses a multi-disciplinary lens to explore how the natural cognitive intelligence in a human could interface with the artificial cognitive intelligence of a machine. The scientific journey over the last 50 years will be examined to understand the Human-AI relationship, and to present the nature of, and the role of trust in, this relationship. Risks and opportunities sitting at the human-AI interface will be studied to reveal some of the fundamental technical challenges for a trustworthy human-AI relationship. The critical assessment of the literature leads to the conclusion that any social integration of AI into the human social system would necessitate a form of a relationship on one level or another in society, meaning that humans will “always” actively participate in certain decision-making loops—either in-the-loop or on-the-loop—that will influence the operations of AI, regardless of how sophisticated it is.
... In 2000 O'Hanlon identified and forecasted key technology areas where the pace of change was likely to have a moderate, high or revolutionary impact on the conduct of military operations by 2020. 2 O'Hanlon described three gradations of anticipated technological innovation, including modest advances resulting in up to 20% improvement in performance from current levels; high advances resulting in 50-100% improvement in performance from current levels; and revolutionary advances where the type and pace of progress would render old weapons, tactics, and operational approaches obsolete while making possible the accomplishment of new and important battlefield tasks that cannot now be even attempted. 2 Knott and Perconti (2018), in a recent empirical assessment of the accuracy of longterm predictions, found that forecasting military technologies over a [20][21][22][23][24][25][26][27][28][29][30] year time horizon is both feasible and sufficiently accurate for the purposes of supporting decision making on research priorities and investments. 3 To identify key technologies and make forecasts O'Hanlon focused on the foundational laws of physics to determine the limits of possibility and examined the scientific, engineering, and defence literature on various types of technological research to understand what was likely to be developed over the 2000-2020 time period. After forming initial estimates of key trends, O'Hanlon consulted with experts, including at several of the major US weapons laboratories, for their feedback and advice. ...
... It also includes training systems using virtual reality along with real time physiological monitoring to provide biofeedback to develop enhanced mental and physiological selfregulation capabilities; the same technologies can be used to adapt information to the physiological status of the individual operator (e.g., "augmented cognition"). 22 Teams may train and operate with shared systems drawing on wearable sensors and effectors (e.g., "brain net", a mentalized internet). 23 Major components of a Matrix-like science fiction concept have been demonstrated for a capability to artificially sense and activate muscles to bend the body to dodge bullets. ...
Objectives
To demonstrate the need for the military human performance research community to anticipate and evolve with the emergence of new and disruptive battlefield technologies that are changing the fundamental role of the human combatant.
Methods
An international team of military performance researchers drew on relevant literature and their individual national perspectives and experiences to provide an integrated forecast of research priorities and needs based on current trends.
Results
Rapid advances and convergence in fields such as robotics, information technology and artificial intelligence will continue to have a revolutionary impact on the battlefield of the future. The disruption associated with these technologies will most acutely be experienced by the human combatant at the tactical level, with increasing cognitive demands associated with the employment and use of new capabilities. New research priorities may include augmented performance of humans-machine teams, enhanced cognitive and immunological resilience based on exercise neurobiology findings, and psychophysiological stress tolerance developed in realistic but safe synthetic training environments. Solving these challenges will require interdisciplinary research teams that have the capacity to work across the physical, digital and biological boundaries whilst collaborating seamlessly with end-users, human combatants. New research methodologies taking full advantage of sensing technologies will be needed to provide rigorous, evidence-based data in real and near-real world environments. Longer term research goals involving biological manipulation will be shaped by moral, legal and ethical considerations and evolving concepts of what it means to be human.
Conclusion
This paper outlines key recommendations to assist military human performance researchers to adapt their practice in order to match the increasing pace of military modernisation. By anticipating technological change and forecasting possible emerging technologies the military human performance research community can manoeuvre
to prioritise research activities today in line with future needs and requirements.
... Consumer-grade wearable devices increasingly incorporate onboard sensors that capture physiological measures and can enable the Cognitive State Assessment [4][5][6] required by any augmented cognition system. Wearable sensors can capture direct measurements of physiological processes such as heart rate, skin temperature, electrocardiogram (ECG), pulse oximetry, and respiration. ...
Physiological and behavioral sensors on consumer-grade wearable devices have great potential to inform augmented cognition applications in everyday living. However, lack of understanding of usability and function of new devices when embedded within larger technical systems is a barrier to their implementation in translational research. Practical evaluation methods are needed to overcome this fundamental barrier in a rapidly changing consumer-grade device landscape. We describe a stepwise evaluation methodology developed in our lab that is designed to rapidly evaluate and position consumer-grade sensors for use in larger studies.
... According to enactive cognition theories, the user's state of mind is a complex function of perceptual experience and cognitive task as both unfold over time [25]; therefore instead of increasing the number of sensors to gather information about user environment, we might be able to focus on sensor data that enables us to fix the terms of a cognitive model that will give us some information about the user's cognitive processes. The problem is that the most informative sensory systems such as functional magnetic resonance imaging (fMRI) and magneto-encephalography (MEG) are not deployable in a workspace environment due to their large size and heavy weight [23] and sensitivity to electrical and magnetic interference; less invasive systems such as electroencephalography and pupil dilation are not applicable to a web-based system in which users cannot be instrumented in this way. The only sensory information that is always available is the user's interaction. ...
... This research was based on the belief that a safer middle ground could be achieved by inferring and then adapting a workload and information delivery medium in order to influence the emotion of clinicians. Indeed, a number of studies in the areas of Augmented Cognition (AugCog) (Schmorrow & Kruse, 2002a) and Attentive User Interfaces (AUIs) (Vertegaal, 2003) have shown promise in this regard. Vertegaal described an AUI as a user interface that employs technology to monitor gestures, head position, and speech to infer the attributes of an individual's attention at a given moment. ...
... Clearly, eye-movement tracking can be employed in such studies; comparison of two or more adaptive systems or adaptive vs. non-adaptive system comparison with respect to the eye-movement data can be conducted. For example, eye-movement trackers allow for a measurement of cognitive workload, through the dilatations of the pupil [12]. These dilatations happen involuntary, and therefore provide an objective measure of users' cognitive processing and changes in the mental workload during competition of a task [6]. ...
Eye-movement tracking proved its potentials in many areas of human-computer interaction. Resting on a hypothesis that eye-direction and mind are linked, some of the HCI researchers have employed eye-movement trackers to investigate the visual attention focus of the participants completing their tasks. Others have used the eye-movement tracking in real-time applications, either as a direct interaction device or as an input to gaze-aware interfaces. Inspired by the previous HCI applications, we propose to utilize eye- movement trackers in adaptive systems research and development in two ways. First, the evaluations of adaptive systems could get an access to the information otherwise unavailable, as for instance to how the visual attention and cognitive processing are influenced by an adaptivity implemented into the evaluated system. Second, we propose to employ the eye-movement tracking technologies for a real-time registration of users' loci of visual attention, therefore increasing the awareness of the adaptive systems about their current users. We discuss possible potentials, difficulties and pitfalls of eye-movement tracking when applied to adaptive systems. We argue that a methodological framework of applying eye-tracking into adaptive systems shall be developed.
... Engineering approaches that design displays by human perceptual organization and chunking have demonstrated significant performance benefits (Wickens and Hollands, 2000). When assessing human information processing such as the use of attentional resources and managing cognitive load, the 'Augmented Cognition' research initiative has provided some guidance on the configuration and tempo of visual presentations (McBride and Schmorrow, 2005;Schmorrow and Kruse, 2002). 'Visual Analytics' is a newer term that refers to the convergent challenges of facilitating analytical reasoning through interactive visual interfaces (Johnson et al., 2006;Thomas and Cook, 2006). ...
Managing the layout of multi-dimensional visualizations is a crucial concern for the development of effective visual analytic interfaces. In these environments, heterogeneous and multi-dimensional information must be structured and combined into data representations that demand low cognitive resources but yield accurate mental models and insights. In this paper, we use Information-Rich Virtual Environments (IRVE) to articulate crucial tradeoffs in the use of Depth and Gestalt cues in text label layouts. We present a design space and evaluation methodology to explore the usability effects of these tradeoffs and collect results from a series of user studies. These lessons are posed as a set of design guidelines to aid developers of new, advantageous interfaces and specifications.
... Furthermore, operator performance can be increased, when it is possible to maintain the operator's cognitive load on an optimal level by adapting the way data is presented appropriately (Chen and Vertegaal, 2004;Fournier et al., 1999;Kohlmorgen et al., 2007). A lot of the research which follows this motivation for task demand estimation has been done in a military context, namely for the DARPA augmented cognition programme (Schmorrow and Kruse, 2002;St. John et al., 2004). ...
This paper attempts to explore the feasibility of classifying relaxed and stressful mental states based on two-channel prefrontal EEG signal from 35 healthy human subjects. Specific objective of this paper is to explore the best choice of features and compare the performance of various feature classification algorithms suitable for this purpose. Here, we included different bivariate features in time domain and frequency domain and compared the classification performance of artificial neural network, linear discriminant analysis, quadratic discriminant analysis (QDA), K nearest neighbour and support vector machine algorithms. Common spatial patterns (CSP) algorithm was used successfully for feature reduction. Best classification performance (99.69%) was observed with the QDA classifier taking cross-correlation estimate as feature. We also explored the effect of combining different kinds of features, effect of varying the number of features on classifier performance, robustness of the chosen methods against in inter-individual variability and the feasibility of developing subject-independent classifiers.
... In terms of augmented content [20], [21], this machinehuman communication is an augmented sensing, as a graphical overlay is presented to the user over real world scenery. Utilizing of the introduced methods, it is able to produce augmented and world-aligned content on the 2D screen as it was in the 3D space. ...
In this paper techniques are introduced for orien-tation estimation of wearable devices (like Google Glass) through bias compensation of gyroscope. The standard problem of using gyroscopes is that integration of raw angular rates with non-zero bias will lead to continuous drift of the estimated orientation. To examine the nature of this bias, a simple error model was constructed for the whole device in terms of inertial sensing. For eliminating the bias, a sensor fusion algorithm was developed using the benefits of optical flow from the camera of the device. Our orientation estimator and bias removal method is based on complementary filters, in combination with an adaptive reliability filter for the optical flow features. The feedback of the fused result is combined with the raw gyroscope angular rates to compensate the bias. Various measurements were recorded on a real device running the demanding optical flow onboard. This way a robust and reliable fusion was constructed, which matched our expectations, and has been validated with simulations and real world measurements.
... In recent studies [1][2][3], many researchers proposed to develop quantitative techniques for ongoing assessment of cognitive effort, engagement, and workload, by investigat-ing the neurobiological mechanisms underlying electroencephalographic (EEG) brain dynamics. A way to determine the relationship between different stimuli and human cognitive responses accompanying correct, incorrect, and absent motor responses is the use of event-related brain potential (ERP) signals. ...
We proposed an electroencephalographic (EEG) signal analysis approach to investigate the driver's cognitive response to traffic-light experiments in a virtual-reality-(VR-) based simulated driving environment. EEG signals are digitally sampled and then transformed by three different feature extraction methods including nonparametric weighted feature extraction (NWFE), principal component analysis (PCA), and linear discriminant analysis (LDA), which were also used to reduce the feature dimension and project the measured EEG signals to a feature space spanned by their eigenvectors. After that, the mapped data could be classified with fewer features and their classification results were compared by utilizing two different classifiers including k nearest neighbor classification (KNNC) and naive bayes classifier (NBC). Experimental data were collected from 6 subjects and the results show that NWFE+NBC gives the best classification accuracy ranging from 71%∼77%, which is over 10%∼24% higher than LDA+KNN1. It also demonstrates the feasibility of detecting and analyzing single-trial EEG signals that represent operators' cognitive states and responses to task events.
... Human multitasking performance is growing evermore important in many sectors of society. The military researches the enhancement of human performance for defense applications [1]- [3]. Multitasking is also a critical component of the medical field; recently, much research has focused on enhancing the performance of surgeons, anesthesiologists, and nurses [4]- [6]. ...
Human multitasking performance is important in many areas, such as defense, medicine, and everyday life. However, multitasking can be difficult to research due to a lack of objective metrics. This gap is remedied by applying information theory-based models to multitasking systems. The human operator informatic model is a throughput model that has been successfully applied to the multiple attribute task battery (MATB) software. In this study, auditory and haptic cueing were applied to the monitoring and targeting components of MATB, respectively. Interestingly, overall information throughput was not significantly affected by the cueing. These results can be traced to a mathematical change in operator strategy. In the presence of multisensory cueing, operators responded at higher information rates to the monitoring and targeting components; however, this came at a proportional cost to the communications and resource components. We propose that each operator has an information throughput capacity—an asymptotic limit to the amount of information he/she can process. This theoretical limit is analogous to the “channel capacity” for single tasking proposed by Miller in 1956.
... past decade (Blankertz et al., 2010; Zander et al., 2009), and, at the same time, the concept of Augmented Cognition (AugCog) was borne out of the Defense Advanced Research Projects Agency's (DARPA) pushing for technologies that enhanced the Warfighter's communication skills and those technologies that involved biosensors for medical applications (Schmorrow, 2002). In this context, the most studied mental state, due to its strong relationship with the increasing or the degrading of user's performance, has been the mental workload. Mental workload is a complex construct that is assumed to be reflective of an individual's level of attentional engagement and mental effort (Wickens, 1984 ). Measurement ...
In the last decades, it has been a fast-growing concept in the neuroscience field. The passive brain–computer interface (p-BCI) systems allow to improve the human–machine interaction (HMI) in operational environments, by using the covert brain activity (eg, mental workload) of the operator. However, p-BCI technology could suffer from some practical issues when used outside the laboratories. In particular, one of the most important limitations is the necessity to recalibrate the p-BCI system each time before its use, to avoid a significant reduction of its reliability in the detection of the considered mental states. The objective of the proposed study was to provide an example of p-BCIs used to evaluate the users’ mental workload in a real operational environment. For this purpose, through the facilities provided by the École Nationale de l’Aviation Civile of Toulouse (France), the cerebral activity of 12 professional air traffic control officers (ATCOs) has been recorded while performing high realistic air traffic management scenarios. By the analysis of the ATCOs’ brain activity (electroencephalographic signal—EEG) and the subjective workload perception (instantaneous self-assessment) provided by both the examined ATCOs and external air traffic control experts, it has been possible to estimate and evaluate the variation of the mental workload under which the controllers were operating. The results showed (i) a high significant correlation between the neurophysiological and the subjective workload assessment, and (ii) a high reliability over time (up to a month) of the proposed algorithm that was also able to maintain high discrimination accuracies by using a low number of EEG electrodes (~ 3 EEG channels). In conclusion, the proposed methodology demonstrated the suitability of p-BCI systems in operational environments and the advantages of the neurophysiological measures with respect to the subjective ones.
... Preventing such accidents is thus a major focus of efforts in the field of active safety research in vehicle safety-driving systems. In recent studies [1]- [3], many researchers had proposed to develop quantitative techniques for ongoing assessment of cognitive effort, engagement and workload, by investigating the neurobiological mechanisms underlying electroencephalographic (EEG) brain dynamics. In these studies, brain event-related potential (ERP) signals were used to determine the relationship between different stimuli and human cognitive responses corresponding to correct/incorrect motor reactions. ...
Accidents caused by errors and failures in human performance among traffic fatalities have a high death rate and become an important issue in public security. They are mainly caused by the failures of the drivers to perceive the changes of the traffic lights or the unexpected conditions happening accidentally on the roads. In this paper, we devised a quantitative analysis for assessing driver's cognitive responses by investigating the neurobiological information underlying electroencephalographic (EEG) brain dynamics in traffic-light experiments in a virtual-reality (VR) dynamic driving environment. The VR technique allows subjects to interact directly with the moving virtual environment instead of monotonic auditory and visual stimuli, thereby provides interactive and realistic tasks without the risk of operating on an actual machine. Independent component analysis (ICA) is used to separate and extract noise-free ERP signals from the multi-channel EEG signals. A temporal filter is used to solve the time-alignment problem of ERP features and principle component analysis (PCA) is used to reduce feature dimensions. The dimension-reduced features are then input to a self-constructing neural fuzzy inference network (SONFIN) to recognize different brain potentials stimulated by red/green/yellow traffic events, the accuracy can be reached 87% in average eight subjects in this visual-stimuli ERP experiment. It demonstrates the feasibility of detecting and analyzing multiple streams of ERP signals that represent operators' cognitive states and responses to task events.
... The focus of augmented cognition systems is real-time assisted human interaction with complex computer systems based on biofeedback detection and manipulation [40]. The efficacy of this scientific approach to human abilities enhancement has been widely proved over more than a decade of independent research at DARPA [34,35,41]. ...
Building on augmented cognition theory and technology, our novel contribution in this work enables accelerated, certain brain functions related to task performance as well as their enhancement. We integrated in an open-source framework, latest immersive virtual reality (VR) head-mounted displays, with the Emotiv EPOC EEG headset in an open neuro- and biofeedback system for cognitive state detection and augmentation. Our novel methodology allows to significantly accelerate content presentation in immersive VR, while lowering brain frequency at alpha level—without losing content retention by the user. In our pilot experiments, we tested our innovative VR platform by presenting to N = 25 subjects a complex 3D maze test and different learning procedures for them on how to exit it. The subjects exposed to our VR-induced entrainment learning technology performed significantly better than those exposed to other “classical” learning procedures. In particular, cognitive task performance augmentation was measured for: learning time, complex navigational skills and decision-making abilities, orientation ability.
... A More Practical Cutting-edge. A later but perhaps more practical part of augmentation history comes from DARPA's Augmented Cognition project (Schmorrow and Kruse, 2002). As a legacy of Douglas Englebart's work in the 1960's (Englebart, 1962) and "Decade of the Brain" initiatives in the 1990s, Augmented Cognition (AC -Schmorrow and Stanney, 2009)) became an integral part of the Human-Computer Interaction (HCI) field. ...
Human Augmentation (HA) spans several technical fields and methodological approaches, including Experimental Psychology, Human-Computer Interaction, Psychophysiology, and Artificial Intelligence. Augmentation involves various strategies for optimizing and controlling cognitive states, which requires an understanding of biological plasticity, dynamic cognitive processes, and models of adaptive systems. As an instructive lesson, we will explore a few HA-related concepts and outstanding issues. Next, we focus on inducing and controlling HA using experimental methods by introducing three techniques for HA implementation: learning augmentation, augmentation using physical media, and extended phenotype modeling. To conclude, we will review integrative approaches to augmentation, which transcend specific functions.
... A More Practical Cutting-edge. A later but perhaps more practical part of augmentation history comes from DARPA's Augmented Cognition project (Schmorrow and Kruse, 2002). As a legacy of Douglas Englebart's work in the 1960's (Englebart, 1962) and "Decade of the Brain" initiatives in the 1990s, Augmented Cognition (AC -Schmorrow and Stanney, 2009)) became an integral part of the Human-Computer Interaction (HCI) field. ...
Human Augmentation (HA) spans several technical fields and methodological approaches, including Experimental Psychology, Human-Computer Interaction, Psychophysiology, and Artificial Intelligence. Augmentation involves various strategies for optimizing and controlling cognitive states, which requires an understanding of biological plasticity, dynamic cognitive processes, and models of adaptive systems. As an instructive lesson, we will explore a few HA-related concepts and outstanding issues. Next, we focus on inducing and controlling HA using experimental methods by introducing three techniques for HA implementation: learning augmentation, augmentation using physical media, and extended phenotype modeling. To conclude, we will review integrative approaches to augmentation, which transcend specific functions.
... Third, the proposed model can contribute to prevent both disuse and misuse of automation by monitoring the status of operators' trust calibration towards automation. Also, one of the main goals of the augmented cognition (AC) program is to develop robust, noninvasive, real-time, cognitive state detection technology for measuring the cognitive processing state of the user (e.g., trust calibration) in cognitively challenging environments (Schmorrow & Kruse, 2002), such as situations where humans interact with automation. Therefore, this framework to detect trust calibration in automation will lead to improvement of detecting cognitive states, thus contributing to Augmented Cognitive systems. ...
... Reporting on building risk resilience directly in the aerospace manufacturing sector, change was identified as an expectation, with one being "…ready to go to Plan B if Plan A is not available, and then move on to consider Plans C and D, and perhaps Plan E if circumstances dictate" [64]. In terms of Big Data and automation technologies in aircraft the need for the humans to adapt more fluidly are significant in the sense of changing and working through times of sudden disorder and uncertainty [65], [66]. ...
Functional near-infrared spectroscopy (fNIR), a non-invasive neuroimaging modality designed to monitor the hemodynamic change, can help identify neural correlates of human brain functioning mediated by different events. In this thesis, fNIR has been used to monitor prefrontal cortical activity with the primary objective to determine a set of neurophysiological markers that detect changes in neural activation elicited by levels of mental engagement. Two studies were selected to assess change in the cognitive state of mental engagement at both high and low levels of neural activation. At the high end of neural activation, human performance studies were conducted to assess cognitive workload, in particular signal changes associated with overload. At the low end of cognitive engagement, the capacity of fNIR to detect changes associate with the depth of anesthesia was investigated using patients undergoing general anesthesia. In the human performance study, participants were cognitively challenged by a complex task. By contrast, in the anesthetic depth assessment study, cognitive activity was deliberately suppressed by anesthetic agents. In both studies, neurophysiological markers of hemodynamic changes were extracted from the fNIR measurements. The hypothesis underlying the human performance study is the positive correlation of blood oxygenation in the prefrontal cortex with increasing task difficulty and sustained cognitive effort. In addition, increased blood oxygenation demonstrates a positive relationship with behavioral performance measures in this task. A naval air warfare management and control task with varying levels of difficulty has been chosen to test this engagement condition. The results of this study showed that changes in blood oxygenation in relevant areas of the prefrontal cortex are associated with increasing cognitive workload, defined as sustained attention in a verbal and spatial working memory and decision-making task. The results suggest a reliable, positive association between cognitive workload and increases in cortical oxygenation responses (r=0.89 & p<0.001). The data analysis also supports the hypothesis that the rate of oxygenation change in the dorsolateral prefrontal cortex as measured by fNIR can provide an index of sustained attention in a complex working memory and decision-making task. Furthermore, this study reveals that an abrupt drop in the rate of oxygenation change in dorsolateral prefrontal cortex under high workload conditions is associated with a user’s decline in performance. Awareness is an unintended mental state during general anesthesia. An accurate, objective measure of return to consciousness would provide an important safeguard for patients and physicians alike. This exploratory investigation on predicting awareness under general anesthesia examines the hypothesis that the transition from deep to light anesthetic stages is associated with reliable changes in oxygenated, deoxygenated, and total hemoglobin in frontal cortex. Hemodynamic changes during deep and light anesthesia were examined in 26 patients. The results suggest that the rate of deoxygenated hemoglobin change can be used as a descriptive neuromarker to differentiate between deep and light anesthesia stages (F1,25 = 7.61, p<0.01). This marker is proposed for further development as an index of the depth of anesthesia for the purpose of monitoring awareness under general anesthesia. In addition to the neuropsychological findings, this research demonstrates engineering and signal processing solutions in the form of customized algorithms and procedures that allow fNIR to measure usable signals under field conditions. Independent and principal component analyses (ICA, PCA) were combined in a novel procedure that employed dark current (i.e., signal from non-cortical sources) as a reference measurement. This method provided improved signal-to-noise ratio for the hemodynamic measurements acquired in the operating room, and can be used to increase the signal quality of fNIR for many other applications and field situations.
During the past years, the growing number of traffic fatalities has become an important issue in public security. In this paper, we develop a quantitative analysis for ongoing assessment of cognitive response by investigating the neurobiological brain dynamics in traffic-light experiments. A single-trial event-related-potential (ERP)-based fuzzy neural network (FNN) is applied to recognize different brain potentials stimulated by red/green/yellow traffic-light events. The system consists of a dynamic virtual-reality (VR)-based motion simulation platform, EEG signal detection and analysis units, and FNN-based classifier. ICA algorithms are used to obtain noise-free ERP signals from the multi-channel EEG signals. A novel temporal filter is also proposed to solve time-alignment problems of ERP features and PCA is used to reduce feature dimensions, which were then fed into a FNN classifier. Experimental results demonstrate the feasibility of detecting and analyzing multiple streams of ERP signals that organize operators' cognitive responses to task events. Comparisons of three kinds of linear and nonlinear classifiers show that our proposed FNN-based classifier can achieve a satisfactory and superior recognition rate (85%). The classification results can be further transformed as the control/biofeedback signals of intelligent driving systems.
The operational characteristics of functional optical brain were investigated using near-infrared during cognitive tasks. A symbiotic relation between operator and operational environment can be realized by an advanced computing platform designed to adapt to cognitive and physiological state of user. The experimental protocol used a complex task, resembling a video game, called the Warship Commander Task (WCT). The WCT was designed to approximate naval air warfare management. The functional near-infrared (fNIR) data analysis explored relations among cognitive workload, participant's performance and effect of divided attention as manipulated by secondary component of WCT.
The battlefield of the future will require the warfighter to multitask in numerous ways, seriously taxing the cognitive and perceptual capabilities of even the most advanced warrior. A principal concern in developing a better understanding of how current and proposed computational technologies can supplement and augment human performance in this and other environments is determining when such assistance is required. This challenge can be parsed into 2 components: determining what set of measurements accurately reflects cognitive state, and identifying techniques for synthesizing this set of measurements into a single collective cognitive state variable. The primary thesis of this proposal is that automatic human behavioral responses serve as inherent probes for cognitive state. Further, the human perception-action system is uniquely designed to capture, process, integrate, and act on an extraordinarily diverse range of information freely available in the natural environment. Together, this system and the surrounding environment which acts on it-and on which the system acts-form a dynamic coupling. Under normal conditions these couplings remain intact. When stressed, these couplings become degraded. Based on this understanding, the authors propose a unique suite of Cognitive Workload Assessment (CWA) tools, based on real-time measurements of postural control that can serve as both a stand-alone indicator of cognitive state as well as a cueing filter for engaging other CWA sensor suites that are currently under development.
This research proposes a conceptual model to make significant progress in augmented cognition. We intend to design a generic architecture to improve the collective situational awareness (SA) of a team of soldiers on the battlefield during combat effects, and cultural communications. It also establishes a training outline for future landscapes of a similar nature by providing convincing feedback. It will serve as architecture of a model, based on the current laboratory framework. This involves data analysis of the feedback from the neurophysiological body sensors (heart rate, skin temperature, breathing rate and others) placed on a wearable immersive training vest. It includes gesture tracking gloves and fixed spatial locators. Approximately 15 soldiers will be incorporated into the study. Various training scenarios will be designed to capture all possible body perturbations, movements, and interactions. A similar agent-based model (ABM) of the training environment will be created to augment the soldier's cognitive competencies. ABM will integrate social systems interaction complexity into the decision makin g capability of the soldier in real time. Automated feedback can be provided using the model developed in the event of any social faux pas via gesture tracking and monitoring cognitive capacity. Model can help discover novel talents and render knowledge during training scenarios.
Complex systems are characterized by a large number and variety of, and often a high degree of dependency between, subsystems.
Complexity, in combination with coupling, has been shown to lead to difficulties with monitoring and comprehending system
status and activities and thus to an increased risk of breakdowns in human-machine coordination. In part, these breakdowns
can be explained by the fact that increased complexity tends to be paralleled by an increase in the amount of data that is
made available to operators. Presenting this data in an inappropriate form is crucial to avoiding problems with data overload
and attention management. One approach for addressing this challenge is to move from fixed display designs to adaptive information
presentation, i.e., information presentation that changes as a function of context. This paper will discuss possible approaches
to, challenges for, and effects of increasing the flexibility of information presentation.
Accidents caused by errors and failures in human performance among traffic fatalities have a high rate causing death and become an important issue in public security. The key problem causing these car accidents is mainly because that the drivers failed to perceive the changes of the traffic lights or the unexpected conditions happening accidentally on the roads. In this paper, we devised a quantitative analysis for ongoing assessment of driver's cognitive responses by investigating the neurobiological information underlying electroencephalographic (EEG) brain dynamics in traffic-light experiments in a virtual-reality (VR) dynamic driving environment. Three different feature extraction methods including nonparametric weighted feature extraction (NWFE), principal component analysis (PCA), discriminant analysis feature extraction (DAFE) are applied to reduce the feature dimension and project the measured EEG signals to a feature space spanned by their eigenvectors. After that, the mapped data can be classified with fewer features and their classification results are compared by utilizing three different classifiers including Gaussian classifier (GC), k nearest neighbor classification (KNNC), and naive Bayes classifier (NBC). Experimental results show that the successful rate of nonparametric weighted feature extraction combined with Gaussian classifier is higher more than 10% compared with other combinations. It also demonstrates the feasibility of detecting and analyzing single-trail ERP signals that represent operators' cognitive states and responses to task events.
In this paper, we propose a new multiple sensory fused human identification model for providing human augmented cognition.
In the proposed model, both facial features and mel-frequency cepstral coefficients (MFCCs) are considered as visual features
and auditory features for identifying a human, respectively. As well, an adaboosting model identifies a human using the integrated
sensory features of both visual and auditory features. In the proposed model, facial form features are obtained from the principal
component analysis (PCA) of a human’s face area localized by an Adaboost algorithm in conjunction with a skin color preferable
attention model. Moreover, MFCCs are extracted from human speech. Thus, the proposed multiple sensory integration model is
aimed to enhance the performance of human identification by considering both visual and auditory complementarily working under
partly distorted sensory environments. A human augmented cognition system with the proposed human identification model is
implemented as a goggle type, on which it presents information such as unknown people’s profile based on human identification.
Experimental results show that the proposed model can plausibly conduct human identification in an indoor meeting situation.
The OpenAnswer system has the goal of exploiting teacher mediated peer-assessment for the evaluation of answers to open ended questions. The system models both the learning state of each student and their choices during peer-assessment. In OpenAnswer, each student is represented as a Bayesian network made of a triple of finite-domain variables: K for student’s Knowledge about a topic, J for the estimated ability to evaluate ("Judge") the answer of another peer, C for Correctness of the answer to a given question. The student's individual sub-networks are connected through further Bayesian variables which model each peer-assessment choice, depending on the type of peer-assessment performed: (G for grading, B for choosing the best, W for choosing the worst).
During an assessment session, each student grades a fixed number of peers’ answers. The final result for a given session is a full set of grades for all students’ answers, although the teacher had actually graded only a part of them. The student's assessments are instantiated in the network as evidence, together with the teacher's (perhaps partially complete) grades, so that OpenAnswer deduces the remaining grades. In the former OpenAnswer implementation, all variables were represented through a probability distribution over three values (Good/Fair/Bad for K and J, correct/fair/wrong for C). We present experiments and simulations showing that, by increasing the domain granularity for all variables from 3 to 6 values (A to F), the information obtained from the Bayesian network achieves higher reliability.
Wearable Augmented Reality (AR) combines research in AR, mobile/ubiquitous computing, and human ergonomics in which a video or optical see-through head mounted display (HMD) facilitates multi-modal delivery of contextually relevant and computer generated visual and auditory data over a physical, real-world environment. Wearable AR has the capability of delivering on-demand assistance and training across a variety of domains. A primary challenge presented by such advanced HCI technologies is the development of scientifically-grounded methods for identifying appropriate information presentation, user input, and feedback modalities in order to optimize performance and mitigate cognitive overload. A proposed framework and research methodology are described to support instantiation of physiologically-driven, adaptive AR to assess and contextually adapt to an individual’s environmental and cognitive state in real time. Additionally a use case within the medical domain is presented, and future research is discussed.
In this paper we propose and examine a robust method for orientation estimation of wearables (like Google Glass) through bias compensation of gyroscope. Through integration of raw angular rates, the resulting orientation estimate will drift, due to the non-zero bias of the measured gyroscope was also accumulated. A simple error model was constructed for the measurement capabilities of the device, in terms of inertial sensing of rotation. Using the benefits of feature point displacements (as optical flow) from the camera of the device, a sensor fusion algorithm was developed to lower, and compensate the bias. The introduced orientation estimator and bias removal method is based on adaptive reliability filter for the optical flow feature points. To fuse the remaining feature point displacements into one single value, various aggregation methods have been tested, and a maximum a-posteriori estimator was applied on the dataset. To further lower the bias from the gyroscope, the output of the system was used as a feedback to calculate a bias estimate. To measure the performance and investigate the benefits of the asynchronous work-flow, the optical flow was performed on various devices in real time. To validate the results of the simulations, real world measurements were performed with industrial robots. Our reliable, adaptive filter matched our expectations, could compensate the bias well, and the integrated output of the system had no significant drift.
The field of Neuroergonomics sits at the interface between humans’ brains and humans’ working environment with an aim to improve the work humans do. In this chapter, I will review the scientific journey that led to the birth of Neuroergonomics. The journey starts from the Biocybernetics and Brain–Computer Interfaces projects in 1960s, followed by work on adaptive aiding in 1970s, and all the way to early 2000s with work on Augmented Cognition and Neuroergonomics. An extension to Neuroergonomics gave birth to Cognitive-Cyber Symbiosis, whereby Neuroergonomics is augmented with artificial intelligence agents that act as relationship managers between the human brain and the information-centric work environment. Some challenges facing these fields today are then discussed using a human-swarm teaming lens.
Functional near infrared (fNIR) sensor measures hemodynamic changes in the cortex using a portable, safe, affordable and negligibly intrusive NIR based optical brain imaging system. We hypothesize that there is a positive correlation between blood oxygenation in the relevant areas of dorsolateral prefrontal cortex and cognitive effort defined as attention and working memory. In this paper, we present the employment of the fNIR sensor to assess the cognitive state of the user. The paper also introduces the experimental setup and analysis of the fNIR data acquired from the forehead during a video-game like task namely, "Warship Commander Task" (WCT). In the WCT, the primary task is air warfare management. While performing the air warfare task, task difficulty and task load are manipulated by changing the number of tracks per wave and the number of more difficult "yellow" tracks. We explored in this paper, the correlation between cognitive task load, the subject's performance and dorsolateral prefrontal cortex oxygenation level changes. Our data analysis results show that fNIR blood oxygenation changes are significantly sensitive to wave size (task load) changes.
ResearchGate has not been able to resolve any references for this publication.