Conference PaperPDF Available

Measuring vigilance decrement using computer vision assisted eye tracking in dynamic naturalistic environments

Authors:

Abstract and Figures

Eye tracking offers a practical solution for monitoring cognitive performance in real world tasks. However, eye tracking in dynamic environments is difficult due to high spatial and temporal variation of stimuli, needing further and thorough investigation. In this paper, we study the possibility of developing a novel computer vision assisted eye tracking analysis by using fixations. Eye movement data is obtained from a long duration naturalistic driving experiment. Source invariant feature transform (SIFT) algorithm was implemented using VLFeat toolbox to identify multiple areas of interest (AOIs). A new measure called `fixation score' was defined to understand the dynamics of fixation position between the target AOI and the non target AOIs. Fixation score is maximum when the subjects focus on the target AOI and diminishes when they gaze at the non-target AOIs. Statistically significant negative correlation was found between fixation score and reaction time data (r =-0.2253 and p<;0.05). This implies that with vigilance decrement, the fixation score decreases due to visual attention shifting away from the target objects resulting in an increase in the reaction time.
Content may be subject to copyright.
Measuring Vigilance Decrement using Computer Vision Assisted Eye
Tracking in Dynamic Naturalistic Environments.
Indu P. Bodala1,Student Member, IEEE, Nida I. Abbasi2,Student Member, IEEE, Yu Sun3,
Member, IEEE, Anastasios Bezerianos4,Senior Member, IEEE, Hasan Al-Nashash5,Senior Member, IEEE
and Nitish V. Thakor6,Fellow, IEEE
Abstract Eye tracking offers a practical solution for mon-
itoring cognitive performance in real world tasks. However,
eye tracking in dynamic environments is difficult due to high
spatial and temporal variation of stimuli, needing further and
thorough investigation. In this paper, we study the possibility
of developing a novel computer vision assisted eye tracking
analysis by using fixations. Eye movement data is obtained
from a long duration naturalistic driving experiment. Source
invariant feature transform (SIFT) algorithm was implemented
using VLFeat toolbox to identify multiple areas of interest
(AOIs). A new measure called ‘fixation score’ was defined to
understand the dynamics of fixation position between the target
AOI and the non target AOIs. Fixation score is maximum when
the subjects focus on the target AOI and diminishes when they
gaze at the non-target AOIs. Statistically significant negative
correlation was found between fixation score and reaction time
data (r=0.2253 and p<0.05). This implies that with vigilance
decrement, the fixation score decreases due to visual attention
shifting away from the target objects resulting in an increase
in the reaction time.
I. INTRODUCTION
Measuring vigilance decrement with time has always been
a challenge [1], [2]. The physiological or behavioural vari-
ables that are used for this purpose must be sensitive enough
to measure the changes in individual subject behaviour
with time and robust enough to apply them across various
tasks and applications [3]. Eye tracking provides a practical
solution that can be applicable in natural environments to
measure changes in vigilance. Eye movement data can pro-
vide rich information about the dynamics of perception and
attention in both temporal and spatial domains. For example,
*This work was supported by the National University of Singapore for
Cognitive Engineering Group at Singapore Institute for Neurotechnology
(SINAPSE) under Grant R-719-001-102-232.
1Indu P. Bodala is a Ph.D. student in the faculty of NUS Graduate
School of Integrative Science and Engineering (NGS), National University
of Singapore, Singapore. indu11@u.nus.edu
2Nida I. Abbasi is Master’s student in the department of
Biomedical Engineering, National University of Singapore, Singapore.
nida.itratabbasi@u.nus.edu
3Yu Sun is a senior research fellow in the Cognitive Engineering
group at the Singapore Institute for Neurotechnology (SINAPSE), National
University of Singapore, Singapore. kissalladin@gmail.com
4Anastasios Bezerianos is the head of Cognitive Engineering group at the
Singapore Institute for Neurotechnology (SINAPSE), National University of
Singapore (NUS), Singapore. tassos.bezerianos@nus.edu.sg
5Hasan Al-Nashash is a professor in the department of Electrical Engi-
neering, American University of Sharjah, UAE. hnashash@aus.edu
6Nitish V. Thakor is the director of the Singapore Institute of Neurotech-
nology (SINAPSE), National University of Singapore (NUS), Singapore.
eletnv@nus.edu.sg
duration of fixations or saccades can reflect the amount of
attention towards target stimuli [4], [5], [6]. Similarly, the
frequency of eye blinks was used as a measure of workload
experienced in a task [7], [8]. Other complex measures such
as gaze path, fixation heat map and analysis of eye movement
clusters were developed to provide intuitive insights about
viewing patterns of participants [9], [10].
However, analysis of the eye movement data collected in
a naturalistic environment for long duration is a difficult
problem. Tasks conducted in non-conventional, real world
scenarios like driving or surveillance always include many
necessary and unnecessary stimuli [8]. Real world tasks com-
prise dynamic stimuli where the areas of interest constantly
change unlike static stimuli usually used in the laboratory
settings to study cognitive performance. Hence, it is very
important to develop methods to analyze data collected in
such environments to enable interventions that can measure
and enhance performance in everyday tasks.
In this study, we designed a driving experiment where the
subjects performed a specific braking task in an immersive
virtual environment. The task is designed as close as possible
to the naturalistic driving conditions. In the braking task,
subjects were asked to follow a ‘lead car’ on a country road.
In addition to the lead car which acted as the target stimulus,
there were several non-target stimuli such as oncoming traffic
in the next lane, curvy roads and rich scenery with moun-
tains, grass and houses to the side of the road. Some of these
non-target stimuli may serve as distractors (for example,
scenery) while some may still have significance (for example,
oncoming traffic) with respect to the task objective. Eye
movement data of the participants was collected throughout
the task which lasted for about 25 minutes.
To analyse the eye movement data, we adopted areas-of-
interest (AOI) analysis approach. Analysis of AOIs is very
useful in understanding the dynamics of visual attention
in the environments with stimuli distributed and varying
spatially. Metrics such as dwell time, AOI hit and first return
were used by researchers to understand visual perception,
attention shifts and the relation to performance [11], [12].
Since the stimulus in our experiment was a dynamic scene,
the spatial locations of the AOIs varied. Hence we used
a computer vision algorithm to obtain object locations in
each frame of the video stimulus. Also, these AOIs were
developed based on the semantic characteristics of the objects
defined with respect to the task objectives. Fixations were
(a)
(b)
(c)
(d)
Figure 1. (a) An arbitrary frame with the corresponding fixation overlaid. (b) Binary mask for the lead car with mask value
m1, (c) Binary mask for the oncoming traffic with mask value m2 and (d) Binary mask for the control deck and the mirror
with mask value m3 with the corresponding fixation overlaid were obtained from each frame. In this example, m1 is 1 and
m2 and m3 are 0.
then analyzed based on their spatial location and to which
AOIs they belonged.
In the following section, the details of the experiment
task, data collection, identifying dynamic AOIs and analysis
of fixations are discussed in detail. The results of the data
analysis are shown in section 3. The implications of the
findings and the future directions are discussed in section
4 followed by concluding remarks in section 5.
II. METHODS
A. Participants
Six healthy subject aged 20-35 years, with normal or
corrected-to-normal vision and with no previous history of
nervous or psychiatric disorders were recruited from the
National University of Singapore (NUS) to participate in
the study. All the subjects possessed a valid driving license.
The study was approved by the Institutional Review Board
(IRB) of NUS. Written informed consent was obtained from
each participant before the beginning of the experiment. All
participants received monetary compensation for their time
upon the completion of the experiment.
B. Experimental task
The driving experiment was designed using an immersive,
virtual environment based driving simulator where the sub-
jects performed a specific braking task. All the experiments
were conducted at the same time of the day during afternoon.
Subjects were instructed to drive continuously for 25 mins on
a country road on a sunny day setting where they followed
a lead car without over-taking it (Figure 1.(a)). When the
lead car would brake at infrequent intervals, the subjects
responded by braking to avoid a crash. A trial was defined
as the interval from the start of braking by the lead car to the
start of braking by the subject for which the reaction time
(RT) was calculated. The inter-trial interval varied randomly
between 45-75 seconds. The speed control was automatically
handled by the car in order to minimize variability between
subjects’ driving habits. However, the subjects controlled the
steering to make the car stay in the given lane. The subjects
were strictly instructed to avoid any crashes with the lead
car and the oncoming traffic.
C. Eye tracker data acquisition and preprocessing
While the subjects were performing the driving task, eye
movement data was collected using Tobii TX300 eye tracker
in a standalone external video setup at a sampling rate of 300
Hz. The subjects’ eyes were calibrated to the setup before
starting the experiment. Tobii Studio software was used to
simultaneously capture stimulus video and eye movements
for further analysis. Raw data obtained from the experiment
was classified into fixations and saccades using the I-VT
fixation filter available in Tobii studio. The algorithm for eye
movement classification was explained in [13]. I-VT fixation
filter classifies eye movement data based on velocity of the
directional shifts of the eye. If the velocity of a particular
sample is above a certain threshold, then it is classified
as a saccade. Otherwise it is classified as a fixation. The
velocity threshold was set to be 300/sec [14]. In addition to
classifying the samples, the I-VT filter algorithm also had a
gap fill-in function to extrapolate missing data samples and
noise reduction before classifying eye movement data. It will
also look for broken fixations and merge them and/or discard
short fixations after the eye movement classification. Markers
to indicate the start of the trial (braking of lead car) and the
response (braking by the subject) were introduced into the
recordings. Gaze data and video segments corresponding to
each trial were extracted using these markers.
D. Obtaining AOIs and fixation scores
1) Object detection using SIFT algorithm: The video seg-
ments corresponding to each trial were divided into frames
at the rate of 5 Hz. Each frame was then processed to
identify AOIs that fall into predefined semantic categories.
The objects present in the experiment were broadly divided
into four semantic categories namely lead car, traffic, scenery
and mirror and control deck. We used source invariant
feature transform (SIFT) algorithm from VLFeat toolbox
for MATLAB to identify the lead car and the oncoming
traffic in each frame [15]. SIFT algorithm comprises a feature
detector and feature descriptor. SIFT detector extracts a
number of frames (attributed regions) from an image in a
way that is consistent with variations in the illumination,
viewpoint and other viewing conditions of the given template
(corresponding to the object to be identified). SIFT descriptor
associates a signature to the regions which will be used in
identifying their appearance in a compact and robust manner.
We manually checked each frame for correct object identifi-
cation and manually annotated those with wrongly identified
objects. It is to be noted that the algorithm identified the
objects accurately in 94% of the total number of frames.
Binary masks, where 1s were assigned to the identified object
and 0s to the background, were generated to denote the
identified objects (Figure 1.(b), (c)). Since the coordinates of
the car’s mirror and the control deck were constant in each
frame, we used only one binary mask to represent them for
all the frames (Figure 1.(d)). The rest of the scene is marked
as scenery. Mask value for each mask in (m1,m2,m3as
explained in Figure 1) is defined as the pixel value of each
mask (0 or 1) at the fixation position.
2) Fixation scores: Amongst the above defined semantic
categories, the highest weightage was given to the lead
car (s1=1), since it is the target, followed by the traffic
(s2=0.5), the mirror and the control deck (s3=0.25). Each
fixation position obtained from the preprocessed gaze data
were compared against the binary mask to identify the AOIs
which contained it. Fix score is defined for a fixation as the
average of the product of semantic weights of the AOIs that
contained the fixation as given in (e1). It is to be noted that
if the fixation belongs to the scenery, the fix score will be 0
as all the mask values will be 0. The average of Fix score
for all the fixations in a particular trial is taken as the ‘trial
fixation score’ as given in (e2).
Fix score =(si×mi)
(mi)(e1)
where i=1,2,3.
Tr ial Fixat ion Score =(Fix score j)
n(e2)
where j=1,..,nand nis the number of fixations in that
trial.
III. RESULTS
In this section, we present the results from the analysis
of the reaction time and eye tracking data. The 25 mins
reaction time data is divided into two parts. The first 10 mins
data is defined as ‘baseline’ where the vigilance decrement is
assumed to be minimum. The last 15 mins data is assumed
to reflect higher vigilance decrement. ANOVA analysis is
performed for the average reaction time for all the subjects
for baseline vs. vigilance decrement data. Figure 2 shows
that there is a statistically significant (p<0.05) increase
in the reaction time from the baseline (mean =0.994sand
std dev =0.122s) to the last 15 mins data (mean =1.325s
and std dev =0.24s) indicating that the driving task success-
fully simulated vigilance decrement among subjects.
Figure 2. ANOVA analysis of the average reaction time for
all the subjects for the first 10 mins (baseline) and the last
15 mins (vigilance decrement). pval ue <0.05
Fixation scores are higher when the subjects focus more
on the lead car (target) than on the non-targets. Correlation
analysis between the fixation score and reaction time across
all trials for all the subjects is conducted. Figure 3 shows
statistically significant negative correlation between fixation
score and the reaction time (r=0.2253 and p<0.05).
This agrees with our assumption that the increase in reaction
time is due to the subjects gazing at AOIs which are
not significant for task objectives. This implies that, with
vigilance decrement, the visual attention wanders away from
the target objects to non-target objects.
IV. DISCUSSION
A. Computer vision assisted eye tracking
In this paper, we discussed about the importance of devel-
oping eye tracking metrics that can accommodate dynami-
cally varying visual environments for assessing performance
Figure 3. Correlation analysis between Trial fixation score
and Reaction time. Correlation coefficient (r) =0.2253 and
pvalue <0.05.
of subjects during naturalistic tasks. Eye trackers can be
easily integrated into natural environments to assess the
cognitive performance of individuals. However, there are
very few studies that investigated dynamic AOIs in natural
stimuli [16]. Although these techniques are incorporated into
modern commercially available softwares, the accuracy of
these techniques is limited for stimuli such as in this study
due to various confounding factors such as low contrast be-
tween object and background, multiple objects of interest and
sudden shifts in the position of the cars due to sharp curves in
the road. Hence, it is necessary to identify sophisticated and
robust computer vision algorithms that can identify objects
in dynamic environments with higher accuracy and can learn
semantic importance depending on the task objectives.
B. Measuring vigilance decrement in dynamic environments
With decrement in vigilance, subjects’ focus wanders away
from the task objectives [17]. This will result in the gaze data
shifting from target objects to distractors and a decrease in
the efficiency of the task relevant visual trajectories. Hence,
the decrease in fixation score is followed by an increase in
the reaction time. Even though the correlation is significant,
the lower value of correlation coefficient may be due to
low sample size or any possible outliers. This approach
has facilitated us to understand the dynamics of attention
across multiple dynamic AOIs which is a common scenario
in naturalistic tasks. Other AOI related features such as dwell
time, transition between AOIs or time to return target AOIs
can be further explored to develop a real time monitoring
system. Also, the sample size will be improved in the future
studies.
V. CONCLUSION
In this paper, we developed a novel approach to study
eye movements in naturalistic environments. Computer vi-
sion algorithms were used to detect dynamically changing
AOIs. Fixation score metric is developed to account for
the gaze shifts between target and non-target AOIs. Statis-
tically significant negative correlation was found between
fixation score and reaction time indicating that fixation score
decreases with vigilance decrement. This research is very
useful in developing computer vision assisted eye tracking
systems that can monitor cognitive performance in dynamic
naturalistic environments.
ACKNOWLEDGMENT
We would like to express our sincere gratitude to Dr. Quin,
Asst. Professor, Singapore University of Technology and
Design (SUTD) for allowing us to borrow the eye tracker and
data analysis software. We also thank Rohan Ghosh, graduate
student SINAPSE, NUS for his suggestions regarding SIFT
algorithm and Mohammed Sharif, FYP student, ECE, NUS
for helping with data analysis.
REFERENCES
[1] Martel, A., D¨
ahne, S., & Blankertz, B. (2014). EEG predictors of
covert vigilant attention. Journal of neural engineering, 11(3), 035009.
[2] Dong, Y., Hu, Z., Uchimura, K., & Murayama, N. (2011). Driver
inattention monitoring system for intelligent vehicles: A review. IEEE
transactions on intelligent transportation systems, 12(2), 596-614.
[3] Berka, C., Levendowski, D. J., Lumicao, M. N., Yau, A., Davis,
G., Zivkovic, V. T., Olmstead, R.E., Tremoulet, P.D. & Craven, P.L.
(2007). EEG correlates of task engagement and mental workload in
vigilance, learning, and memory tasks. Aviation, space, and environ-
mental medicine, 78(5), B231-B244.
[4] Di Stasi, L. L., Catena, A., Canas, J. J., Macknik, S. L., & Martinez-
Conde, S. (2013). Saccadic velocity as an arousal index in naturalistic
tasks. Neuroscience and biobehavioral reviews, 37(5), 968-975.
[5] Martinez-Conde, S., Macknik, S. L., & Hubel, D. H. (2004). The
role of fixational eye movements in visual perception. Nature reviews
neuroscience, 5(3), 229-240.
[6] Bodala, I. P., Ke, Y., Mir, H., Thakor, N. V., & Al-Nashash, H. (2014).
Cognitive workload estimation due to vague visual stimuli using
saccadic eye movements. In Engineering in medicine and biology
society (EMBC), 2014 36th Annual international conference of the
IEEE, pp. 2993-2996. IEEE.
[7] McIntire, L. K., McKinley, R. A., Goodyear, C., & McIntire, J. P.
(2014). Detection of vigilance performance using eye blinks. Applied
ergonomics, 45(2), 354-362.
[8] Bodala, I. P., Li, J., Thakor, N. V., & Al-Nashash, H. (2016). EEG
and eye tracking demonstrate vigilance enhancement with challenge
integration. Frontiers in human neuroscience, 10.
[9] Caldara, R., & Miellet, S. (2011). iMap: a novel method for statistical
fixation mapping of eye movement data. Behavior research methods,
43(3), 864-878.
[10] Istance, H., & Hyrskykari, A. (2011). Gaze-aware systems and atten-
tive applications. Gaze interaction and applications of eye tracking:
Advances in assistive technologies, 175.
[11] Goldberg, J. H., Stimson, M. J., Lewenstein, M., Scott, N., &
Wichansky, A. M. (2002). Eye tracking in web search tasks: Design
implications. In proceedings of the 2002 symposium on Eye tracking
research and applications, pp. 51-58. ACM.
[12] Riby, D., & Hancock, P. J. (2009). Looking at movies and cartoons:
eye tracking evidence from Williams syndrome and autism. Journal of
intellectual disability research, 53(2), 169-181.
[13] Olsen, A. (2012). The tobii i-vt fixation filter. Tobii Technology.
[14] Olsen, A., & Matos, R. (2012). Identifying parameter values for an
I-VT fixation filter suitable for handling data sampled with various
sampling frequencies. In proceedings of the symposium on Eye
tracking research and applications, pp. 317-320. ACM.
[15] Vedaldi, A., & Fulkerson, B. (2010). VLFeat: An open and portable
library of computer vision algorithms. In proceedings of the 18th ACM
international conference on multimedia, pp. 1469-1472. ACM.
[16] Papenmeier, F., & Huff, M. (2010). DynAOI: A tool for matching
eye-movement data with dynamic areas of interest in animations and
movies. Behavior research methods, 42(1), 179-187.
[17] Thomson, D. R., Besner, D., & Smilek, D. (2015). A resource-control
account of sustained attention: Evidence from mind-wandering and
vigilance paradigms. Perspectives on psychological science, 10(1), 82-
96.
... Fixation distance from the target is another measurement derived from eye-tracking data that has been shown to have a negative correlation with vigilance decrement. This is because at low vigilance levels, visual attention shifts from target objects to non-target objects (Bodala et al., 2017). ...
... A decrease in the number of fixations is associated with a decrease in vigilance level. This commonly occurs along with attention shifting from the target (Gartenberg et al., 2018;Bodala et al., 2017, Naeeri et al., 2019Lavine et al., 2002). • Mean eye fixation duration increases with vigilance decrement (Naeeri et al., 2019;Moacdieh & Sarter, 2017;Lavine et al., 2002;Moacdieh & Sarter, 2012). ...
... • Mean eye fixation duration increases with vigilance decrement (Naeeri et al., 2019;Moacdieh & Sarter, 2017;Lavine et al., 2002;Moacdieh & Sarter, 2012). • Fixation distance: Eye fixation distance from the target increases when the vigilance level declines (Bodala et al., 2017;Lavine et al., 2002). ...
Conference Paper
Vigilance is the ability to sustain attention for more than 5 to 10 minutes at a time. Maintaining vigilance over a prolonged duration is challenging, and the ability to do so generally declines over time, a phenomenon that is known as “vigilance decrement.” Vigilance decrement is often associated with physiological changes. Although previous studies have examined the relationship between physiological responses and vigilance decrement, the results are inconsistent and the trends are not sufficiently clear. Recognizing the need for a comprehensive overview of the existing results, in this paper, we review the most recent studies focusing on physiological changes as indicators of vigilance decrement. We consider electroencephalography (EEG), electrocardiography (ECG), eye movement, and electromyography (EMG). We present an overview of the overall correlations between these measures and vigilance levels; we also highlight the limitations and challenges of previous studies and provide some insight into future research directions in this field.
... The driving simulator consists of 3 large LCD screens and the Logitech G27 Racing Wheel (driving wheel, pedals and gear box). City Car Driving 1.5 was employed to virtualize cars and roads, forming a simulated country side scenario [24,25]. The multi-screen display provided a wide view matching to the field of sight of a human eye. ...
... The multi-screen display provided a wide view matching to the field of sight of a human eye. Subjects were instructed to continuously drive the controlled car for 90 mins [24]. The experiment comprised 2 sessions, where in each session, the subjects were instructed to follow a guiding car and brake whenever the tail red lights of the guiding car were lit, signaling the guiding car started to brake. ...
Article
Full-text available
-Mental fatigue in drivers is one of the leading causes that give rise to traffic accidents. Electroencephalography (EEG) based driving fatigue studies showed promising performance in fatigue monitoring. However, complex methodologies are not suitable for practical implementation. In our simulation based setup that retained the constraints of real driving, we took a step closer to fatigue estimation in a practical scenario. We adopted a pre-processing pipeline with low computational complexity, which can be easily and practically implemented in real-time. Moreover, regression-based continuous fatigue estimation was achieved using power spectral features in conjunction with time as the fatigue label. We sought to compare three regression models and three time windows to demonstrate their effects on the performance of fatigue estimation. Dynamic time warping was proposed as a new measure for evaluating the performance of fatigue estimation. The results derived from the validation of the proposed framework on 19 subjects showed that our proposed framework was promising towards practical implementation. Fatigue estimation by the support vector regression with radial basis function kernel and 5-second window length achieved the best performance. We also provided a comprehensive analysis on the spatial distribution of channels and frequency bands mostly contributing to fatigue estimation, which can inform the feature and channel reduction for real-time fatigue monitoring in practical driving. After reducing the number of electrodes by 75%, the proposed framework retained comparable performance in fatigue estimation. This study demonstrates the feasibility and adaptability of our proposed framework in practical implementation of mental fatigue estimation.
... Corresponding author email address: andrei.drag@gmail.com task performance that can be attributed to vigilance decrement [4]. In this study, we have combined two established techniques in network analysis to investigate the underlying neural mechanisms associated with vigilance decrement. ...
... The experimental task was divided into two phases: Phase 1, or the baseline alert phase (0-12 mins) and Phase 2, or the vigilance decrement phase (12-25 mins from the start of the experiment). This study is an extension of our previous work that has investigated the changes in behavioural and ocular parameters attributing to vigilance decrement [4]. EEG signals were collected using a standard 64 channel ANT-neuro cap (sampling frequency 512 Hz), connected to an amplifier. ...
Conference Paper
Full-text available
Understanding modular organization between brain regions can provide deeper insight into the complex neural mechanisms associated with processes like vigilance decrement. Distinct but interacting modules in brain connectivity networks have been known to support integration of specific mechanisms relevant in high-order cognitive processes. To investigate the neuronal mechanisms associated with vigilance decrement, we conducted an experiment where the participants performed a driving task. EEG graph metrics within communities, like clustering coefficient (C_intra), efficiency (E_intra), density (D_intra), and between communities, like intermodule density (D_inter), were computed from the source-localized surface brain signals. Further, we also calculated the nodal out degree to investigate the difference in information flow in the brain during vigilance decrement. Increase in the intermodule density, D_inter , was observed from the left fronto-parietal cluster to the right temporo-parietal cluster. Moreover, significant reduction in the intramodule metrics, E_intra and C_intra was observed in the right temporo-parietal cluster. Thus, our findings signify a flexible topographical architecture to compensate the hub disruption effect caused due to decline in vigilance.
... • Configure a high pass filter for 30 • /s saccade velocity [36], [37], and 0.1 • saccade amplitude. ...
Article
Full-text available
Vigilance is the capacity to remain alert for an extended time while performing a task. Staying alert is obligatory in many jobs, particularly those that involve monitoring, such as surveillance tasks, security monitoring, and air traffic control. These monitoring tasks require a specific level of arousal to maintain an adequate level of cognitive efficiency. In this study, we investigate the possibility of assessing the vigilance levels using a fusion of electroencephalography (EEG) and eye tracking data. Vigilance levels were established by performing a modified version of the Stroop color word task (SCWT) for 30 minutes. Feature-level fusion based on the canonical correlation analysis (CCA) was employed to each brain region to improve the classification accuracy of vigilance level assessment. Results obtained using support vector machines (SVM) classifier show that fusion of EEG+eye tracking modalities has improved the classification accuracy compared to individual modality. The EEG+Eye tracking fusion on the right central brain region achieved the highest classification accuracy of 97.4 ± 1.3%, compared to the individual Beta EEG with 92.0±7.3% and Eye tracking with 76.8±8.4%, respectively. Likewise, EEG and Eye tracking fusion on the right frontal region showed classification accuracy of 96.9 ± 1.1% for both the Alpha and Beta bands. Meanwhile, when all brain regions were utilized, the highest classification accuracy of EEG+Eye tracking was 96.8 ± 0.6% using Delta band compared to the EEG alone with 88.18 ± 8.5% and eye tracking alone with 76.8 ± 8.4 %, respectively. The overall results showed that vigilance is a brain region specific and the fusion of EEG+ and Eye tracking data using CCA has significantly improved the classification accuracy of vigilance levels assessment.
... There are not many papers that have investigated the relationship between eye-fixation variables and fatigue. One of the few exceptions is a study by Bodala et al. (2017). They devised a fixation score measure which can be used as an indicator of vigilance decrement with the aid of computer vision assisted eye-tracking. ...
Conference Paper
Full-text available
The impact of fatigue on train drivers is one of the most important safety-critical issues in rail. It affects drivers’ performance, significantly contributing to railway incidents and accidents. To address the issue of real-time fatigue detection in drivers, most reliable and applicable psychophysiological indicators of fatigue need to be identified. Hence, this paper aims to examine and present the current state of the art in physiological measures for real-time fatigue monitoring that could be applied in the train driving context. Three groups of such measures are identified: EEG, eye-tracking and heart-rate measures. This is the first paper to provide the analysis and review of these measures together on a granular level, focusing on specific variables. Their potential application to monitoring train driver fatigue is discussed in respective sections. A summary of all variables, key findings and issues across these measures is provided. An alternative reconceptualization of the problem is proposed, shifting the focus from the concept of fatigue to that of attention. Several arguments are put forward in support of attention as a better-defined construct, more predictive of performance decrements than fatigue, with serious ramifications on human safety. Proposed reframing of the problem coupled with the detailed presentation of findings for specific relevant variables can serve as a guideline for future empirical research, which is needed in this field.
... The driving experimental task is designed similar to the protocol being followed in our previous work [33], [60], [61]. The subject set in the front of the screen for a distance of 1.8 meters, these screens display the landscape along the road and obey to the left driving rules according to Singapore standards. ...
Article
Full-text available
The raising number of traffic accidents has become a major issue in our daily life, which has attracted the concern of society and governments. To deal with this issue, in our previous study, we have designed a real-time driving fatigue detection system using power spectrum density and sample entropy. By using the wireless technology and dry electrodes for EEG collection, we further integrated virtual reality simulated driving environment, which made our study more applicable to realistic settings. However the high accuracy of classification for driving fatigue has not been obtained. To measure the time series complexity of EEG signal, we proposed a fusion entropy (sample entropy, approximate entropy, and spectral entropy) analysis method of EEG and EOG. Firstly, sample entropy was applied for feature extraction from the horizontal and vertical EOG. Secondly, approximate entropy, sample entropy and spectral entropy features of each sub-band of EEG are extracted. Thirdly, feature fusion for sub-band is performed by canonical correlation analysis (CCA). Finally, the features of EOG and EEG are classified using a relevant vector machine (RVM). Twenty-two subjects participated in the driving fatigue experiments for a duration of 90 minutes. The results demonstrated that the fusion entropy analysis combining EOG and EEG could provide an alternative method for driving fatigue detection, and the average accuracy rate was up to 99.1±1.2%. The authors further analyzed the effect of feature fusion in four sub-bands (δ, α, β, θ) and compared with every single sub-band on classification performance, it is proved that the former is superior to the latter presenting the proposed method can provide effective indicators for driving fatigue detection.
Article
Full-text available
Background and objective: The cognitive workload is an important component in performance psychology, ergonomics, and human factors. Publicly available datasets are scarce, making it difficult to establish new approaches and comparative studies. In this work, COLET-COgnitive workLoad estimation based on Eye-Tracking dataset is presented. Methods: Forty-seven (47) individuals' eye movements were monitored as they solved puzzles involving visual search activities of varying complexity and duration. The participants' cognitive workload level was evaluated with the subjective test of NASA-TLX and this score is used as an annotation of the activity. Extensive data analysis was performed in order to derive eye and gaze features from low-level eye recorded metrics, and a range of machine learning models were evaluated and tested regarding the estimation of the cognitive workload level. Results: The activities induced four different levels of cognitive workload. Multi tasking and time pressure have induced a higher level of cognitive workload than the one induced by single tasking and absence of time pressure. Multi tasking had a significant effect on 17 eye features while time pressure had a significant effect on 7 eye features. Both binary and multi-class identification attempts were performed by testing a variety of well-known classifiers, resulting in encouraging results towards cognitive workload levels estimation, with up to 88% correct predictions between low and high cognitive workload. Conclusions: Machine learning analysis demonstrated potential in discriminating cognitive workload levels using only eye-tracking characteristics. The proposed dataset includes a much higher sample size and a wider spectrum of eye and gaze metrics than other similar datasets, allowing for the examination of their relations with various cognitive states.
Article
Most of existing eye movement-based fatigue detectors utilize statistical analysis of fixations, saccades, and blinks as inputs. Nevertheless, these parameters require long recording time and heavily depend on eye trackers. In an effort to facilitate proactive detection of mental fatigue, we introduced a complemental fatigue indicator, named gaze-bin analysis, which simply presents the eye-tracking data with histograms. A method which engaged the gaze-bin analysis as inputs of semisupervised bagged trees was developed. A case study in a vessel traffic service center demonstrated that this approach can alleviate the burden of manual labeling as well as improve the performance of fatigue detection model. In addition, the results show that the approach can achieve an excellent accuracy of 89%, which outperformed other methods. In general, this study provided a complemental indicator for detecting mental fatigue as well as enabled the application of a low sampling rate eye tracker in the traffic control center.
Article
Full-text available
Maintaining vigilance is possibly the first requirement for surveillance tasks where personnel are faced with monotonous yet intensive monitoring tasks. Decrement in vigilance in such situations could result in dangerous consequences such as accidents, loss of life and system failure. In this paper, we investigate the possibility to enhance vigilance or sustained attention using “challenge integration,” a strategy that integrates a primary task with challenging stimuli. A primary surveillance task (identifying an intruder in a simulated factory environment) and a challenge stimulus (periods of rain obscuring the surveillance scene) were employed to test the changes in vigilance levels. The effect of integrating challenging events (resulting from artificially simulated rain) into the task were compared to the initial monotonous phase. EEG and eye tracking data is collected and analyzed for n = 12 subjects. Frontal midline theta power and frontal theta to parietal alpha power ratio which are used as measures of engagement and attention allocation show an increase due to challenge integration (p < 0.05 in each case). Relative delta band power of EEG also shows statistically significant suppression on the frontoparietal and occipital cortices due to challenge integration (p < 0.05). Saccade amplitude, saccade velocity and blink rate obtained from eye tracking data exhibit statistically significant changes during the challenge phase of the experiment (p < 0.05 in each case). From the correlation analysis between the statistically significant measures of eye tracking and EEG, we infer that saccade amplitude and saccade velocity decrease with vigilance decrement along with frontal midline theta and frontal theta to parietal alpha ratio. Conversely, blink rate and relative delta power increase with vigilance decrement. However, these measures exhibit a reverse trend when challenge stimulus appears in the task suggesting vigilance enhancement. Moreover, the mean reaction time is lower for the challenge integrated phase (RTmean = 3.65 ± 1.4s) compared to initial monotonous phase without challenge (RTmean = 4.6 ± 2.7s). Our work shows that vigilance level, as assessed by response of these vital signs, is enhanced by challenge integration.
Article
Full-text available
Objective: The present study addressed the question whether neurophysiological signals exhibit characteristic modulations preceding a miss in a covert vigilant attention task which mimics a natural environment in which critical stimuli may appear in the periphery of the visual field. Approach: Subjective, behavioural and encephalographic (EEG) data of 12 participants performing a modified Mackworth Clock task were obtained and analysed offline. The stimulus consisted of a pointer performing regular ticks in a clockwise sequence across 42 dots arranged in a circle. Participants were requested to covertly attend to the pointer and press a response button as quickly as possible in the event of a jump, a rare and random event. Main results: Significant increases in response latencies and decreases in the detection rates were found as a function of time-on-task, a characteristic effect of sustained attention tasks known as the vigilance decrement. Subjective sleepiness showed a significant increase over the duration of the experiment. Increased activity in the α-frequency range (8-14 Hz) was observed emerging and gradually accumulating 10 s before a missed target. Additionally, a significant gradual attenuation of the P3 event-related component was found to antecede misses by 5 s. Significance: The results corroborate recent findings that behavioural errors are presaged by specific neurophysiological activity and demonstrate that lapses of attention can be predicted in a covert setting up to 10 s in advance reinforcing the prospective use of brain-computer interface (BCI) technology for the detection of waning vigilance in real-world scenarios. Combining these findings with real-time single-trial analysis from BCI may pave the way for cognitive states monitoring systems able to determine the current, and predict the near-future development of the brain's attentional processes.
Article
Full-text available
Research has shown that sustained attention or vigilance declines over time on task. Sustained attention is necessary in many environments such as air traffic controllers, cyber operators, and imagery analysts. A lapse of attention in any one of these environments can have harmful consequences. The purpose of this study was to determine if eye blink metrics from an eye-tracker are related to changes in vigilance performance and cerebral blood flow velocities. Nineteen participants performed a vigilance task while wearing an eye-tracker on four separate days. Blink frequency and duration changed significantly over time during the task. Both blink frequency and duration increased as performance declined and right cerebral blood flow velocity declined. These results suggest that eye blink information may be an indicator of arousal levels. Using an eye-tracker to detect changes in eye blinks in an operational environment would allow preventative measures to be implemented, perhaps by providing perceptual warning signals or augmenting human cognition through non-invasive brain stimulation techniques.
Book
Recent advances in eye tracking technology will allow for a proliferation of new applications. Improvements in interactive methods using eye movement and gaze control could result in faster and more efficient human computer interfaces, benefitting users with and without disabilities. Gaze Interaction and Applications of Eye Tracking: Advances in Assistive Technologies focuses on interactive communication and control tools based on gaze tracking, including eye typing, computer control, and gaming, with special attention to assistive technologies. For researchers and practitioners interested in the applied use of gaze tracking, the book offers instructions for building a basic eye tracker from off-the-shelf components, gives practical hints on building interactive applications, presents smooth and efficient interaction techniques, and summarizes the results of effective research on cutting edge gaze interaction applications.
Article
In this chapter, we examine systems that use the current focus of a person's visual attention to make the system easier to use, less effortful and, hopefully, more efficient. If the system can work out which object the person is interested in, or is likely to interact with next, then the need for the person to deliberately point at, or otherwise identify that object to the system can be removed. This approach can be applied to interaction with real-world objects and people as well as to objects presented on a display close to the system user. We examine just what we can infer about a person's focus of visual attention, and their intention to do something from studying their eye movements, and what, if anything, the system should do about it. A detailed example of an attentive system is presented where the system estimates the difficulty a reader has understanding individual words when reading in a foreign language, and displays a translation automatically if it thinks it is needed.
Article
Visual perception is affected by the quality of stimulus. In this paper, we investigate the rise in cognitive workload of an individual performing visual task due to vague visual stimuli. We make use of normalized average peak saccadic velocity to estimate the cognitive workload. Results obtained from 16 human subjects show that the mean of peak saccadic velocity increases with workload indicating that faster saccades are required to obtain information as the workload increases. This technique should find application in assessment of vigilance and cognitive performance in many demanding professional, industrial and transportation situation.
Article
Staying attentive is challenging enough when carrying out everyday tasks, such as reading or sitting through a lecture, and failures to do so can be frustrating and inconvenient. However, such lapses may even be life threatening, for example, if a pilot fails to monitor an oil-pressure gauge or if a long-haul truck driver fails to notice a car in his or her blind spot. Here, we explore two explanations of sustained-attention lapses. By one account, task monotony leads to an increasing preoccupation with internal thought (i.e., mind wandering). By another, task demands result in the depletion of information-processing resources that are needed to perform the task. A review of the sustained-attention literature suggests that neither theory, on its own, adequately explains the full range of findings. We propose a novel framework to explain why attention lapses as a function of time-on-task by combining aspects of two different theories of mind wandering: attentional resource (Smallwood & Schooler, 2006) and control failure (McVay & Kane, 2010). We then use our "resource-control" theory to explain performance decrements in sustained-attention tasks. We end by making some explicit predictions regarding mind wandering in general and sustained-attention performance in particular. © The Author(s) 2014.
Article
Selecting values for fixation filters is a difficult task as not only the specifics of the selected filter algorithm has to be taken into account, but also what it is going to be used for and by whom. In this paper the selection and testing process of values for an I-VT fixation filter algorithm implementation is described.