Conference PaperPDF Available

The Effect of Whole-Body Haptic Feedback on Driver’s Perception in Negotiating a Curve


Abstract and Figures

It remains uncertain regarding the safety of driving in autonomous vehicles that, after a long, passive control and inattention to the driving situation, how the drivers will be effectively informed to take-over the control in emergency. In particular, the active role of vehicle force feedback on the driver’s risk perception on curves has not been fully explored. To investigate it, the current paper examined the driver’s cognitive and visual responses to the whole-body haptic feedback during curve negotiations. The effects of force feedback on drivers’ responses on curves were investigated in a high-fidelity driving simulator while measuring EEG and visual gaze over ten participants. The preliminary analyses of the first two participants revealed that pupil diameter and fixation time on the curves were significantly longer when the driver received whole-body feedback, compared to none. The findings suggest that whole-body feedback can be used as an effective “advance notification” of hazards.
Content may be subject to copyright.
The Effect of Whole-Body Haptic Feedback on Driver’s Perception in
Negotiating a Curve
Erfan Pakdamanian1, Lu Feng1,2, Inki Kim1
1 Systems and Information Engineering, University of Virginia, Charlottesville, VA
2 Computer Science, University of Virginia, Charlottesville, VA
It remains uncertain regarding the safety of driving in autonomous vehicles that, after a long,
passive control and inattention to the driving situation, how the drivers will be effectively
informed to take-over the control in emergency. In particular, the active role of vehicle force
feedback on the driver’s risk perception on curves has not been fully explored. To investigate it,
the current paper examined the driver’s cognitive and visual responses to the whole-body haptic
feedback during curve negotiations. The effects of force feedback on drivers’ responses on curves
were investigated in a high-fidelity driving simulator while measuring EEG and visual gaze over
ten participants. The preliminary analyses of the first two participants revealed that pupil diameter
and fixation time on the curves were significantly longer when the driver received whole-body
feedback, compared to none. The findings suggest that whole-body feedback can be used as an
effective advance notification of hazards.
Although the future of car industries will be
dominated by autonomous vehicles and the car will drive
itself, there will still be a need for drivers to take over
the car (Banks & Stanton, 2016). Human intervention is
particularly necessary to prevent tragic accidents when
the autonomous vehicle encounters curves, bad weather
and unpredictable pedestrian behavior (Wright,
Svancara, & Horrey, 2017). Although autonomous
vehicles will overall decrease the physical and mental
workload of drivers by assigning these tasks to an
automated system, human drivers would still play a
critical role in car safety responsibility (Parasuraman and
Wickens, 2008). However, it was been shown that a
sudden alarm and notification to the driver about the
upcoming potential hazards would incur higher stress
and cognitive load (Shah et al., 2015). Once drivers
allow the automated system to control the car, meaning
the driver tends to allocate his attention resources to
non-driving tasks (e.g. video gaming, talking on the
phone etc.), his or her attention will be taken away from
the primary task of driving. In such circumstances, any
simple form of visual, auditory or haptic signals would
not be sufficient to communicate critical information
about the vehicle conditions, only to startle and stress the
driver in emergency (Petermeijer, Cieler, & Winter,
One approach to cope with the stress from
unexpected alarms is to examine the effects of signals on
potentially safety-compromising situations, and
accustom the driver to it. In this regard, this paper
intends to investigate the effects of whole-body haptic
feedback, delivering haptic cues to drivers’ full body, on
the drivers’ visual perception and cognitive states during
curve negotiation, as an alternative to its counterpart
alarming signals. Assessing the drivers’ cognitive states
can help infer what type of haptic feedback the cars
should provide to mitigate the stress of taking over
during critical moments. In literature, vibrotactile haptic
feedback was shown to enhance the reaction time of
taking control back at life-threatening moments
(Prewett, Elliott, Walvoord, & Coovert, 2012). It noted,
however, that once the drivers were spatially aware, the
vibrotactile directional cue may not be as effective as
visual directional alternatives. Therefore, this study
intends to focus on whole-body haptic feedback to
complement this drawback.
Morrell and Wasilewski (2010) designed and developed
a haptic-feedback seat for traditional vehicles that aimed
to share spatial information, and improve situation
awareness (SA). The drivers were informed about the
location of car-following and close-by vehicles, through
vibrotactile feedback from the seat back in a way that the
closer the car is, the more sensors vibrated. Nonetheless,
on the one hand, evaluating the time in the blind spot
may not be the accurate measurement for the risk
assessment. Nonetheless, on the one hand, evaluating the
time in the blind spot may not be the accurate
measurement for the risk assessment. On the other hand,
as auto industries attend to autonomous technologies,
alert systems need to become adaptive to vehicle speed
and situation but not particularly designed for a specific
Petermeijer et al. (2017) designed a vibrotactile
feedback seat that contains static and dynamic vibration
for automated vehicles. The authors aimed to analyze the
accuracy of drivers’ response rate and their reaction time
Not subject to U.S. copyright restrictions. DOI 10.1177/1541931218621005
Proceedings of the Human Factors and Ergonomics Society 2018 Annual Meeting 19
to the requested time for maneuvering. After receiving
tactile stimuli, drivers had to respond accordingly to the
vibration direction by moving to the left or the right.
However, in all the presented scenarios, there was not
any additional warning cue. Furthermore, the
participants reported difficulties in understanding
whether the cue was to their left or right; alarms were
only triggered about one second prior to an event
occurrence, which was shorter than the realistic average
reaction time needed (3.5 sec) for a transition control in
automated vehicles (Melcher, Rauh, Diederichs,
Widlroither, & Bauer, 2015).
This research aims to examine perceptual and
cognitive effects of using whole-body force feedback on
the control responses of the drivers. Through the
controlled experiments in simulation setting, it is
expected that the whole-body force feedback will be
shown its values, in a way that does not only warn the
driver when a takeover is required, but also assists the
driver during the critical phases, including their lack of
SA (shifting of attention) and cognitive processing. In
this regard, we hypothesized that the whole-body haptic
feedback would allow the drivers to be effectively aware
of upcoming curves in a simulated driving environment.
The experiment was conducted in a high-fidelity
driving simulator (the 401cr motion system by Force
Dynamics) equipped with three monitors. The simulator
mimics various acceleration dynamics thereby creating a
realistic response upon the driver’s body. The motion-
capable high-fidelity simulator was used with two
configurations: 1) without whole-body motion feedback,
2) with whole-body motion feedback with approximately
18 inches of movement in 360 degrees. This also
allowed six degrees of freedom to replicate the motions
associated with driving in a way that vibration of the seat
serves as an “intelligent messenger”. It ensures human
stays informed of the vehicle safety. The study was
approved by University of Virginia Institutional Review
Board (Protocol # 2017-0296-00).
The speedometer and the RPM gauge is located in
the center of the middle monitor (Figure. 4). Moreover,
the implemented automation system had a longitudinal
capability similar to common ACC systems, which allow
drivers to follow the indicated speed limits as well as
keep the car in the center of the lane. Data were recorded
at a frequency of 100 Hz, including the vehicle’s
position, accelerations and steering wheel angle (they
were not included in the preliminary study and will be
reported in further analysis).
Data acquisition
A wearable eye-tracker glass (Tobii Pro-Glasses 2,
Danderyd, Sweden; Tobii Pro-Glasses 2, 2017) was used
to track the driver’s gaze behavior at a sampling rate of
60 Hz (i.e., 60 gaze data points collected per second for
each eye; 4 eye cameras, H.264 1920x1080 pixels at 25
fps) (Figure 1). The Tobii Pro Glasses 2 eye-tracker is
wireless with live view capability for insights in any
real-world environment. Since the driving simulator and
curves are dynamic scenes, head-mounted eye tracker
was required. Also, it ensures that the participant’s full
and complete range of motions for their head.
A B-Alert X24 system with 24 channels was used
with the sample rate of 256Hz to record the
Electroencephalography (EEG) data (Figure 1). Wireless
EEG signals were sent via Bluetooth to the data
acquisition system. Also, in order to record the electoral
activity of the brain, the sensor strip was placed
according to the 10/20 extended standard.
The sampled data was sent wirelessly to iMotion
(biometric research platform) which allowed collection
of the synchronized EEG and eye-tracker data (Attention
tools, 2016).
Two graduate students (both male, 22 and 35 years
old) holding a driver’s license voluntarily participated in
this preliminary study (ten participants equally balanced
between male and female aged between 18 to 40 will be
recruited). None of the participants had visual
impairments, or any other symptoms or diseases that
could compromise their ability to drive.
Once the participant arrived, the relevant
information regarding gender, age and driving
experiences was gathered. Subsequently, participants
were verbally instructed regarding how to use the
devices and simulator as well as their primary task of
driving with their hands on the wheel by the
experimenter. Furthermore, both drivers were told that
they need to keep their speed under 60mph and drive as
they would normally do. The experimenter allowed the
participants to familiarize themselves with the system
with 2-5 mins test drive. Once they showed that they
were comfortable with all the devices and driving the
simulator, there were asked to take 3mins break between
the sessions in order to maximize the concentration level
and minimize fatigue throughout the 18 min session. The
experimenter started the three curve and force-feedback-
free trials as the Baseline session. Afterwards, the
participants drove through the counterbalanced
designated scenario six times (three trials with force
Proceedings of the Human Factors and Ergonomics Society 2018 Annual Meeting 20
feedback and three without). Each scenario took
approximately 3mins, depends on the speed.
Signal Pre-processing
256Hz sampled data was filtered using high and
low band pass filter with a cut-off frequency between 0.5
Hz, to remove DC drift, and 80 Hz respectively to
remove power-line noise and low frequencies separately
(Gheorghe, 2017). Also, a notch filter at 60Hz was used.
EEG data pre-processing initiated by referencing to the
left ear lobe channel as well as applying Fast Fourier
transform (FFT) algorithm to filter the different
frequency band.
To analyze the EEG data, initially the blink artifacts
were removed by using Independent Component
Analysis (ICA) and wavelet analyses were used to
generate a continuous record of theta band by using
Matlab (2017, The MathWorks, Inc., Natick,
Massachusetts, United Statest) and EEGLab toolbox. An
electrode impedance test was performed to ensure proper
conductivity of the electrodes. The impedance level
threshold of 20 was used. Also, the EEG calibration
procedure was implemented before data collection.
Collected data were extracted using iMotion
software. In order to perform a comparison analysis
between three conditions (Baseline, with whole body
force feedback and without), approximately four seconds
before curves was analyzed following the approach
taken by Gheorghe (2017). Each trial consisted of
twenty curves, including simple curves, compound
curves, reverse curves and deviation. However, we were
only interested in simple curves for our preliminary
Figure 1. Experimental set-up for recording EEG and Eye
Analysis of visual attention
iMotion provides the following metric for analyzing
eye movement: Time spent-fixation, fixation duration,
Time to First Fixation (TTFF-F) and pupil diameter
(Table 1). Table 1 summarizes the time spent and
fixation duration on the AOI. In order to identify when
curves as the critical section of the road on the visual
display were fixed, AOI analysis was performed (Figure
4). Comparing the TTFF values (Figure 3) indicates both
participants tended to concentrate slightly more on the
curves at the presence of force feedback which indicates
higher SA. Likewise, when the force feedback was
applied to drivers, pupil diameter was larger approaching
the curves (Table 1). Therefore, the drivers tend to fixed
their gaze on curves significantly higher at the presence
of the whole-body feedback.
The differences between the two types of vibration
patterns including force and none-force feedback was
assessed using t-test. T-test yielded statistically
difference between the force feedback and none in the
dependent variables (t(11) = 4.96, p= 0.002; t(11)
=12.38, p<0.001; t(11) = 3.51, p=0.008, for time spent,
fixation duration, and pupil diameter, respectively).
Analysis of cognitive states
Analyzing three frequencies (Theta, Alpha and Beta)
revealed that the Theta power increases in force feedback
cases. Also, on the beta band, grown power was obtained.
Still, the amount of power increasing on Theta band was
higher, which may indicate the greater drivers’
engagement while using haptic feedback. The findings
represent that the force-feedback could correlate with
higher ability in decision making and ultimately increase
the capability of controlling the vehicle properly at the
time of hazard encounter. It was initially expected to get
Table 1. Mean and Standard Deviation for Metrics of
Eye Movements
Independent Variables
With force
Without force
Time spent (sec)
6.83 (1.92)
4.49 (0.28)
Fixation duration
3.45 (0.89)
3.04 (0.61)
Pupil diameter
35 (9.5)
27 (4.3)
Proceedings of the Human Factors and Ergonomics Society 2018 Annual Meeting 21
the consistent results with Almahasneh et al. (2014)
findings, however, the topographical map result (Figure
3) indicates that the difference between baseline and both
cases is caused by more activity in corresponding brain
region of the right frontal hemisphere near reaching the
curves. Since most of cognitive activities occur at the
frontal lobe (Lin et al., 2011) the findings are aligned with
the role of frontal lobe in decision making and attention
(Burgess, Alderman, Volle, Benoit, & Gilbert, 2009). The
topographical analysis extracted from the scalp above the
sensorymotor cortex indicates more activity on the
bipolar channels C3 and C4 (Figure 3). Electrode C4
represents the highest activation throughout the six curves
which may cause by Motor execution phase of driving.
Slightly higher activation in motor cortex at the presence
of whole-body haptic feedback supports an enhancement
to drivers’ engagement of required cognitive tasks of
braking and steering control (Saha, Konar, Nagar, 2017).
However, band frequency modulation based on ERP will
be analyzed at the critical time intervals of curve
negotiation. Our intent is to analyze the variability of
frequency bands inside some temporal windows around
200 ms and 400 ms of latency.
Figure 2. Topographical analysis of six simple curves. The first
row represents the distribution of difference between baseline
and scenarios at the absence of force feedback (the first row) and
the at the presence of it (the second row)
Figure 3. Eye- tracking results for fixation behaviors over
different feedback condition
Figure 4. AOI and driver’s view
The main differences between the two types of
feedback found in this study is containing driver’s visual
responses. Fixed duration and pupil diameters found
significantly higher while driving with haptic feedback
in this preliminary study which could be due to the
higher cognitive engagement. If it was the case, finding
higher power in Theta band in frontal lobe is due to high
vibration of system during haptic feedback activation
and it is not relevant to the type of feedback. Therefore,
the findings could be supported by the results that the
high-fidelity driving simulator that can simulate various
scenarios with high validity improvement of drivers’
performance engages driver better (Groeger & Banks,
This preliminary study confirms the possibility of
EEG usage to alarm drivers properly within less than
few seconds, once the system recognizes driver’s
cognitive stage and driving environment. We expect that
the need for more number of channels for prediction of
performance and drivers’ cognitive state prior to hazard
with other EEG measurements (e.g. ERP) would help us
to develop a safer whole-body feedback to reduce
cognitive workload and stress level of the driver, thereby
enhance their control ability.
In the future, we will design and analyze a haptic
force feedback which could communicate with drivers
through the seat and serves as an “intelligent messenger”
that ensures human stays informed of the vehicle safety
as well as driving environment which could play the role
of “advance notification”. In that regards, we will
validate the preliminary findings with further analysis of
power variation in each frequency within temporal
duration as well as Event Related Potential (ERP). It
could assist us to identify the perceptual operations of
drivers on curves.
Proceedings of the Human Factors and Ergonomics Society 2018 Annual Meeting 22
Almahasneh, H., Chooi, W. T., Kamel, N., & Malik, A. S. (2014).
Deep in thought while driving: An EEG study on drivers’
cognitive distraction. Transportation Research Part F: Traffic
Psychology and Behaviour, 26(PA), 218226.
Attention Tool, 6.2, iMotions, Boston, MA, 2016.
Banks, V. A., & Stanton, N. A. (2016). Keep the driver in control:
Automating automobiles of the future. Applied Ergonomics,
53, 389395.
Gheorghe, L. A. (2017). Detecting EEG correlates during preparation
of complex driving maneuvers PAR. École polytechnique
fédérale de Lausanne.
Groeger, J. A., & Banks, A. P. (2007). Anticipating the content and
circumstances of skill transfer: Unrealistic expectations of
driver training and graduated licensing?. Ergonomics, 50(8),
Koo, J., Kwac, J., Ju, W., Steinert, M., Leifer, L., & Nass, C. (2015).
Why did my car just do that? Explaining semi-autonomous
driving actions to improve driver understanding, trust, and
performance. International Journal on Interactive Design and
Manufacturing, 9(4), 269275.
Melcher, V., Rauh, S., Diederichs, F., Widlroither, H., & Bauer, W.
(2015). Take-Over Requests for Automated Driving. Procedia
Manufacturing, 3(Ahfe), 28672873.
Morrell, J., & Wasilewski, K. (2010). Design and evaluation of a
vibrotactile seat to improve spatial awareness while driving. In
IEEE Haptics Symposium (pp. 281288).
Petermeijer, S. M., Cieler, S., & Winter, J. C. F. De. (2017).
Comparing spatially static and dynamic vibrotactile take-over
requests in the driver seat. Accident Analysis and Prevention,
99, 218227.
Prewett, M. S., Elliott, L. R., Walvoord, A. G., & Coovert, M. D.
(2012). A meta-analysis of vibrotactile and visual information
displays for improving task performance. IEEE Transactions
on Systems, Man and Cybernetics Part C: Applications and
Reviews, 42(1), 123132.
Shah, S. J., Bliss, J. P., Chancey, E. T., & Christopher Brill, J. (2015).
Effects of alarm modality and alarm reliability on workload,
trust, and driving performance. Proceedings of the Human
Factors and Ergonomics Society, 2015January(2008), 1535
Wright, T. J., Svancara, A., & Horrey, W. J. (2017). An Initial
Review of the Instructional and Operational Variability
Among Automated Systems in Passenger Vehicles.
Proceedings of the Human Factors and Ergonomics Society
Annual Meeting, 61(1), 1979.
Proceedings of the Human Factors and Ergonomics Society 2018 Annual Meeting 23
... At the high level of goal-directed actions (Hutchins, Hollan, & Norman, 1985;Norman, 2013), Norman presumes that an individual interacts with the world (and machines) by specifying and executing an action plan that best implement their intent, subject to the goal, and constraints in the environment. The neural basis of intent as a precursor to action has found some evidence. ...
... Fu & Parasuraman, 2006). The driver readiness potentials obtained from ERP components could be used to alarm the driver to take over properly via various modalities such as auditory (Forster, Naujoks, Neukum, & Huestegge, 2017); visual (Liu et al., 2014); haptic (Pakdamanian, Feng, & Kim, 2018); or multimodal stimuli, which has shown superior to unimodal alarms in prompt attention orientation (Lee, McGehee, Brown, & Marshall, 2006). ...
The state-of-the-art vehicle automation and navigation technologies promise to augment or even replace diverse human functions in driving. For safety assurance on roadways, vehicles need to be informed about the humans (not only their presence but also mental states) in and around the vehicle. Yet, the uncertainties of drivers’ mental states under varying traffic situations make it difficult to provide such information. In this regard, neuroergonomics have potential to help bridge human and automation toward the next level of integration. In this chapter, we aim to provide an appreciation of neuroergonomic application to driving and navigation, with an emphasis on drivers’ cognitive tasks and performance. Particularly, the four main cognitive constructs associated with driving—attention, situation awareness (SA), intent, and mental workload—are reviewed in terms of their theoretical foundations and recent applications in neuroergonomic studies. The effects of special demographic population and environmental selection for such study are also discussed. To the readers who are interested in understanding the effects of next-generation vehicle technologies on humans or who aspire for the breakthrough in neuroergonomics for new driving and navigation technology, this chapter may provide a useful source.
... In addition, CAVs do not violate speed limits or other traffic rules. CAVs are expected to reduce the fuel consumption by dampening traffic oscillations and following the speed limits (Pakdamanian et al., 2018;Ramin et al., 2018;Sheng et al., 2019;Adomah et al., 2021;Gaweesh et al., 2021;Stogios et al., 2018;Qu et al., 2020). However, other researchers have reported that CAVs may reduce capacity if they operate conservatively with lower speeds and longer headways (Hartmann et al., 2017). ...
... Vision has been asserted as the largest single resource available to the driver and the primary processing input in driving [14]. Behavioral measures like eye movement patterns and head positions can easily be collected by eye trackers, or by application-specific computer vision techniques [13,30,31,32] Various measures from eye movement matrices like eye blinks, fixation, gaze angle, pupil dilation, and glance duration have been used to find distraction [14,29,33,34]. It was found that distracted driving is in association with the high frequency of off-road glances, longer total eye-off-road time [17], high frequency and duration increment of eye blink, and dilated pupil [35,36]. ...
Full-text available
With the increasing number of people killed by distracted driving, drivers' attention has become a fundamental challenge recently. Performing non-driving-related tasks can make drivers manually, visually, and cognitively distracted. Therefore, there are various definitions and categories for distraction based on the distraction source. Yet, detecting manual and visual distractions are easily but cognitive distractions are hard to identify and measure. Thus, detection, assessment, and evaluation of driver's distraction seem crucial in the safety area. This study aims to (a) present different definitions of driver distraction, (b) introduce various methods to detect driver inattention, and (c) discuss each method's pros and cons and give an insight for an all-inclusive measurement to detect drivers' distraction.
... These findings may be linked to other driving parameters such as the acceleration or the braking possibilities on the simulator. As in the case of the force exerted on the braking pedal, no significant differences were detected in terms of the speed between the two groups on the straight line; whereas a higher average rpm was observed in the group tested on the simulator than in the CD group, which suggests that although the participants were driving more slowly than in real life, they maintained a lower gear ratio, possibly because they misjudged the rpm due to either the absence of vibration feedback (Tachiiri, Tanaka, & Sano, 2017), or the artificial sound of the engine emitted by the driving simulator (Pakdamanian, Feng, & Kim, 2018). ...
Car manufacturers expect driving simulators to be reliable research and development tools. Questions arise, however, as to whether drivers’ behavior on simulators exactly matches that observed when they are driving real cars. Drivers’ performances and their subjective feelings about their driving were compared between two groups during a 40-min driving test on the same circuit in a real car (n = 20) and a high-fidelity dynamic simulator (n = 27). Their speed and its variability, the braking force and the engine revolutions per minute (rpm) were recorded five times on a straight line and three times on a curve. The differences observed in these measurements between circuit driving (CD) and simulator driving (SD) from the 6th to 40th minute showed no significant changes during the drive. The drivers also completed the NASA Raw Task Load Index (NASA RTLX) questionnaire and the Simulator Sickness Questionnaire (SSQ) and estimated the ease and standard of their own driving performances. These subjective feelings differed significantly between the two groups throughout the experiment. The SD group’s scores on the NASA RTLX and SSQ questionnaires increased with time and the CD group’s perceived driving quality and ease increased with time, reaching non-significantly different levels from their usual car driving standards by the end of the drive. These findings show the existence of a fairly good match between real-life and simulated driving, which stabilized six minutes after the start of the test, regardless of whether the road was straight or curved. These objective findings and subjective assessments suggest possible ways of improving the match between drivers’ performances on simulators and their real-life driving behavior.
... In addition, CAVs do not violate speed limits or other traffic rules. CAVs are expected to reduce the fuel consumption by dampening traffic oscillations and following the speed limits (Pakdamanian et al., 2018;Ramin et al., 2018;Sheng et al., 2019;Adomah et al., 2021;Gaweesh et al., 2021;Stogios et al., 2018;Qu et al., 2020). However, other researchers have reported that CAVs may reduce capacity if they operate conservatively with lower speeds and longer headways (Hartmann et al., 2017). ...
In this research, we propose novel mathematical models and algorithms for optimizing connected and automated vehicles’ (CAVs) trajectories at freeway weaving segments assuming 100 percent CAV market penetration. The proposed system receives vehicle arrival information and generates optimal trajectories that are relayed to the CAVs. The algorithm simulates vehicle arrivals and develops optimal vehicle trajectories on a 2-lane weaving section (one mainline and one auxiliary lane). We test the optimization algorithm under a variety of demand scenarios and using real-world arrival data. Results suggest that the proposed algorithm increases the average travel speed and capacity by 12 to 16% and up to 11%, respectively when compared to HCM estimates. Also, our case study shows the algorithm increases the average speed by 17%, 30%, and 38% for minimum time headways of 1.7 s, 1.4 s, and 1 s, respectively, compared to conventional vehicles.
... Nevertheless, due to current technology limitations and legal restrictions, AVs may still need to handover the control back to drivers occasionally (e.g., under challenging driving conditions beyond the automated systems' capabilities) [39]. In such cases, AVs would initiate takeover requests (TORs) and alert drivers via auditory, visual, or vibrotactile modalities [44,47,61] so that the drivers can resume manual driving in a timely manner. However, there are challenges in making drivers safely take over control. ...
Full-text available
Automated vehicles promise a future where drivers can engage in non-driving tasks without hands on the steering wheels for a prolonged period. Nevertheless, automated vehicles may still need to occasionally hand the control back to drivers due to technology limitations and legal requirements. While some systems determine the need for driver takeover using driver context and road condition to initiate a takeover request, studies show that the driver may not react to it. We present DeepTake, a novel deep neural network-based framework that predicts multiple aspects of takeover behavior to ensure that the driver is able to safely take over the control when engaged in non-driving tasks. Using features from vehicle data, driver biometrics, and subjective measurements, DeepTake predicts the driver's intention, time, and quality of takeover. We evaluate DeepTake performance using multiple evaluation metrics. Results show that DeepTake reliably predicts the takeover intention, time, and quality, with an accuracy of 96%, 93%, and 83%, respectively. Results also indicate that DeepTake outperforms previous state-of-the-art methods on predicting driver takeover time and quality. Our findings have implications for the algorithm development of driver monitoring and state detection.
... In this work, the impact this system has on drivers fatigue during monotonous driving is investigated by monitoring the standard deviation of lateral position. The benefit of haptic feedback is also evaluated in [116], where the authors found that whole-body feedback during curves can be effective in avoiding hazardous situations when take-over is needed. Furthermore, in [117], a real-time adaptable haptic feedback is proposed based on the level of distraction of the driver. ...
Full-text available
Driving simulation has become a very useful tool for vehicle design and research in industry and educational institutes. This paper provides a review of driving simulator components, including the vehicle dynamics model, the motion system, and the virtual environment, and how they interact with the human perceptual system in order to create the illusion of the driving. In addition, a sample of current state-of-the-art vehicle simulators and algorithms are described. Finally, current applications are discussed, such as driver-centered studies, chassis and powertrain design, and autonomous systems development.
... Traffic crashes are rare events occurring in a complex process of human responses to exogenous factors such as traffic flow characteristics, roadway geometry, road users, and environmental conditions (Abhari et al., 2018;Azizi et al., 2012;Hosseinzadeh et al., 2020Hosseinzadeh et al., , 2021Mokhtarimousavi et al., 2019;Mousavi et al., 2019aMousavi et al., , 2019bNasr Esfahani and Song, 2020;Pakdamanian et al., 2018;Parsa et al., 2020;Rahimi et al., 2020;Razi-Ardakani et al., 2018Shirani-Bidabadi et al., 2020;Younes et al., 2020). Analyzing these rare events and defining their likelihood requires available data not only related to all contributing factors but also to their complicated interactions, which is nearly impossible to be achieved (Mannering et al., 2016;Mannering and Bhat, 2014). ...
Traditional hierarchical modeling has been proposed to account for unobserved heterogeneity in the crash analysis. Previous studies investigated the grouping of individual observations between different clusters by considering a single random factor at level-2 of data structure. This approach, however, hinders exploring the possible crossed effects of additional random factors at the level-2 of data hierarchy on the response variable. The current study aims to expand the previous attempts by introducing the concept of Cross-Classified Random Effects Modeling (CCREM) and utilizing crossed random intercepts to account for the crossed effects of two random factors. Aligned with the Connected Vehicle Pilot Deployment Program on Interstate-80 (I-80), this paper intends to cluster critical crashes, involving fatal or incapacitating injuries, versus non-critical crashes through a 402-mile I-80 in Wyoming during the first five months of 2017. Aggregated environmental conditions were conflated with disaggregated real-time traffic observations. Concerning road surface conditions and longitudinal grade categories, four Logistic Regression models were calibrated under Bayesian Inference. Model-1 considered these two factors as fixed parameters; however, in each of Model-2 and Model-3, one of these factors was treated as a random intercept. Model-4 considered both factors as random intercepts and investigated their crossed effect on the critical crash probability. Model-4 outperformed the others and showed that the maximum probability of critical crashes arises on dry pavements and steep downgrades. In contrast, the combined effect of wet pavements and less steep downgrades is associated with the minimum risk of critical crashes. It was revealed that the probability of critical crashes varies at any given value of real-time traffic-related predictors according to different combinations of longitudinal grade and road surface conditions. This finding indicates an essential need for Active Traffic Management to timely apply interventions not only based on real-time traffic-related predictors but also according to various combinations of environmental conditions.
... For the purpose of objectively obtaining the psychophysical state of the driver as accurate as possible, there is a need to shift from the simple questionnaires to a direct assessment of driver physiological responses and driver behavioral pattern [5,18]. Applying neuropsychological [10,11] and physiological measurements [6,14,18] on drivers to investigate the relationship between mental behavior and performance while taking-over could provide us a profound understanding of what modalities provide useful TOR for autonomous vehicles. In this preliminary study, we focus on the influence of workload, stress and the alarm type on takeover behavior on two recruited participants with the help of physiological monitoring systems. ...
Conference Paper
In this work, we introduce the preliminary analysis of driver’s physiological data after receiving take-over request (TOR). Studies have shown that physiological measurements on drivers may provide better insights into the cognitive behavior and performance of drivers. Our goal is to examine the effect of two common TOR modalities (visual-auditory and generic auditory), in the limited take-over time budget, on psychophysical states and take-over behavior of the drivers. We applied multimodal physiological data streams -i.e. eye-tracker, EEG, GSR and PPG to have a comprehensive overview of driver’s workload, stress and reaction time for each TOR modality. The preliminary results suggest that visual-auditory modality leads to a safer take-over behavior than generic auditory tone. EEG and heart rate variability results showed a significantly greater engagement on the visual-auditory TOR than for the auditory TOR. Results of this study can be used to investigate the safer modality inducing the least startle reactions.
Distracted driving is known to be one of the main causes of crashes in the United States, accounting for about 40% of all crashes. Drivers’ situational awareness, decision-making, and driving performance are impaired as a result of temporarily diverting their attention from the primary task of driving to other unrelated tasks. Detecting driver distraction would help in adapting the most effective countermeasures. To tackle this problem, we employed a random forest (RF) classifier, one of the best classifiers that has attained promising results for a wide range of problems. Here, we trained RF using the data collected from a driving simulator, in which 92 participants drove under six different distraction scenarios of handheld calling, hands-free calling, texting, voice command, clothing, and eating/drinking on four different road classes (rural collector, freeway, urban arterial, and local road in a school zone). Various driving performance measures such as speed, acceleration, throttle, lane changing, brake, collision, and offset from the lane center were investigated. Using the RF method, we achieved 76.5% prediction accuracy on the independent test set, which is over 8.2% better than results reported in previous studies. We also obtained a 76.6% true positive rate, which is 14% better than those reported in previous studies. Such results demonstrate the preference of RF over other machine learning methods to identify driving distractions.
Full-text available
Automated driving is no longer a future scenario. Several automotive OEM have already presented automated vehicles, which do not require driver's constant attention on the road. But, there are still some challenges to solve before series vehicles can pass from assisted driving to highly automated driving [1; 2]. A principal research question to deal with is how to design Take-Over-Requests (TOR) with respect to the human machine interface (HMI) and reaction times to comply with a TOR. On this account, a driving simulator study with 44 drivers has been conducted at the Fraunhofer Institute for Industrial Engineering. The study took place in a highly automated driving vehicle which controlled longitudinal and lateral control on a highway scenario. Approaching a construction site different TOR strategies were presented. Within this study the time users needed to react on a TOR was measured for a highway scenario. The drivers were fully distracted by a secondary task, a challenging quiz game on a mobile phone. The different TOR strategies comprised a variation of the location for TOR presentation (integrated mobile phone or in-vehicle HMI) as well as a variation of the TOR modality (TOR with brake jerk/without brake jerk). This paper will present and discuss the results in terms of reaction times and driver behavior strategies to comply with the TOR. It delivers advice on the design of transition strategies between automated and manual driving.
Full-text available
Automated automobiles will be on our roads within the next decade but the role of the driver has not yet been formerly recognised or designed. Rather, the driver is often left in a passive monitoring role until they are required to reclaim control from the vehicle. This research aimed to test the idea of driver-initiated automation, in which the automation offers decision support that can be either accepted or ignored. The test case examined a combination of lateral and longitudinal control in addition to an auto-overtake system. Despite putting the driver in control of the automated systems by enabling them to accept or ignore behavioural suggestions (e.g. overtake), there were still issues associated with increased workload and decreased trust. These issues are likely to have arisen due to the way in which the automated system has been designed. Recommendations for improvements in systems design have been made which are likely to improve trust and make the role of the driver more transparent concerning their authority over the automated system. Copyright © 2015 Elsevier Ltd and The Ergonomics Society. All rights reserved.
Full-text available
This study explores, in the context of semi-autonomous driving, how the content of the verbalized message accompanying the car’s autonomous action affects the driver’s attitude and safety performance. Using a driving simulator with an auto-braking function, we tested different messages that provided advance explanation of the car’s imminent autonomous action. Messages providing only “how” information describing actions (e.g., “The car is braking”) led to poor driving performance, whereas “why” information describing reasoning for actions (e.g., “Obstacle ahead”) was preferred by drivers and led to better driving performance. Providing both “how and why” resulted in the safest driving performance but increased negative feelings in drivers. These results suggest that, to increase overall safety, car makers need to attend not only to the design of autonomous actions but also to the right way to explain these actions to the drivers.
Full-text available
Many studies have investigated the effect of vibrotactile cues on task performance, but a wide range of cue and task types have made findings difficult to interpret without a quantitative synthesis. This report addresses that need by reviewing the effectiveness of vibrotactile cues in a meta-analysis of 45 studies. When added to a baseline task or to existing visual cues, vibrotactile cues enhanced task performance. When vibrotactile cues replaced visual cues; however, some effects were attenuated and others moderated by cue information complexity. To summarize such moderating effects, vibrotactile alerts are an effective replacement for visual alerts, but vibrotactile direction cues are not effective when replacing visual direction cues. This meta-analysis of vibrotactile applications underscores the benefits of vibrotactile and multimodal displays, highlights conditions in which vibrotactile cues are particularly effective, and identifies areas in need of further investigation.
Full-text available
There is substantial evidence that driving skills improve during driver training, but the long-term safety benefit of such formal training remains unproven. Restricting the exposure of newly licensed drivers to more hazardous driving circumstances, as in graduated driver licensing (GDL) regimes, demonstrably reduces crash risk, but drivers remain at risk after the restrictions are eased. GDL and most other licensing regimes advocate increased basic training and practice, but thereafter require neither advanced training nor systematic increase in exposure to risk. This assumes that basic skills acquired during formal training will transfer positively to new and more demanding traffic circumstances. This paper reviews the theoretical basis for these assumptions and offers a way of systematically identifying the extent of transfer desired. It is concluded that there is little theoretical or empirical foundation for the supposition that what is learned during or after training will have a safety benefit in later driving.
As automotive manufacturers are increasing the amount of technology and automation available in vehicles, drivers must increase their understanding of how to properly use these technologies before the full safety benefits can be realized. Unfortunately, recent work has suggested that even drivers with the most advanced vehicles often have little understanding of the available technology (McDonald et al., 2017). Yet, this poor understanding of advanced driver assistance systems cannot entirely be blamed on the driver; other factors, such as the availability and quality of instructional sources are also at play (e.g., Abraham et al., 2017). Facilitating access to safety-critical information and standardizing instructional and operational components are two potential routes to increase drivers’ knowledge. However, target areas must be identified. The current study reviewed the degree to which automated systems in passenger vehicles and related information varied among those moderately-priced and luxury models marketed and sold in the United States. Information regarding ten existing longitudinal (e.g., adaptive cruise control) and ten lateral control automated systems (i.e., those that provide sustained lateral control and lane centering) were gathered from OEM websites and operator manuals. Information was coded and synthesized, including the name of the system, the stated functionality, operation, and system constraints as well as the source of the information. Results of this exercise suggest a number of target areas for researchers, OEMs and policy makers to consider in attempts to increase accessibility, knowledge, and ultimately safe usage of these technologies. First, a high degree of variability was observed even among basic characteristics of these technologies such as the system name. This was particularly an issue among luxury automated longitudinal control systems where every system had a fairly different name. While the activation methods were largely consistent across both lateral and longitudinal systems, the deactivation methods varied a great deal both in the allowance of alternative methods and the type of method (if allowed). Moreover, all the systems provided some form of visual feedback to the driver, but this feedback still varied greatly, particularly among lateral systems, in the location of the feedback and the modality that supplemental feedback was presented. System limits for both automated lateral and longitudinal systems were also elucidated. The majority of OEMs reported five types of system limits: curves, stopped vehicles, weather, small obstacles (e.g., pedestrians, bicyclists), and hills. Similar to the lateral control systems, two OEMs also reported difficulty handling an occupied adjacent lane. These limits were elucidated primarily through the operator manual for each respective model. While some information concerning these limits was available on the website, generally information gathered at this location was restricted. This limited accessibility to information is likely leading to individuals not getting this information, resulting in the poor understanding of these systems observed among owners and even dealers of this technology (Abraham et al., 2017; McDonald et al., 2017). When these individuals do take the time to read the operator manual and do get this information, the variability across systems likely limits their understanding and building of a mental model that supports safe usage over time as they encounter other similar systems.
Vibrotactile stimuli can be effective as warning signals, but their effectiveness as directional take-over requests in automated driving is yet unknown. This study aimed to investigate the correct response rate, reaction times, and eye and head orientation for static versus dynamic directional take-over requests presented via vibrating motors in the driver seat. In a driving simulator, eighteen participants performed three sessions: 1) a session involving no driving (Baseline), 2) driving a highly automated car without additional task (HAD), and 3) driving a highly automated car while performing a mentally demanding task (N-Back). Per session, participants received four directional static (in the left or right part of the seat) and four dynamic (moving from one side towards the opposite left or right of the seat) take-over requests via two 6 × 4 motor matrices embedded in the seat back and bottom. In the Baseline condition, participants reported whether the cue was left or right, and in the HAD and N-Back conditions participants had to change lanes to the left or to the right according to the directional cue. The correct response rate was operationalized as the accuracy of the self-reported direction (Baseline session) and the accuracy of the lane change direction (HAD & N-Back sessions). The results showed that the correct response rate ranged between 94% for static patterns in the Baseline session and 74% for dynamic patterns in the NBack session, although these effects were not statistically significant. Steering wheel touch and steering input reaction times were approximately 200 ms faster for static patterns than for dynamic ones. Eye tracking results revealed a correspondence between head/eye-gaze direction and lane change direction, and showed that head and eye-gaze movements where initiated faster for static vibrations than for dynamic ones. In conclusion, vibrotactile stimuli presented via the driver seat are effective as warnings, but their effectiveness as directional take-over requests may be limited. The present study may encourage further investigation into how to get drivers safely back into the loop.
Conference Paper
In this paper, we describe the design and evaluation of a vibrotactile driver's seat that is used to display spatial information during two driving tasks. Many studies have recently shown the effectiveness of haptic and vibrotactile feedback to augment collision warning systems in automobiles. Simultaneously, driver distraction and situational awareness have been identified as significant safety issues in all areas of transportation. We hypothesize that vibrotactile feedback may be used to enhance and improve spatial awareness while driving if it is used continuously and naturally so that it is part of the normal operation of the automobile. We designed a tactile feedback seat from low cost pager motors and characterized the spatial resolution of the seat. We then developed a driving simulation in which the location of vehicles behind and next to the driver's vehicle is communicated through vibrotactile feedback from the seat back. The effectiveness of the seat was evaluated in two driving tasks designated commuting and racing. In the commuting exercise, the test subjects (N=12) maintained a target speed while simultaneously avoiding other vehicles and performing a secondary task. A "near-miss" blind spot recording method was used to evaluate the effect of the feedback in reducing hazard exposure. In the racing exercise, the test subjects (N=10) raced other virtual competitors while using the feedback to maintain awareness of other vehicles in close proximity. Effectiveness was measured by comparing the accumulated time that cars were blocked behind the driver's car. Three feedback conditions were tested: only vibrotactile feedback, rear view mirror and vibrotactile feedback, rear view mirror only. Our preliminary results showed that vibrotactile feedback used in conjunction with the rear view mirror improved performance over using just the rear view mirror. We discuss some of the challenges of creating driving simulations and evaluation metrics that are both rea- - listic and repeatable.
Effects of alarm modality and alarm reliability on workload, trust, and driving performance
  • S J Shah
  • J P Bliss
  • E T Chancey
  • J Christopher Brill
Shah, S. J., Bliss, J. P., Chancey, E. T., & Christopher Brill, J. (2015). Effects of alarm modality and alarm reliability on workload, trust, and driving performance. Proceedings of the Human Factors and Ergonomics Society, 2015-January(2008), 1535-1539.