Conference PaperPDF Available

Ghost Busting: A Novel On-Road Exploration of External HMIs for Autonomous Vehicles


Abstract and Figures

The absence of a human driver in future autonomous vehicles means that explicit pedestrian-driver communication is not possible. Building on the novel ‘Ghost Driver’ methodology to emulate an autonomous vehicle, we developed prototype external human-machine interfaces to replace existing cues, and report preliminary, qualitative findings captured from a sample of pedestrians (n=64) who encountered the vehicle when crossing the road, as well as reflecting on the method.
Content may be subject to copyright.
Ghost Busting: A Novel On-Road Exploration of
External HMIs for Autonomous Vehicles
David R. Large, Madeline Hallewell, Xuekun Li, Catherine Harvey, Gary Burnett
Human Factors Research Group, University of Nottingham
The absence of a human driver in future autonomous vehicles means that explicit pedestrian-driver
communication is not possible. Building on the novel ‘Ghost Driver’ methodology to emulate an
autonomous vehicle, we developed prototype external human-machine interfaces to replace existing
cues, and report preliminary, qualitative findings captured from a sample of pedestrians (n=64) who
encountered the vehicle when crossing the road, as well as reflecting on the method.
Autonomous vehicles, external HMI, Ghost Driver
There has been considerable interest amongst behavioural scientists in the potential impact of
highly and fully autonomous ‘self-driving’ vehicles (AVs) on the behaviour of pedestrians. These
vehicles, operating at SAE level 4 or 5, are unlikely to have a human driver present, and as such,
explicit visual cues (head, eye, hand/arm gestures etc.) that are traditionally exchanged between a
driver and a pedestrian, will be absent. Typically, these aim to establish a mutual understanding of
perception (Have you seen me?) and intent (Will you give way?) (Merat et al., 2018), and are
important to overall traffic safety especially in low-speed crossing scenarios in complex urban
settings (Lee et al., 2020). However, studying genuine, naturalistic behaviours of people responding
to AVs presents a number of challenges (limited public trials, requirement to have a ‘safety driver’
present etc.). A novel solution is to use a Wizard-of-Oz (WoZ) approach to give the appearance that
the car is driving on its own, even when it is not. This can be achieved by hiding the driver using a
bespoke seat cover (aka ‘Ghost Driver’ method) (Rothenbücher et al., 2016). To date, no such
studies have been reported in the UK. In addition, the Ghost Driver method has not been employed
specifically to evaluate external human-machine interfaces (eHMIs).
A ‘Ghost Driver’ WoZ study was devised in which the driver was hidden in a bespoke seat-suit,
thereby giving the appearance that the vehicle (Nissan Leaf) was driving by itself (Figure 1). The
seat-suit was designed and fabricated to enable the driver to maintain safe control of the vehicle,
whilst ensuring that they could not be seen by a passing pedestrian glancing into the vehicle. Three
eHMIs were created. These were informed by the literature and prototyped using an individually
addressable RGB-LED matrix and strip attached to the outside of the vehicle (front of bonnet and at
top of windscreen, respectively). The eHMIs were programmed using an Arduino Mega board and
manipulated with push-button controls from inside the vehicle. The eHMI designs employed
varying degrees of anthropomorphism (implicit, explicit, low) to aid interpretation and build trust.
The first (implicit) utilised the LED strip only and mimicked the pupillary response of an eye:
lateral movement demonstrated scanning/awareness, and blinking provided an implicit cue of the
vehicle’s intention to give way. The second (explicit) presented a face and eyes on the LED matrix
to scan the road and used humanlike language to ‘talk’ to the pedestrians (Figure 1). The third (low)
used a vehicle icon and vehicle-centric language on the LED matrix. For each eHMI design, four
modes were created: scanning, giving way (pedestrian on right), giving way (pedestrian on left) and
giving way (pedestrians on both sides of road). A second researcher, seated in the back seat of the
ego-vehicle, controlled the current state of the eHMI in response to the observable pedestrians in the
vicinity of the vehicle. The study took place on the extensive University of Nottingham campus and
a circuitous route was selected that included several marked and unmarked crossings. Over 10 hours
of video data were captured using a dashcam and GoPro recorder to document pedestrians’
responses to the ‘driverless’ vehicle and eHMIs. In addition, researchers were located at specific
crossing points, and invited pedestrians who encountered the vehicle to complete a survey.
Results and Discussion
Video analysis is ongoing. Here, we report qualitative findings, including illustrative comments and
responses related to the vehicle and eHMI concepts. Results show that over eighty percent of
respondents believed that the car was driving on its own (“There was no driver, just a passenger in
the back passenger seat”), and this surprised many people (“I was mostly just shocked, so I stopped
and observed”). Nevertheless, many people still appeared to interact with the vehicle as if a driver
were present (e.g. waving to thank the vehicle for stopping), highlighting the value of an eHMI to
replace interactions with a driver, and supporting the inclusion of ‘human’ elements. Comments
suggest the eHMIs impacted the trust relationship (“I was a bit curious about why the car
stopped...when I saw the screen that explained a lot”), with most comments suggesting support for
the concepts (“I quickly became aware that it was helping me to cross”, “[the eHMI] matched
observed behaviour of vehicle”, “I understood that the eyes were looking out for people”), whereas
others were more cautious towards the technology (“Would need to encounter it more before I fully
trusted it”), and a few respondents admitted being confused by the messages (“I wasn't entirely sure
what the message was conveying”). This did not necessarily change pedestrians’ crossing
behaviour, with most respondents still stating that they crossed in front of the vehicle as they
normally would. It did, however, inspire some additional curiosity: “Had seen it…earlier and was
curious to see if it would stop or not.The different eHMIs appeared to inspire different emotional
responses. For example, the explicit anthropomorphism encouraged positivity – smiling, laughing
etc., whereas responses to the low anthropomorphism were more perfunctory; survey ratings
indicated that the latter provided the highest clarity in conveying its intended messages. Overall,
initial findings support the use of a hidden ‘ghost driver’ to explore pedestrians’ interactions with an
AV, with observed behaviour suggesting high ecological validity. In addition, explicit
communication using eHMIs (employing elements of anthropomorphism) appears to encourage safe
crossing behaviours, help pedestrians interpret vehicle behaviour and intent, and increase their
confidence and build appropriate trust when interacting with a driverless vehicle.
Figure 1: Driver in seat-suit (left); hidden driver operating car (centre); example eHMIs (right)
The study was undertaken as part of the ServCity project (, which is
funded by the UK Innovation Agency (Innovate UK) and Centre for Connected & Autonomous
Vehicles (CCAV) (Grant number: 105091). We would like to thank Nissan Technical Centre
Europe (NTCE) for use of the Nissan Leaf vehicle and their support and guidance throughout.
Lee, Y. M., Madigan, R., Giles, O., Garach‑Morcillo, L., Markkula, G., Fox, C., . . . Merat, N.
(2020). Road users rarely use explicit communication when interacting in today’s traffic:
implications for automated vehicles. Cognition, Technology and Work.
Merat, N., Lee, Y. M., Markkula, G., Uttley, J., Camara, F., Fox, C., . . . Schieben, A. (2018). How
do we study pedestrian interaction with automated vehicles? Preliminary findings from the
European interACT project. In G. Meyer & S. Beiker (Eds.), Lecture Notes in Mobility (pp.
21-33). Cham: Springer.
Rothenbücher, D., Li, J., Sirkin, D., Mok, B., & Ju, W. (2016). Ghost driver: A field study
investigating the interaction between pedestrians and driverless vehicles. Paper presented at
the 2016 25th IEEE international symposium on robot and human interactive communication
... These include a tinted windshield (Bindschädel et al., 2023;Chen et al., 2020), a dummy steering wheel (Habibovic et al., 2018), a mirror film (Liu & Hirayama, 2023;Liu et al., 2021), or the car being controlled from the passenger seat (Rodríguez Palmeiro, Van der Kint, Vissers, et al., 2018). However, the most common option is that of a seat suit (Faas & Baumann, 2021;Fuest et al., 2018;Hensch et al., 2020;Joisten et al., 2020;Large et al., 2023;Li et al., 2020;Liang et al., 2016;Loew et al., 2022;Rothenbücher et al., 2016;Taima & Daimon, 2023;Wang et al., 2021). In our study, the vehicle was operated by a concealed driver in a costume (see Figure 1) in half of the encounters, emulating an AV. ...
... It is unknown how many participants believed the vehicle was driving automatically. Previous Wizard-of-Oz research showed believability percentages ranging from 97 to 100% (Faas & Baumann, 2021;Habibovic et al., 2018;Joisten et al., 2020), but also 60% to 88% (Currano et al., 2018;Faas & Baumann, 2019;Hensch et al., 2020;Large et al., 2023;Li et al., 2020;Moore et al., 2019;Rothenbücher et al., 2016), and 40% (Rodríguez Palmeiro, Van der Kint, Vissers, et al., 2018). These percentages appear to be context-dependent, for example, whether a modernlooking vehicle was used. ...
Full-text available
As automated vehicles (AVs) become increasingly popular, the question arises as to how cyclists will interact with such vehicles. Our research investigated whether cyclists spontaneously notice if a vehicle is driverless, how well they perform a driver-detection task when explicitly instructed, and how they complete this task. Using a Wizard-of-Oz method, 37 participants cycled a designated route and encountered an AV multiple times in two experimental sessions. In Session 1, participants cycled the route uninstructed, while in Session 2, they were instructed to verbally report whether they detected the presence or absence of a driver. Additionally, we recorded participants’ gaze behaviour with eye-tracking and their responses in post-session interviews. The interviews revealed that 30% of the cyclists spontaneously mentioned the absence of a driver (Session 1), and when instructed (Session 2), they detected the absence and presence of the driver with 93% accuracy. The eye-tracking data revealed that cyclists looked more frequently and longer at the vehicle in Session 2 compared to Session 1. Furthermore, participants exhibited intermittent sampling of the vehicle, and looked in front of the vehicle when it was far away and towards the windshield region when it was closer. The post-session interviews indicated that participants were curious but felt safe, and reported a need to receive information about the AV’s driving state. In conclusion, cyclists can detect the absence of a driver in the AV, and this detection may influence their perceptions of safety. Further research is needed to explore these findings in real-world traffic conditions.
Full-text available
To be successful, automated vehicles (AVs) need to be able to manoeuvre in mixed traffic in a way that will be accepted by road users, and maximises traffic safety and efficiency. A likely prerequisite for this success is for AVs to be able to communicate effectively with other road users in a complex traffic environment. The current study, conducted as part of the European project interACT, investigates the communication strategies used by drivers and pedestrians while crossing the road at six observed locations, across three European countries. In total, 701 road user interactions were observed and annotated, using an observation protocol developed for this purpose. The observation protocols identified 20 event categories, observed from the approaching vehicles/drivers and pedestrians. These included information about movement, looking behaviour, hand gestures, and signals used, as well as some demographic data. These observations illustrated that explicit communication techniques, such as honking, flashing headlights by drivers, or hand gestures by drivers and pedestrians, rarely occurred. This observation was consistent across sites. In addition, a follow-on questionnaire, administered to a sub-set of the observed pedestrians after crossing the road, found that when contemplating a crossing, pedestrians were more likely to use vehicle-based behaviour, rather than communication cues from the driver. Overall, the findings suggest that vehicle-based movement information such as yielding cues are more likely to be used by pedestrians while crossing the road, compared to explicit communication cues from drivers, although some cultural differences were observed. The implications of these findings are discussed with respect to design of suitable external interfaces and communication of intent by future automated vehicles.
This paper provides an overview of a set of behavioural studies, conducted as part of the European project interACT, to understand road user behaviour in current urban settings. The paper reports on a number of methodologies used to understand how humans currently interact in urban traffic, in order to establish what information would be useful for the design of future AVs, when interacting with other road users, especially pedestrians. In addition to summarising the results from a number of observation studies, we report on preliminary results from Virtual Reality studies, investigating if, in the absence of a human vehicle controller, externally presented interfaces can be used for communication between AVs and pedestrians. Finally, an overview of the mathematical and computational modelling techniques used to understand how AV and pedestrian behaviour can be both cooperative, and effective is provided. The hope is that future AVs can be designed with an understanding of how humans cooperate and communicate in mixed traffic, promoting good traffic flow, user acceptance and user trust.