Development of an Immersive Simulation Platform to Study Interactions Between Automated Vehicles and Pedestrians

Abstract

The importance of informal communication between manual vehicles drivers and pedestrians in order to prevent misinterpretation, and thus accidents, in road-crossing situations has been widely shown in the literature. Such crucial communication consequently raises the issue of the introduction of automated vehicles (AVs) on the roads, in which case the status of the driver becomes less obvious. In this paper, we present a novel simulation platform, the V-HCD, allowing the conduct of immersive experimentations, both from the pedestrian's and the driver's point of view. This platform will be used to study the acceptance of the automated vehicle for the European SUaaVE project, and further to support the human-centred design of a future empathic AV.

Full text

Development of an Immersive Simulation Platform to Study
Interactions Between Automated Vehicles and Pedestrians
Lucie Lévêque1, Thierry Bellet1, Jean-Charles Bornard2, Jonathan Deniel1, Maud Ranchet1,
Estelle De Baere1, Bertrand Richard1
1Laboratory of Ergonomics and Cognitive Sciences applied to Transport, Université Gustave Eiffel, Lyon, France
2Engineering Systems International (ESI) Group, CIVITEC, Lyon, France
Keywords: Vehicle/pedestrian interactions, automated vehicle (AV), simulation, human-centred design
Abstract: The importance of informal communication between manual vehicles drivers and pedestrians in order to
prevent misinterpretation, and thus accidents, in road-crossing situations has been widely shown in the
literature. Such crucial communication consequently raises the issue of the introduction of automated
vehicles (AVs) on the roads, in which case the status of the driver becomes less obvious. In this paper, we
present a novel simulation platform, the V-HCD, allowing the conduct of immersive experimentations, both
from the pedestrian’s and the driver’s point of view. This platform will be used to study the acceptance of
the automated vehicle for the European SUaaVE project, and further to support the human-centred design of
a future empathic AV.
1. INTRODUCTION
With the rapid development of new technologies and
automation, the introduction of automated vehicles
(AVs), or connected automated vehicles (CAVs),
i.e., cars without active drivers, offers a strong
potential to increase both traffic safety and
accessibility. In fact, in the near future, the traffic
system will be shared between fully automated
vehicles, partially automated vehicles, and manually
driven vehicles (Litman, 2020). However, it should
be noted that automated vehicles (i.e., levels 3 to 5
of driving automation (SAE, 2018)) will not only
coexist with more conventional vehicles (i.e., levels
0 to 2 of driving automation), but also with
vulnerable road users, including pedestrians.
Generally, pedestrians are considered an important
indicator of a society’s health and safety. With a
view to contribute to a safe traffic system, and
consequently to increase the acceptance of AVs, one
key challenge is therefore to investigate how the
latter interact with pedestrians.
To achieve safe interactions, manual vehicle
drivers and pedestrians need to share their
understanding and awareness of the traffic situation
(Endsley, 1995; Bellet et al., 2009). Otherwise,
critical conflicts may occur; 21% of fatal road traffic
accidents happen to involve pedestrians (WHO,
2015). It is crucial to account that misinterpretation
of others’ intentions is one of the main causes of
accidents involving pedestrians (Habibovic et al.,
2012). This is particularly true in the case of road
crossing decision-making, and especially when
priority rules are unclear (e.g., absence of zebra
crossing). In such a context, pedestrians and manual
drivers frequently interact using non-verbal
communication to clarify their intentions. Several
studies have shown the importance of this informal
communication in the literature.
For instance, Schmidt et al. found that
pedestrians who want to cross a street tend to look at
the approaching vehicle to get acknowledgement
from the driver; if the driver returns their eye
contact, pedestrians assume that they have been seen
and that they have achieved a mutual understanding
(Schmidt et al., 2009). Similarly, Sucha et al. found
that pedestrian’s decision to cross, as well as their
feeling of safety, are directly impacted by various
signals provided by the driver, like eye contact,
postures, waving hand, or flashing lights (Sucha et
al., 2017). Such conclusions were also drawn by
Rasouli et al. who showed that the most prominent
signal to transmit pedestrians’ crossing intention is
looking, or at least glancing, towards oncoming

traffic (Rasouli et al., 2017). Finally, in their study,
Schneemann et al. found that, when pedestrians
interact with vehicles at low speed, they tend to rely
on eye contact with the driver; whereas, at faster
speed, they generally base their decisions on the
dynamics of the vehicle (Schneemann et al., 2017).
To summarise, all these studies clearly indicate
that an active communication between pedestrians
and manual vehicle drivers is a crucial element to
manage situational risks, to support pedestrians’
decision-making, and to increase their safety while
crossing roads. A key concern regarding the
introduction of automated vehicles public roads is
therefore due to the changing status of the drivers.
Indeed, AVs may negatively impact interactions
with pedestrians as they will not be able to rely on
cues from drivers’ behaviours anymore, potentially
leading to uncertainty and mistrust (Vissers et al.,
2017). Malmsten Lundgren et al. suggested that the
introduction of automated vehicles in the urban
context may lead to a notable change in how
pedestrians experience AVs compared to
conventional vehicles (Malmsten Lundgren et al.,
2016). In their study, pedestrians rated eye contact
with a driver as promoting safe interaction; whereas
apparent driver distraction in an AV (e.g., phoning
or reading the newspaper) tended to increase
pedestrians’ anxiety.
2. RESEARCH OBJECTIVES
Regarding automated vehicles, it should be noted
that one key challenge is not only to study the
acceptability (i.e., before use), but also the
acceptance (i.e., after use) (Schade et al., 2003;
Distler et al., 2018). However, automated and/or
autonomous vehicles are not commonly traveling on
European roads yet. Therefore, the only way to be
able to study the aforesaid acceptance is by using
new generation immersive simulation tools, which
allow users to plan for the future thanks to virtual
reality (Kyriadikis et al., 2019). By developing such
immersive simulation environments, it makes it
possible to invite real humans to practically
experience future technologies and situations.
Investigating the adequacy of future systems to end
users’ needs, exploring potential risks, and
evaluating the acceptance then become conceivable.
The in-depth study of how pedestrians and
automated vehicles interact with each other is a
crucial issue for the SUaaVE (SUpporting
acceptance of automated VEhicle) project. In this
context, developing an immersive platform
gathering such virtual reality tools would allow the
people of today to experience the AVs of tomorrow.
In order to reach this goal, Université Gustave Eiffel
(ex-IFSTTAR) and ESI/CIVITEC decided to create
such an immersive simulation platform for the
SUaaVE project, through a pre-existing tool.
3. DESIGN AND DEVELOPMENT
OF A V-HCD PLATFORM
With a view to better understand and integrate users’
needs in the design of innovative advanced driving
aid systems (ADAS), IFSTTAR and ESI Group
jointly developed a virtual human-centred design
platform; the V-HCD (Bellet et al., 2012, 2018). As
a virtual simulation toolbox, the V-HCD is able to
handle human-based simulation, that is to say based
on a virtual driver model, to virtually assess accident
risks as well as the potential benefits of future
ADAS.
The V-HCD integrative platform is made of two
main components: a virtual driver, and a virtual
prototyping platform. The virtual driver is based on
the cognitive model COSMODRIVE, i.e., COgnitive
Simulation Model of the DRIVEr (Bellet et al.,
2009). The virtual prototyping platform, named ESI
Pro-SiVICTM, integrates simulated infrastructures,
road users, vehicle dynamics, and multi-technology
perception sensors (Gruyer et al., 2006).
In a recent study, Bellet et al. (Bellet et al., 2019)
used this platform for the human-centred design of a
driver monitoring system, with a view to identify
risks of collision caused by visual distraction during
driving, and in charge to manage human-machine
interactions in real time. Figure 1 represents an
example of simulation with the COSMODRIVE
model implemented in the V-HCD, to simulate the
effects of visual distraction on the driver’s situation
awareness.

Fig. 1: Illustration of the simulation of drivers’ visual distraction effects with the V-HCD.
Fig. 2: Illustration of an example of simulated
accident risks due to visual distraction with the
V-HCD platform.
In this previous project, the V-HCD was used in
the initial stages of the design process, during which
there was no prototype of the future system, and
therefore no experimentation with real humans nor
user testing was possible. In such an early stage,
making use of a user model, such as
COSMODRIVE, can enable a better comprehension
of the risks due to visual distraction depending on
the situational context. COSMODRIVE indeed
allowed the simulation of various driving scenarios
with different levels of driver distraction in order to
identify the most critical situations, as illustrated in
Figure 2. These critical scenarios were then used as
‘reference use cases’ (Bellet et al., 2019) to
consequently specify functionality to embed in
future ADAS with a view to prevent accidents.
Beyond this early use, the V-HCD can also be
used at more advanced stages of the design process.
Indeed, as soon as models and/or real or virtual
prototypes of the future ADAS are available, the V-
HCD can be used as an integrated simulator of: (1)
the ADAS, (2) the road environment, and (3) the
vehicle. Then, real humans can sit in the vehicle to
put themselves in the shoes of future users of this
ADAS, and consequently have a realistic experience
of this future technology before it becomes available
on real cars.
In the framework of the SUaaVE project, the
objective was to make changes to the original V-
HCD platform, in order to focus on the study of AV-
pedestrians interactions and, more precisely, on the
acceptance of such automated vehicles. As
introduced in the previous section, the latter are not
traveling on the roads yet; therefore, the only way to
investigate acceptance is through the use of virtual
immersion.
It is with this in mind that the new version of the
V-HCD platform was designed and developed for
the SUaaVE project, i.e.: to allow users today to
plunge into an immersive experience of interaction
with an autonomous vehicle, either from the
driver/passenger point of view interacting with a
virtual pedestrian, or from the pedestrian point of
view, willing to cross in front of an AV.

Fig. 3: Illustration of an example of scenario implemented on the V-HCD platform to study the interactions
between a pedestrian and an AV with a more or less attentive driver.
Figure 3 describes this perspective by presenting
a short scenario where a pedestrian is crossing the
street while an automated vehicle is approaching.
More precisely, Figure 3 (view 1) shows the
designer’s overview, where the parameters (e.g.,
cars colours, distances, speeds, accelerations) of
each object can be tuned.
Figure 3 (view 2) represents a possible
perspective of the aforementioned scenario, where a
human subject becomes the pedestrian. Thanks to a
virtual reality (VR) headset, they can turn their head
to check for oncoming traffic and take the pedestrian
crossing. The avatar settled in the oncoming AV can
then be positioned at different places (i.e., front or
rear seats), and participate in various activities (e.g.,
driving carefully, sleeping on the steering wheel,
talking to a rear passenger), as illustrated in Figure
4. The objective of these diverse situations is to
analyse whether, and to what extent, the decision to
cross is modified.
Finally, Figure 3 (view 3) corresponds to a more
traditional use of simulation, where a participant
takes the place of an occupant of the AV. In such a
case, they experience the situation either through the
use of a traditional driving simulator cabin, or thanks
to a VR headset creating a virtual cockpit. The
subject can also multi-task while driving (e.g.,
reading a book, playing a game). The subject can
hence experience diverse AV behavirous, and later
express their feelings about each of them.
Furthermore, different demeanours can be associated
with the pedestrian (e.g., crossing quickly, changing
their mind and stepping back).
When implemented simultaneously thanks to the
V-HCD, both perspectives (i.e., as a pedestrian and
as an AV occupant (driver or passenger),
respectively) can enable two participants to interact
with each other in a simulated world. In this
instance, the first volunteer sits in the driving
simulator, while the second one puts themselves in
the position of the pedestrian by means of the VR
headset. Both participants therefore experience the
same situation at the same time, but from a different
point of view. Each of them can witness the other’s
behaviour, such as the potential distraction of the
AV occupant, or the possible hesitation of the
pedestrian before crossing.
The V-HCD platform thereby becomes a set of
tools concomitantly integrating automated vehicles
(AVs), advanced driving aid systems (ADAS), and
different ways to immerse current road users in the
future. The AVs are customisable with tailored
behaviours originating from cognitive simulation,
allowing the fine-tuning of diverse situations while
maintaining necessary realism. Simulating ADAS in
the immersive experience of road users can further
help with the integration of future systems. To
summarise, as jointly developed by Université
Gustave Eiffel (ex-IFSTTAR) and ESI/CIVITEC,
the V-HCD allows the testing of AVs and ADAS, as
well as of their acceptance by the future end-users.

Fig. 4: Illustration of examples of driving scenarios implemented on the V-HCD platform to study the
interactions between a pedestrian and an AV.
4. USE OF THE V-HCD
PLATFORM FOR SUAAVE
In order to study how interactions between
pedestrians and automated vehicles may look like in
the future, and how these interactions may be
affected by the AV behaviour, Université Gustave
Eiffel will implement a two-phase experiment for
the SUaaVE project. The first phase will focus on
the AV passenger/driver’s point of view, whereas
the second phase will be dedicated to the
pedestrian’s point of view.
For the first phase of the experiment, i.e., from
the driver/passenger’s point of view, participants
will experience how the AV reacts when facing
diverse pedestrians’ road crossing behaviours (i.e.,
more or less expected and/or critical). For this phase,
two different conditions will be considered: with, or
without zebra crossing. Several AV behaviours are
implemented on the V-HCD to interact with the
pedestrian; they will be fully managed and
performed by vehicle automation. During each of
these alternatives, participants will have the
opportunity to use the horn to warn the pedestrian.
This way, it will be possible to measure if, and
when, they feel that the interaction is becoming too
critical, and potentially unacceptable. Participants
will afterwards be asked to assess the situational
criticality and the reaction of the AV according to
the pedestrian’s decisions and behaviours. After
having experienced all the scenarios, participants
will be invited to a semi-structured interview to
express their suggestions on how an “empathic”
automated vehicle should react when interacting
with pedestrians, and/or should inform the latter
about its decisions and reactions. At this level, the
objective of these final questions will be to collect
useful information for the future user-centric design
of SUaaVE’s empathic automated vehicle, namely
ALFRED (Automation Level Four and Reliable
Empathic Driver).
The second phase of the experiment will be
conducted thanks to a simulated environment using
virtual reality (VR). Virtual reality indeed allows an
immersive, safe, and controlled study of the
interactions between a pedestrian and an AV
approaching at different speeds, and with distinct
braking behaviours. The second phase of the
experiment will focus on the pedestrian willingness
or unwillingness to cross the street in front of an
automated vehicle, in the absence of zebra crossing,
where priority rules are unclear. More precisely, the
emphasis will be made on the pedestrian’s perceived
safety and decision to cross, or not, the road, when
interacting with an AV. For this test, participants
will be located on a sidewalk, facing a continuous
flow of approaching vehicles. First, a randomised
number of vehicles, separated with short gaps, will
travel without stopping; then, an automated vehicle
will appear on the scene. Different behaviours will
be implemented in the AV (i.e., in terms of
dynamics and ways to stop). The AV occupant,
simulated by an avatar, may have different on-board
activities (e.g., phoning or discussing with another
passenger) or attentive/distracted status, as
illustrated in Figure 4. Participants will be asked to
use a joystick to assess the safety versus
dangerousness of crossing the road from the
pedestrian’s point of view. Depending on the
estimated safety level; the more they will estimate
the situation as safe, the further they will have to
push the joystick forward. On the contrary, the more

the crossing will be assessed as dangerous, the
further they will have to pull the joystick backwards.
Intending to cross a road being an active decision,
keeping the joystick in a neutral position during the
whole scenario will correspond to an intention not to
cross from the beginning. It will thus be possible to
collect the participants’ risk assessment in a dynamic
way throughout the approaching phase of the AV,
and this, whether or not the AV stops.
5. CONCLUSION AND
PERSPECTIVES
In this paper, we presented a new immersive
simulation platform, the V-HCD. This platform was
designed to allow users of today to “anticipate the
future” by virtually plunging into a simulated, yet
realistic, situation of interaction with automated
vehicles. This happens to be particularly useful to
study the acceptance (after a first use of a
technology thanks to virtual reality), and not only
the a priori acceptability (i.e., without any practical
experience), as well as the relevance of virtual
reality to support a cross simulation.
For the SUaaVE (SUpporting acceptance of
automated VEhicles) project, this platform will be
used in two different, notwithstanding
complementary stages. Indeed, a single participant
will be able to experience the situation both from the
pedestrian’s point of view, as well as from the
automated vehicle (AV) occupant’s point of view.
Furthermore, as a result of the efforts performed and
using the experience learnt from this project, it is
also expected to progress towards an interactive
multi-users V-HCD supported by cross-simulation:
where several users can experience a given situation
and interact in real time, some of them taking the
position of pedestrians, and other ones the roles of
AV occupants (i.e., driver or passenger). To
conclude, this new approach of virtual cross-
simulation opens the gate to different types and
multiple synchronised simulations, taking into
account humans as end-users and their different
mobilities.
ACKNOWLEDGEMENTS
This project has received funding from the European
Union’s Horizon 2020 research and innovation
programme under grant agreement No 814999.
REFERENCES
Bellet, T., Bailly-Asuni, B., Mayenobe, P., Banet, A.,
2009. A theoretical and methodological framework for
studying and modelling drivers’ mental
representations. In Safety Science, vol. 47, pp. 1205-
1221.
Bellet, T., Mayenobe, P., Bornard, J-C., Gruyer, D.,
Claverie, B., 2012. A computational model of the car
driver interfaced with a simulation platform for future
Virtual Human Centred Design applications: COSMO-
SIVIC. In Engineering Applications of Artificial
Intelligence, vol. 25, pp. 1488-1504.
Bellet, T., Richard, B., Laverdure, S., Bornard, J-C., 2018.
Use of a cognitive simulation model of the driver to
support the Virtual Human Centred Design (V-HCD)
of ADAS and automated vehicles. Graz Symposium
Virtual Vehicle, Graz, Austria.
Bellet, T., Bornard, J-C., Deniel, J., Richard, B., 2019.
Driver modelling and simulation to support the Virtual
Human Centred Design of future aids. Human Systems
Integration Conference, Biarritz, France.
Distler, V., Lallemand, C., Bellet, T., 2018. Acceptability
and acceptance of autonomous mobility on demand:
The impact of an immersive experience. Conference
on Human Factors in Computing Systems, Montréal,
Canada.
Endsley, M., 1995. Toward a theory of situation awareness
in dynamic systems. In Human Factors, vol. 37, pp.
32-64.
Gruyer, D., Royère, C., Du Lac, N., Michel, G.,
Blosseville J-M., 2006. SiVIC and RTMaps,
interconnected platforms for the conception and the
evaluation of driving assistance systems. World
Congress and Exhibition on Intelligent Transportation
Systems and Services, London, United Kingdom.
Kyriakidis, M., De Winter, J., Stanton, N., Bellet, T., Van
Arem, B., Brookhuis, K., Martens, M., Bengler, K.,
Andersson, J., Merat, N., Reed, N., Flament, M.,
Hagenzieker, M., Happee, R, 2019. A human factors
perspective on automated driving. In Theoretical
Issues in Ergonomics Science, pp. 1-27.
Habibovic, A., Davidsson, J., 2012. Causation
mechanisms in car-to-vulnerable road users crashes:
Implications for active safety systems. In Accident
Analysis and Prevention, vol. 49, pp. 493-500.
Litman, T., 2013-2020. Autonomous vehicle
implementation predictions. Victoria Transport Policy
Institute.
Malmsten Lundgren, V., Habibovic, A., Andersson, J.,
Lagrström, T., Nilsson, M., Sirrka, A., 2016. Will
there be new communication needs when introducing
automated vehicles to the urban context? In Advances
in Human Aspects of Transportation, pp. 485-497.
Rasouli, A., Kotseruba, I., Tsotsos, J., 2017. Agreeing to
cross: How drivers and pedestrians communicate. In
IEEE Intelligent Symposium (IV), Washington, USA.

Schade, J., Schlag, B., 2003. Acceptability of urban
transport pricing strategies. In Transportation
Research Part F: Traffic Psychology and Behaviour,
vol. 5, pp. 45-61.
Schmidt, S., Färber, B., 2009. Pedestrians at the kerb:
Recognising the action intentions of humans. In
Transportation Research Part F: Traffic Psychology
and Behaviour, vol. 12, pp. 300-310.
Schneemann, F., Gohl, I., 2017. Analyzing driver-
pedestrian interaction at crosswalks: A contribution to
autonomous driving in urban environments. In IEEE
Intelligent Vehicles Symposium (IV), Washington,
USA.
Society of Automotive Engineers (SAE), 2018. Taxonomy
and definitions for terms related to driving automation
systems for on-road motor vehicles. J3016-201806.
Sucha, M., Dostal, D., Risser, R., 2017. Pedestrian-driver
communication and decision strategies at marked
crossings. In Accident Analysis and Prevention, vol.
102, pp. 41-50.
Vissers, L., Van der Kint, S., Van Schagen, I.,
Hagenzieker, M., 2017. Safe interaction between
cyclists, pedestrians, and automated vehicles: What do
we know and what do we need to know? Institute for
Road Safety Research, Report R-2016-16.
World Health Organisation (WHO), 2015. Global status
report on road safety.