Content uploaded by Paul Schmitt
Author content
All content in this area was uploaded by Paul Schmitt on Oct 01, 2023
Content may be subject to copyright.
The Road Ahead: Advancing Interactions between Autonomous
Vehicles, Pedestrians, and Other Road Users
Avram Block1Swapna Joshi2Wilbert Tabone3Aryaman Pandya4Seonghee Lee5Vaidehi Patil6
Nicholas Britten7Paul Schmitt8
Abstract— While great strides have been taken in advancing
the field of Human-Robot Interaction (HRI), challenges abound
in understanding and improving how Autonomous Vehicles
(AVs) will interact with and within society. Through this paper,
the authors attempt to paint the picture of challenges unique
to the study and advancement of interfaces between AVs and
vulnerable road users (VRUs). In turn, these gaps in research
highlight the opportunities for academia, industry, and public
policy to collaborate and advance the state of the art of AV-
VRU interaction, and the need for a dedicated forum for sharing
insights across these various sectors.
I. INTRODUCTION
Automated and autonomous vehicles (AVs) are predicted
to be prevalent on highway, city, and suburban streets in
the near future [1]. Indeed, AVs bring much promise of
safe, accessible, and abundant transportation and delivery
of goods to urban and disabled populations [2] [3]. As
AVs become integrated into society, a number of issues
have been postulated and researched, such as building trust
and confidence, communication between humans and AV,
ethical questions about decision-making and responsibility,
and meeting the disparate needs and preferences of users.
Some of these issues might seem to resonate with classic
Human-Robot Interaction problems, by considering the robot
as the autonomous vehicle, and the human as the VRU (a
passenger, pedestrian, cyclist, another human driver, or even
a first responder depending upon the scenario and context).
With respect to the passenger in particular, we agree that
significant preexisting research on HRI for digital agents,
such as virtual voice assistants and avatars, can be applied,
and is incorporated by ongoing User Interface and User
Experience efforts in this space. However, in the current
work, we focus on the less well-trodden subdomain of
interactions that reach beyond the vehicle’s interior. For the
*This work was not supported by any organization
1Avram Block is affiliated with MassRobotics, Boston, USA
aviblock@msn.com
2Swapna Joshi is affiliated with Field Robotics Lab, Northeastern Uni-
versity, Boston, MA swapna@umass.edu
3Wilbert Tabone is affiliated with Delft University of Technology, Delft,
NL w.tabone@tudelft.nl
4Aryaman Pandya is affiliated with Motional, Boston, MA
aryaman.pandya@motional.com
5Seonghee Lee is affiliated with Cornell University, Ithaca, NY
sl994@cornell.edu
6Vaidehi Patil is affiliated with Carnegie Melon University, Pittsburgh,
PA vkpatil@andrew.cmu.edu
7Nicholas Britten is affiliated with Virginia Tech, Blacksburg, VA
brittenn@vt.edu
8Paul Schmitt is affiliated with MassRobotics, Boston, USA
pauls@massrobotics.org
remainder of this paper, we refer to this subdomain as AV-
VRU interaction. The interaction between AVs and VRUs
brings up new and unique challenges as AVs are designed
primarily for transportation and typically interact with users
through displays and controls, as compared to social robots
intended to provide companionship, entertainment, or ed-
ucation. In addition to this, AVs are subject to different
regulations and standards and, as such, issues of safety are
under closer scrutiny here than within the larger field of
social robots. Finally, the diversity of types of road users
and their individual behaviors, attitudes, and expectations, as
well as their varying levels of awareness and understanding
of autonomous vehicle technology adds another layer of
complexity to the interaction between AVs and external road
users.
The focus of this paper is to bring the HRI community’s at-
tention to the unique research, challenges, and advancements
of AV-VRU interaction, and the need for a new, fundamen-
tally interdisciplinary framework for global collaboration.
II. PRI OR AND CURRENT WORK
In this section, the authors provide a wide overview of
AV and VRU interfaces in literature, research programs, and
novel concepts. Prior to and in parallel with this work, many
studies have focused on understanding human-to-human road
communication patterns and models in Europe, and North
America [4] [5] [6].
In extending this type of research to include AVs, some
studies have attempted to evaluate the relative effective-
ness of different AV intent communication modes includ-
ing external Human-Machine Interface (eHMI), exaggerated
sound, and dynamic vehicle motion [7] [8] [9]. For example,
Bengler used interviews, field studies, and assessments of
eHMI designs to suggest how a combination of eHMIs and
vehicle behaviors helped people to know if the vehicle is
autonomous and that it recognized their presence [9]. Schmitt
further showed how expressive behaviors (like gradually
stopping and stopping farther away from pedestrians) can
help decision-making for pedestrians and increase safety,
confidence, and intention understanding [7]. [10] considers
the effectiveness and likeability of auditory cues for signaling
information to pedestrians.
Other studies have delved deeper into the visual media,
studying the pragmatic effectiveness of specific technologies
such as LED light strips and digital screens that produce
patterns on the windshield and elsewhere on the exterior of
the vehicle [11] [12] [13]. Habibovic et al. found that the use
of an external interface significantly increased the likelihood
of a positive experience and improved perceived safety
in pedestrian encounters with AVs[14]. Cumming’s work
comparing the effectiveness of various methods of presenting
vehicle-to-pedestrian street crossing information suggested
that although pedestrians rely on traditional vehicle behav-
iors over information on an external display they believe
additional displays are needed on autonomous vehicles [11].
These include car with eyes [15] [16], crosswalk projecting
headlights [17], a smiling car [18], vehicle mounted visual
displays (Patent US009196164) [19], textual displays [20],
and augmented reality (AR) interfaces [21] [22].
In addition to these largely product development-oriented
efforts, the formal research programs that focus on AV
and VRU interfaces and effectiveness generally come from
Europe and include InterACT [23] and SHAPE-IT [24]
efforts, supported by the EU’s Horizon research and inno-
vation funding program. Interact focuses on a broad range
of projects such as assessing AV intentions, controlling AV
behavior, and establishing evaluation methods for studying
road user interactions with AVs [23]. Shape-It funds aca-
demic research and innovation in collaboration with industry
partners, with quite an emphasis on understanding behaviors
and interactions of VRUs, such as by investigating their
cognitive processes, trust, and acceptance [24].
Although there have been several exciting developments
in this immense effort, the relative scarcity of collaborative,
rigorous research efforts is probably indicative of the novel
technical and financial difficulties of applying HRI perspec-
tives directly to AV-VRU interaction. Many research gaps
and challenges remain, as we attempt to highlight in the
following section.
III. CHA LLE NGES AND CURRENT STATU S
There are many underlying challenges when considering
interfaces between AVs and other road users. While some
challenges are similar to broader HRI research, there are
multiple aspects of the autonomous vehicle that require a
new perspective within the HRI field. These include aspects
such as the heavily constrained physical form of the AV, the
data and research methods available, and the AV’s unique
place in current and future society.
Since AVs have evolved from traditional vehicles, and
are intended to integrate into human society alongside these
vehicles, they are often impacted by widely held notions
based on human-driven vehicles. There is a special expec-
tation placed on AVs that they demonstrate their technical
advancements and simultaneously support human behaviors
and expectations. Physically, this means that their appearance
should not deviate too far from current vehicular norms.
Similarly, the designs of AVs are beholden to many of the
legal regulations which were developed for human-driven
vehicles. These considerations are largely distinct from the
challenges faced by those researching interaction with a
humanoid robot, which involves far fewer legal constraints,
and fewer pre-conceived expectations about their design and
behavior.
Current challenges in the space can be broadly categorized
into three major groups. The first group of challenges (ex-
panded in subsection A below) pertains to the difficulty of
conducting rigorous research in this area, with widespread
peer and expert support. This difficulty stems from a lack of
mature data and consolidated insights, which characterizes
the nascent field of AV-VRU interaction. Given the com-
plexity of the scenarios involved, this calls for a need for
collaborative insight into the availability of existing data and
the type of data desired from new research. The second set
of challenges (subsection B below) arises from the breadth
of the application domain (the many environments in which
an AV may be deployed) and the diversity of the many
agents within. Solutions that cater to one segment of the
population are not guaranteed to work for others, and yet
solutions that are robust to the diversity of humans that
the vehicle may need to interact with are necessary for the
successful worldwide integration of AVs. The third group
of challenges (subsection C) pertains to the boundlessness
of the design/solution space itself. Designers and engineers
building HRI-focused autonomous vehicles face a vast, and
largely unexplored landscape of novel approaches to this
emerging design gap. Nuanced design decisions must be
made when crafting solutions, such as determining the appro-
priate amount of information to be conveyed to vulnerable
road users (VRUs) and the modality of the interface used
to convey this information. The conciseness, specificity, and
legibility of these techniques also carry significant ethical
weight, due to the potential consequences of unnecessary
distraction or miscommunication.
A. Research Challenges
1) Testing and Data Collection: Unlike HRI research,
which is most often conducted in lab settings or field sites
of a controlled nature, to understand how an AV performs
with road users in the complex, dynamic environment of the
streets, ideal testing should involve or closely approximate
real-world conditions [25]. Significant safety concerns must
be taken into account with respect to other vehicles, and
external agents. Additionally, testing can be difficult and
expensive, and may not be feasible in certain locations. In
order to capture the breadth of environmental possibilities,
collecting and analyzing extremely large amounts of data is
essential. To limited success, some groups have attempted to
mitigate this concern by using a combination of real-world
data and data gained through simulated methods, such as
the Robot Operating System (ROS) [26] and virtual reality
[27] [28] [29]. However, it can be difficult to obtain data
that is representative of the distribution of complex, dynamic
environments in which AV-VRU interaction takes place.
This challenge is particularly relevant for those working in
industry, as it makes it much more difficult for commercial
user researchers to produce findings that satisfy scientific
standards of validity, reproducibility, and generalizability.
2) Naturalistic Dataset Availability: Given the difficulties
described above in producing sufficient experimental data,
an emphasis must also be placed on the availability and use
of naturalistic, observational data from real environments.
For instance, while some research indicates the benefits
of AV driving behavior that mimics or exaggerates that
of human drivers, only recently have researchers begun to
examine, clarify, and quantify these human behaviors. And
while such studies would naturally benefit from naturalistic
data sets, very few exist. Those that do exist [30] [31]
[32] [33] [34] were developed mainly by AV companies for
advancing state-of-the-art perception and motion planning
approaches, and so are poorly annotated for finding relevant
sets of various human to human engagement scenarios. These
datasets were also developed to address specific company
needs, and therefore have lower widespread utility to other
organizations. We propose that this is a challenge well-
suited for an academia-based response. Academic research,
such as in the fields of ethnography, urban planning, and
science and technology studies, regularly applies techniques
for conducting scrupulous naturalistic observation, as well as
more longitudinal studies. In addition, an academic focus on
accurately describing trends, rather than producing actionable
recommendations, makes such datasets likely to be more
widely useful across various facets of the AV industry.
3) Cross-Discipline Insights: As with most HRI use
cases, findings from multiple disciplines are needed to char-
acterize the gaps, propose solutions, and assess efficacy.
These may include elements of robotics, computer science,
engineering, psychology, sociology, transportation planning,
and design. However, AV-VRU interaction especially stands
to benefit from the inclusion of perspectives from ethno-
graphic and sociological studies such as [35], human factors
[36], user experience, and public policy.
While engineers and computer scientists work on develop-
ing the technology and algorithms that allow AVs to perceive
and understand their environment, cognitive psychologists,
human factors experts, and user researchers study how people
interact with AVs, and how the design of AVs and their
communication systems can influence the behavior of pedes-
trians, drivers, and other road users. Further, sociologists and
transportation planners could help study the broader social
and economic impacts of AVs, and policy makers work on
creating regulations and guidelines that will ensure the safe
and efficient deployment of AVs on public roads. The authors
believe that the study of AV-VRU interaction lies right at
the intersection point of each of these seemingly disparate
disciplines, and calls for academia, industry and policy to
come together to share insights and support each others’
future work.
B. Environmental Diversity
1) Acceptance and Cultural Differences: Another signifi-
cant challenge facing AV-VRU interaction researchers is the
difference in cultural norms and driving practices that exist
between many areas of the world [37]. For example, road
user acceptance of vehicles with an empty or nonexistent
driver seat is unclear. To pedestrians and other road users,
how significant a factor is the appearance of a driver behind
the driver’s seat? Indeed, acceptance at local and regional
governance levels is an open question as well [38] [39] [40].
A potential solution in one culture may not work well in
another. Socio-cultural differences can affect the acceptance
of AVs and result in varying levels of technology trust, which
can affect perceptions of risk associated with sharing roads
with them [41] [42]. Some external users may be more
concerned about the data collected by autonomous vehicles
as being used for surveillance or targeted advertising [43].
In some cultures, the autonomous vehicle is seen as a
new form of transportation that must obey traffic rules
where pedestrians have the right of way [44]. As such,
autonomous vehicles would need to be programmed to be
cautious when interacting with pedestrians, and may stop or
slow down more frequently to allow pedestrians to cross the
street or walk through intersections. However, in a culture
where an AV is seen as a tool to improve traffic flow and
reduce congestion, AVs would have to prioritize being more
assertive and less cautious to preserve the flow of traffic [45].
For academics, this challenge centers on providing localized
expertise and reliable characterizations of specific cultural
contexts. Those working in industry, on the other hand, must
assume the responsibility of internalizing and synthesizing a
wide variety of such contributions and incorporating these
findings into their design solutions.
2) Special Needs Populations: Even if an interface with
one modality is found to be effective, the ideal solution
should be designed for accessibility, to accommodate di-
verse populations such as those with visual and auditory
impairments [46]. While such interfaces have been developed
successfully for laptop and mobile device use cases (and are
being studied for AV interiors), there are few studies [47],
[48], [49] involving AV-VRU interaction at, for example,
noisy, busy intersections [50]. Similarly to the previous
subsection, the focal point of this challenge is for industry
to seek out and absorb information from various subfields
of disability studies, and to reflect the recommendations of
these subfields in their product designs and design processes.
3) Complex Interaction Scenarios: Much of the current
HRI literature is focused on one-to-one interaction between
a robot and an individual or a small group of users. Apply-
ing this perspective would mean modeling and researching
relations of one AV to one external road user [7] [13].
However, it is unclear if and how much pedestrian behavior
may change when exposed to more realistic urban scenarios
involving multiple vehicles and or VRUs simultaneously.
In addition to pedestrians, and cyclists, the population of
VRUs includes wheelchair and mobility scooter users, people
with visual and auditory impairments, first responders [51],
small animals, traffic controllers, people with strollers, and
many more. Additionally, every interaction scenario with
an autonomous vehicle will involve interaction between the
vehicle, its occupants, VRUs, any teleoperators involved, and
other users of similar vehicles. This presents AV-VRU re-
searchers with a complex, multifaceted universe of scenarios
that involve ’robot + user within the robot + user outside the
robot + virtual robot operators + other robots’ interaction.
C. Design Considerations
1) Unique Form Factor: A key distinction in the physical
design of HRI-competent AVs is that the robot lacks a
torso and limbs. Rather, it must build upon the road vehicle
platform, which offers very few degrees of freedom, beyond
the turning of its wheels or perhaps its side view mirrors.
Thus, much of the existing work on movement [52], legibility
[53], exaggeration [54], biomimicry [55], etc., must be re-
interpreted or re-imagined for a different form factor. This
new field of study is filled with its own impressive feats and
daunting challenges[56] [57], though there are extremely few
consensuses or established frameworks. In order to account
for this gap, many ongoing research projects in the area of
AV-VRU interaction include varying combinations of com-
municative motion behaviors or explicit visual displays. At
present, the efficacy of an eHMI is unclear, and some parties
are wary of the potential for eHMIs to cause unwarranted dis-
traction. Similarly, early inquiries have been conducted into
the use of expressive sound as a communication channel for
AVs [7], though this specific subfield is in its relative infancy.
Perhaps a naturalistic driving behavior will be sufficient, but
a framework for naturalistic autonomous driving (e.g., defi-
nitions, data, and metrics) is still far from being established
[58] [59] [9]. Widespread agreement and consistency on the
question of modalities will require a collaboration between
academia and industry, in which academics are provided the
resources to conduct research that is not as tightly bound to
the financial and commercial implications of recommending
one approach over another.
2) Information to Communicate: In addition to accep-
tance of the mere presence of an AV, there is a question
of what specific information to communicate with external
observers. What information is necessary, and what is super-
fluous, distracting, or dangerous?
Exploration into this question includes research on dis-
playing whether the vehicle is in Autonomous Mode or not
[60], whether it is stopped/parked [51], about to move [14],
about to stop [7], about to change lanes [61], or whether it
is in some particular failure mode [51].
However, this particular question requires consideration
of the potential consequences of revealing various pieces
of information about the AV’s state. For instance, is the
disclosure that the AV is in a failure mode likely to result
in tampering? Is the disclosure that the AV is holding
passengers more likely to result in their potential harassment
[62]? Alternatively, researchers must also explore the impact
of omitting these pieces of information, and whether this
causes untenable levels of ambiguity in instances of AV-VRU
interaction.
3) Ethics: Researching interfaces for AV-VRU interac-
tion raises ethical concerns about how autonomous vehicles
should be programmed to make decisions in situations where
there is a risk of harm to pedestrians. Ensuring that the
vehicles’ interfaces are intuitive is not only a matter of
aesthetics and preference. It also carries safety implications,
as an intended message from the AV may be incorrectly
interpreted by its human receiver or vice versa, and lead to
mutually unexpected behaviors.
Additionally, the prospect of AVs on public roads presents
an ethical challenge around data privacy and transparency
[63]. It is correctly assumed by many members of the
public that AVs collect immense amounts of (potentially
identifiable) data about their surroundings [64]. Thus, the
job of an AV-VRU interaction researcher should also include
gaining the trust of the public by embedding some level
of data transparency in interface designs, and by striving
to minimize the privacy encroachment that may result from
these interfaces. Due to concerns around safety, liability, and
responsibility to users and customers, we believe that this
challenge should be of the utmost consideration for industry,
the group that will face the vast majority of repercussions
for missteps in this domain.
IV. REC ENT DE VEL OPM ENT S
Having addressed the multitude of challenges facing this
new area of research, and the current status of work being
conducted in this area, the authors also consider constructive
next steps for this emerging field. Many disparate industries
and research efforts are gaining valuable insights into specific
subsets of this challenge, and it is the intention of the authors
to encourage interdisciplinary collaboration and discussion
of these insights. We envision a near future in which this
important topic brings together experts from many different
domains, both academic and industrial, from technical to
sociological backgrounds. We also recognize some current
trends in this direction, which are already demonstrating
the immense value of a wide-net approach to this work.
This section highlights the main discussion forums that are
“moving the conversation forward” in either highlighting
AV and external road user interface research or discussing
standards or regulations.
1) Conferences: The following is a non-exhaustive
overview of the current conferences that cover aspects of
AV-VRU interaction research. Transportation industry con-
ferences include Transportation Research Board (TRB)
and its spinoff conference, Automated Road Transport
Symposium (ARTS). These conferences feature workshops
such as “Perspectives on Automated Driving Systems Com-
munications to Existing Road Users”, though human-driven
systems remain their primary focus.
Robotics industry conferences include the more
technologically-oriented ICRA,RO-MAN, and IROS
conferences (which often hold workshops on relevant
topics such as intent and gesture recognition), as well as
Human-Robot Interaction (HRI), a conference with a
more human-facing and social-theory based focus. AutoUI,
a conference dedicated specifically to vehicle interface
technology and design, occupies the middle ground between
these robotics and transportation categories.
In addition, a growing number of social science con-
ferences and journals, such as Ethnographic Praxis in
Industry Conference (EPIC) and Current Sociology are
beginning to publish research on the human impact of the
widespread deployment of autonomous vehicles.
2) Public Policy: There are a couple of known AV ex-
ternal interface public policy initiatives. The United Nations
Economic Commission for Europe (UNECE) has a Working
Party for Autonomous and Connected Vehicles (GRAV) that
is actively seeking input and recommendations for external
Human Machine Interfaces for autonomous vehicles [65]
[66].
Additionally, the Singapore Land Transport Authority re-
quires AVs to exhibit lighting for identification of Auto Mode
status [60].
3) Standards: A few teams are actively developing stan-
dards for AV communication with external road users. The
SAE Automated Driving System Lamps Task Force has
released J3134-201905, Automated Driving System Marker
Lamp Recommended Practice [67], and continues to review
opportunities for improvements. The ISO Transport Infor-
mation and Control Systems Working Group (also known
as “ISO/TC 22 SC39 WG 8”) has developed the standard
ISO/TR 23049:2018, “Road Vehicles – Ergonomic aspects
of external visual communication from automated vehicles
to other road users”, and continues to review opportunities
for the next version [68]. Project ISO 4448 Ground Based
Automated Mobility is developing a standard for sidewalk
delivery and service robots [29]. European organizations such
as the European Telecommunications Standards Institute
have also put forth standards such as “ETSI TR 102 638
on Intelligent Transport Systems’ [69].
4) Expert Discussion Forums: One forum dedicated for
thought leaders specifically to review, discuss, and propose
updates and advancements to external AV interfaces is the
MassRobotics Socially Aware Automated Mobility (SAAM)
consortium [70]. The SAAM meets quarterly. No others are
known at this time.
5) Research Programs: Supporting the Interaction of Hu-
mans and Automated Vehicles: Preparing for the Environ-
ment of Tomorrow (SHAPE-IT) is a European Union funded
research project aimed at ”safe, acceptable, and desirable
integration of user-centered and transparent automated vehi-
cles into urban traffic environment”[24]. It funds fifteen PhD
research projects and invites industry supervision.
Considering the various forums where AV and external
road user interfaces are addressed, there does not appear to be
a centralized forum. Rather there are many forums involving
a subset of experts needed to address the challenges. Each
forum looks at the challenge through the resulting facets.
Almost all are infrequent, occurring annually.
V. DISCUSSION
A. Overview
AV-pedestrian interactions have taken HRI from the lab
and field sites to more dynamic, ever-changing open envi-
ronments, such as streets, vehicular roads, sidewalks, and
public spaces.
While the ‘social’ robot is obviously designed for interac-
tions that are beneficial to its users, AV-pedestrian interac-
tions have to overcome the historical conflict of pedestrians
and vehicles and intentionally design interactions to convey
that the autonomous vehicle does not harm pedestrians,
cyclists, or other road users. In addition, HRI often involves
interactions between humans and robots that carry out or
emulate human behaviors, such as teleoperated robots or
robots that are programmed to respond to human gestures
or commands. However, interactions between autonomous
vehicles and pedestrians, so far, are perceived as a vehicle
making decisions on its own based on sensor data and
pre-programmed rules. HRI also typically involves direct
physical or visual interactions between the human and
the robot, while AV and pedestrian interaction is based
on the perception and understanding of the environment,
the vehicle’s prediction of the pedestrian’s behavior, and
importantly the pedestrian’s understanding of the vehicle’s
intentions. Additionally, this research domain goes beyond
interactions with a single human, or small groups of humans,
that are often trained or familiar with the robot’s behavior,
and involves interactions with large and diverse groups of
pedestrians with different ages, cultures, and abilities, who
may have different expectations of the autonomous vehicle’s
behavior.
In summary, researching AV-VRU interaction is a chal-
lenging task that requires a multidisciplinary approach to
address the complexity of the system, safety concerns, ethical
implications, cultural differences, data collection, human be-
havior, and regulatory compliance. Some of these challenges
may be more effectively tackled by industry, academia,
or policy makers individually, while others will require a
collaborative effort among all stakeholders, as presented
below.
B. Opportunities
In reflecting upon the challenges and gaps above, several
opportunities are proposed for academic research, industry,
and public policy collaborations.
1) Academic Community: The authors envision several
opportunities for the academic community. These oppor-
tunities have been designated as such because we believe
that they benefit from a more unbiased perspective. We also
suggest that the open-ended nature of the current phase of
research questions is more well-suited to academic inquiry.
•eHMI - Auditory Signals A significant research op-
portunity exists within the potential for sound to enrich
AV-VRU interaction. While this is a developing area
within the broader HRI field [71] [72], application to
AV external interfaces is in earlier research stages.
Opportunities exist to adapt HRI sound taxonomies for
AV use cases (e.g., “about to accelerate”, “courtesy”,
etc.) and develop an open source library of promising
sounds.
•eHMI - Visual Displays While a larger body of AV
visual display research exists, the literature indicates
that the best solutions to-date are not intuitive but
rather learned. The authors propose a potential for cross-
pollination with the broader HRI body of knowledge.
•Expressive Behaviors Current research indicates that
vehicle dynamics are perhaps the most promising intu-
itive medium for AV intention communication [8] [7].
The authors propose that research opportunities exist
to understand the key vehicle motion parameters and
connect them with AV use case taxonomy. Additionally,
the authors propose opportunities for reinterpretation of
existing HRI robot motion literature for the AV form
factor.
•Multi-modal Techniques Another significant research
gap exists in identifying an optimal mix of the above
modalities for VRU use cases, especially for the blind
and deaf community.
•Standards Key research questions in front of the stan-
dards community are highlighted above within “Reg-
ulations and Standards”. The authors propose studies
that leverage data from diverse community groups to
highlight areas of confusion and quantify potential
benefits of producing standardized, uniform behaviors
across the AV industry. Similarly, increased access to
the sorts of datasets that would produce these standards
can also lead to the adoption of widely agreed-upon
benchmarks of performance in VRU-interaction tasks
such as intersection and traffic navigation and intent
signalling.
2) Industry: The authors also envision several opportu-
nities for the commercial AV industry. These opportunities
have been designated as such because the authors believe that
they benefit from resources that are typically more prevalent
or available in industry, such as high volumes of data, and
the financial backing to realize and ”productize” promising
research findings. As described within “Challenges and Cur-
rent Status”, many existing datasets would benefit from a
VRU interaction lens, but as of today most are best suited
for perception and motion planning studies. The authors
recommend efforts within industry to open source datasets
that enable this type of research. Such datasets would include
a wide variety of indexed external agents. Complex scenarios
involving these agents would be consistently labeled, and
easily searchable.
The authors note that providing large, open source datasets
requires significant resources and as such is not an exercise
taken lightly. As a first step, we recommend the research
community leverage existing open source datasets tailored
for perception system research [30] [31]. Although not ideal
for VRU studies, brute-force search approaches should yield
enough scenarios of interest to demonstrate the benefit of this
approach and hopefully inspire further open source efforts.
In addition to this specific opportunity, the authors sug-
gest more generally that a significant opportunity exists
for industry to seek out and ingest the extremely useful,
well-founded research being produced by specific academic
disciplines with which they may not be traditionally familiar.
While we also suggest the creation of additional forums
for effective knowledge transfer, we emphasize that many
smaller repositories of such knowledge are already being
developed. These represent a potent opportunity for designers
to build on verified hypotheses regarding social expectations
and acceptance of autonomous vehicles.
3) Academia and Industry: While this and prior work
envision a number of AV-VRU interaction use cases, the
authors propose the need for a use case classification tax-
onomy. Such a taxonomy would help clarify and structure
studies developing interface solutions. This work should
align with, but not mirror, the proposed eHMI classification
taxonomy [59]. To clarify, the eHMI taxonomy helps classify
solutions, while the authors’ proposed use case taxonomy
would classify the need and interaction scenarios.
4) Public Policy: The authors envision several opportuni-
ties for public policy makers.
•Engage The authors encourage public policy makers
to engage with community groups [73] on the issue of
AV benefits and challenges for communities. Recognize
the benefit of AVs that integrate well within society.
Identify key areas in society that would benefit from
clear understanding of AV interactions.
•Promote Research The authors encourage public policy
makers to initiate and coordinate research programs
targeting the challenges outlined in this work. While
the EU has strong examples of this [23] [24], other
regions are not as explicit about their investment in this
area. As highlighted above, solutions for one culture or
geographic area may not extrapolate well.
•Advance Policy The authors encourage public policy
makers to take a measured approach to new policy de-
velopment. Leverage new research for policy proposals.
Seek and incorporate input from community groups, but
also cademia and industry.
5) Cross-sector Collaboration: Considering the different
stakeholders involved and the lack of community engagement
in AV-pedestrian interaction research and design, there is
a need for a dedicated consortium, track or a venue for
bringing interdisciplinary experts together for collaborations,
research updates, panel discussions, and idea exchanges.
Ideally, such a consortium would engage frequently given
the forecasted rate of AV adoption and potential to benefit
society. A key challenge and opportunity is to bring the
needed cross-discipline perspectives together in one venue.
AV-VRU interaction research could benefit greatly from
sites of collaboration, active dialogue, and problem-solving
around broader interaction specific to social, cultural, and
physical contexts of specific communities while informing
the broader understanding of interactions between AV and
pedestrians.
While both industry and academia are interested in con-
ducting AV-VRU interaction research and development ac-
tivities to advance the state of the art in autonomous ve-
hicle technology, they may have different perspectives and
priorities. For example, the goal of industry-led research
with its practical and market-driven approach and limited
timeframes may be to develop a working prototype and
prioritize aspects of safety and reliability. On the other
hand, academic research may be more interested in sci-
entific rigor, long-term impact and answering fundamental
theoretical questions. However, their overlapping interests
in evaluation and testing involve learning about stakeholder
perspectives, running simulations, test tracks, and real-world
prototype deployment on public roads. Additionally, both
can and are significant for contributing to the development
of standards and regulations for AV technology and require
working with government agencies and other stakeholders to
develop guidelines and best practices for the safe and ethical
deployment of autonomous vehicles.
VI. CONCLUSION
The interaction between AVs and VRUs is one that will
benefit from more collaborative research from a diverse set
of disciplines. As we are awe-inspired by the technologies
that allow us to continue to roll out driverless vehicles in our
streets, we are also reminded of their current lack of social
awareness. In order for AVs to realize their true potential
and support safety and comfort in personal mobility, it is
essential for them to provide communities with a sense of
trust and ease.
The AV-external user challenges are similar to and yet
distinct from traditional HRI challenges or the broader AV
user challenges. We call for attention from the HRI commu-
nity towards the need for a focused track or consortium to
assess different approaches for design and evaluation within
this problem space. Further, we emphasize the importance
of accounting for different interaction modalities, human
behaviors, and contexts (cultural and situational) of the in-
teractions. Work advanced by industry experts, public policy
officials, research programs and conferences will be crucial
to the path forward in this field.
REFERENCES
[1] W. Tabone, J. de Winter, C. Ackermann, J. B¨
argman, M. Baumann,
S. Deb, C. Emmenegger, A. Habibovic, M. Hagenzieker, P. Hancock,
R. Happee, J. Krems, J. D. Lee, M. Martens, N. Merat, D. Norman,
T. B. Sheridan, and N. A. Stanton, “Vulnerable road users and the com-
ing wave of automated vehicles: Expert perspectives,” Transportation
Research Interdisciplinary Perspectives, 2021.
[2] J. M. Anderson, N. Kalra, K. D. Stanley, P. Sorensen, C. Samaras,
and O. A. Oluwatola, The Promise and Perils of Autonomous Vehicle
Technology. RAND Corporation, 2014, pp. 9–40.
[3] H. Claypool, A. Bin-Nun, and J. Gerlach. (2017) Self-driving cars: The
impact on people with diabilities. https://rudermanfoundation.org/wp-
content/uploads/2017/08/Self-Driving-Cars-The-Impact-on-People-
with-Disabilities FINAL.pdf.
[4] D. Rothenbucher, J. Li, D. Sirkin, B. Mok, and W. Ju, “Ghost driver:
A field study investigating the interaction between pedestrians and
driverless vehicles,” in 2016 25th IEEE Intrl. Symp. on Robot and
Human Interactive Communication (RO-MAN).
[5] N. Mizuno, A. Yoshizawa, A. Hayashi, and T. Kawashima, “Estimation
of drivers ‘ awareness of pedestrians,” in 2018 IEEE 17th Interna-
tional Conference on Cognitive Informatics Cognitive Computing
(ICCI*CC), 2018, pp. 381–386.
[6] G. J. S. Wilde, “Social interaction patterns in driver behavior: An
introductory review,” Human Factors: The Journal of Human Factors
and Ergonomics Society, vol. 18, pp. 477 – 492, 1976.
[7] P. Schmitt, N. Britten, J. Jeong, A. C. K. Clark, S. S. Kothawade,
E. C. Grigore, A. Khaw, C. Konopka, L. Pham, K. Ryan, C. Schmitt,
and E. Frazzoli, “Can Cars Gesture? A Case for Expressive Behavior
Within Autonomous Vehicle and Pedestrian Interactions,” in IEEE
Robotics And Automation Letters, VOL. 7, NO. 2, APRIL 2022.
[8] J. W. Jenness, A. K. Benedick, J. Singer, S. Yahoodik, E. Petraglia,
J. Jaffe, J. M. Sullivan, and A. K. Pradhan, “Automated
Driving Systems’ Communication of Intent With Shared Road
Users,” National Highway Transportation Safety Administration,
Technical Report DOT HS 813 148, 2021. [Online]. Available:
https://rosap.ntl.bts.gov/view/dot/58325
[9] Y. E. Song, C. Lehsing, T. Fuest, and K. Bengler, “eHMIs and their
effect on the interaction between pedestrians and automated vehicles,”
in Intelligent Human Systems Integration, 2018.
[10] K. Mahadevan, S. Somanath, and E. Sharlin, “Communicating aware-
ness and intent in autonomous vehicle-pedestrian interaction,” in Pro-
ceedings of the 2018 CHI Conference on Human Factors in Computing
Systems, ser. CHI ’18, 2018, p. 1–12.
[11] M. Clamann, M. Aubert, and M. Cummings, “Evaluation of vehicle-to-
pedestrian communication displays for autonomous vehicles.” Trans-
portation Research Board 96th Annual Meeting, 2017.
[12] M. Kaup, J.-H. Willrodt, A. Schieben, and M. Wilbrink, “Final design
and HMI solutions for the interaction of AVs with user on-board and
other traffic participants ready for final implementation,” EU’s Horizon
2020 Program, Grant Agreement 723395,” Technical Report, 2019.
[13] A. Block, A. Pandya, S. H. Lee, and P. Schmitt, “I see you! design
factors for supporting pedestrian-av interaction at crosswalks,” in HRI
2023: Proceedings of the 2023 ACM/IEEE International Conference
on Human-Robot Interaction. IEEE Press, 2023.
[14] A. Habibovic, V. Lundgren, J. Andersson, M. Klingeg ˚
ard, T. Lagstr¨
om,
A. Sirkka, J. Fagerl¨
onn, C. Edgren, R. Fredriksson, S. Krupenia,
D. Salu¨
a¨
ar, and P. Larsson, “Communicating intent of automated
vehicles to pedestrians,” Frontiers in Psychology, 2018.
[15] JaguarLandRover. (2018) Jaguar land rover’s virtual eyes look at trust
in self-friving cars. https://media.jaguarlandrover.com/news/2018/
08/jaguar-land-rovers-virtual-eyes-look-trust-self-driving-cars.
[16] C.-M. Chang, K. Toda, X. Gui, S. H. Seo, and T. Igarashi, “Can eyes
on a car reduce traffic accidents?” in Proceedings of the 14th Inter-
national Conference on Automotive User Interfaces and Interactive
Vehicular Applications, 2022.
[17] Mercedes-Benz. (2016) Revolution in headlamp technology: Mer-
cedes shines in hd quality: Digital light. https://group-media.mercedes-
benz.com/marsMediaSite/ko/en/14872032.
[18] Semcon. (2016) Self driving car that sees you.
https://semcon.com/smilingcar/.
[19] AUVSI. (2019) Drive.ai uses external communication panels to talk to
the public. https://www.auvsi.org/industry-news/driveai-uses-external-
communication-panels-talk-public.
[20] Nissan. (2015) Nissan ids concept: Nissan’s vision for the future of
autonomous driving. https://global.nissannews.com/en/releases/
release-3fa9beacb4b8c4dcd864768b4800bd67-151028-01-e.
[21] A. F. Dalipi, D. Liu, X. Guo, Y. Chen, and C. Mousas, “Vr-pavib: The
virtual reality pedestrian-autonomous vehicle interaction benchmark,”
in 12th International Conference on Automotive User Interfaces and
Interactive Vehicular Applications, 2020, pp. 38–41.
[22] W. Tabone, Y. M. Lee, N. Merat, R. Happee, and J. de Winter, “To-
wards future pedestrian-vehicle interactions: Introducing theoretically-
supported AR prototypes,” in 13th International Conference on Auto-
motive User Interfaces and Interactive Vehicular Applications, 2021.
[23] P. Kanellopoulos, “Proceedings of the interACT Final Event,”
European Union’s Horizon 2020 Research and Innovation Programme,
Grant Agreement No 723395,” Technical Report. [Online]. Available:
https://www.interact-roadautomation.eu
[24] J. B¨
argman. (2019) Shape-it: Supporting the interaction of humans
and automated vehicles: Preparing for the environment of tomorrow.
https://www.shape-it.eu/about/.
[25] W. Tabone, R. Happee, J. Garcia de Pedro, Y. M. Lee, M. Lu-
petti, N. Merat, and J. de Winter, “Augmented reality interfaces for
pedestrian-vehicle interactions: An online study,” 11 2022.
[26] M. Pr´
edhumeau, L. Mancheva, J. Dugdale, and A. Spalanzani, “Agent-
based modeling for predicting pedestrian trajectories around an au-
tonomous vehicle,” J. Artif. Int. Res., 2022.
[27] N. Merat, Y. M. Lee, G. Markkula, J. Uttley, F. Camara, C. Fox,
A. Dietrich, F. Weber, and A. Schieben, “How do we study pedestrian
interaction with automated vehicles? preliminary findings from the eu-
ropean interact project,” in Automated Vehicles Symposium. Springer,
2019, pp. 21–33.
[28] S. Schneider and K. Bengler, “Virtually the same? analysing pedestrian
behaviour by means of virtual reality,” Transportation Research Part
F: Traffic Psychology and Behaviour, 2020.
[29] B. Grush, “Making room for robots: A draft iso technical standard
for ground-based automated mobility: Loading and unloading at the
kerbside and footway,” 08 2021.
[30] H. Caesar, V. Bankiti, A. H. Lang, S. Vora, V. E. Liong, Q. Xu,
A. Krishnan, Y. Pan, G. Baldan, and O. Beijbom, “nuScenes: A
multimodal dataset for autonomous driving,” in CVPR, 2020.
[31] P. Sun, H. Kretzschmar, X. Dotiwalla, A. Chouard, V. Patnaik, P. Tsui,
J. Guo, Y. Zhou, Y. Chai, B. Caine, V. Vasudevan, W. Han, J. Ngiam,
H. Zhao, A. Timofeev, S. Ettinger, M. Krivokon, A. Gao, A. Joshi,
Y. Zhang, J. Shlens, Z. Chen, and D. Anguelov, “Scalability in
perception for autonomous driving: Waymo open dataset,” in Proceed-
ings of the IEEE/CVF Conference on Computer Vision and Pattern
Recognition (CVPR), June 2020.
[32] B. Wilson, W. Qi, T. Agarwal, J. Lambert, J. Singh, S. Khandelwal,
B. Pan, R. Kumar, A. Hartnett, J. Kaesemodel Pontes, D. Ramanan,
P. Carr, and J. Hays, “Argoverse 2: Next generation datasets for self-
driving perception and forecasting,” in Proceedings of NeurIPS Track
on Datasets and Benchmarks, 2021.
[33] H. Caesar et al., “nuPlan: A closed-loop ML-based planning bench-
mark for autonomous vehicles,” CoRR, vol. abs/2106.11810, 2021.
[34] Y. Liao, J. Xie, and A. Geiger, “KITTI-360: A novel dataset and
benchmarks for urban scene understanding in 2d and 3d,” arXiv
preprint arXiv:2109.13410, 2021.
[35] E. Vinkhuyzen and M. Cefkin, “Developing socially acceptable au-
tonomous vehicles,” Ethnographic Praxis in Industry Conference
Proceedings, vol. 2016, no. 1, 2016.
[36] W. A. Schaudt, S. Russell, and J. M. Owens, “The role of human
factors in the design of automated vehicle external communication,”
in Road Vehicle Automation 6, G. Meyer and S. Beiker, Eds., 2019.
[37] T. ¨
Ozkan, T. Lajunen, J. E. Chliaoutakis, D. Parker, and H. Summala,
“Cross-cultural differences in driving behaviours: A comparison of
six countries,” Transportation research part F: traffic psychology and
behaviour, vol. 9, no. 3, pp. 227–242, 2006.
[38] B. Wessling. (2021) Toronto city council votes to ban sidewalk
robots. https://www.therobotreport.com/toronto-city-council-votes-to-
ban-sidewalk-robots/.
[39] P. Cornwell. (2022) “Amazon robots scooting along sidewalks”.
https://www.seattletimes.com/seattle-news/eastside/for-now-
kirkland-says-no-to-small-amazon-robots-scooting-along-streets-
and-sidewalks/.
[40] “Regulation EU 2019/2144 of the european parliament and of the
council of 27 november 2019 on type-approval requirements for motor
vehicles and their trailers,” Official Journal of the EU L325, Dec 2019.
[41] A. Rasouli and J. K. Tsotsos, “Autonomous vehicles that interact with
pedestrians: A survey of theory and practice,” IEEE transactions on
intelligent transportation systems, vol. 21, no. 3, pp. 900–918, 2019.
[42] L. M. Hulse, H. Xie, and E. R. Galea, “Perceptions of autonomous
vehicles: Relationships with road users, risk, gender and age,” Safety
science, 2018.
[43] N. Liu, A. Nikitas, and S. Parkinson, “Exploring expert perceptions
about the cyber security and privacy of connected and autonomous
vehicles: A thematic analysis approach,” Transportation research part
F: traffic psychology and behaviour, 2020.
[44] S. Gupta, M. Vasardani, and S. Winter, “Negotiation between vehicles
and pedestrians for the right of way at intersections,” IEEE Transac-
tions on ITS, no. 3, 2018.
[45] E. R. Straub and K. E. Schaefer, “It takes two to tango: Automated
vehicles and human beings do the dance of driving–four social
considerations for policy,” Transportation research part A: policy and
practice, 2019.
[46] E. Kassens-Noor, M. Cai, Z. Kotval-Karamchandani, and T. Decam-
inada, “Autonomous vehicles and mobility for people with special
needs,” Transportation research part A: policy and practice, 2021.
[47] J. Brinkley, E. W. Huff Jr, B. Posadas, J. Woodward, S. B. Daily,
and J. E. Gilbert, “Exploring the needs, preferences, and concerns
of persons with visual impairments regarding autonomous vehicles,”
ACM Transactions on Accessible Computing, 2020.
[48] D. Hong, S. Kimmel, R. Boehling, N. Camoriano, W. Cardwell,
G. Jannaman, A. Purcell, D. Ross, and E. Russel, “Development of a
semi-autonomous vehicle operable by the visually-impaired,” in 2008
IEEE International Conference on Multisensor Fusion and Integration
for Intelligent Systems. IEEE, 2008, pp. 539–544.
[49] P. D. Fink, J. A. Holz, and N. A. Giudice, “Fully autonomous vehicles
for people with visual impairment: Policy, accessibility, and future
directions,” ACM Transactions on Accessible Computing, 2021.
[50] A. Z. Asha, C. Smith, G. Freeman, S. Crump, S. Somanath,
L. Oehlberg, and E. Sharlin, “Co-designing interactions between
pedestrians in wheelchairs and autonomous vehicles,” in Designing
Interactive Systems Conference 2021, 2021, pp. 339–351.
[51] S. H. Lee, V. Patil, N. Britten, A. Block, A. Pandya, M. Jung,
and P. Schmitt, “Safe to approach: Insights on autonomous vehicle
interaction protocols with first responders,” in HRI 2023: Proceedings
of the 2023 ACM/IEEE International Conference on Human-Robot
Interaction. IEEE Press, 2023.
[52] G. Hoffman and W. Ju, “Designing robots with movement in mind,”
Journal of Human-Robot Interaction, vol. 3, no. 1, p. 89, Mar. 2014.
[53] A. D. Dragan, K. C. Lee, and S. S. Srinivasa, “Legibility and
predictability of robot motion,” in 2013 8th ACM/IEEE International
Conference on Human-Robot Interaction (HRI), 2013.
[54] M. J. Gielniak and A. L. Thomaz, “Enhancing interaction through
exaggerated motion synthesis,” in Proceedings of the 7th Annual
ACM/IEEE International Conference on HRI, 2012.
[55] M. Oudshoorn, J. de Winter, P. Bazilinskyy, and D. Dodou, “Bio-
inspired intent communication for automated vehicles,” Transportation
Research Part F: Traffic Psychology and Behaviour, 2021.
[56] G. Nelson, A. Saunders, and R. Playter, The PETMAN and Atlas
Robots at Boston Dynamics, 01 2019, pp. 169–186.
[57] M. Suguitan and G. Hoffman, “Blossom: A handcrafted open-source
robot,” J. Hum.-Robot Interact., vol. 8, no. 1, 2019.
[58] J. de Winter and D. Dodou, “External human–machine interfaces:
Gimmick or necessity?” Transportation Research Interdisciplinary
Perspectives, 2022. [Online]. Available: https://www.sciencedirect.com
[59] D. Dey, A. Habibovic, A. L¨
ocken, P. Wintersberger, B. Pfleging,
A. Riener, M. Martens, and J. Terken, “Taming the eHMI jungle:
A classification taxonomy to guide, compare, and assess the design
principles of automated vehicles’ external human-machine interfaces,”
Transportation Research Interdisciplinary Perspectives, 2020.
[60] SingaporeLTA. (2019) Autonomous vehicles: Identifying avs and trials
on roads. https://www.lta.gov.sg/content/ltagov/en/
industry innovations/ technologies/autonomous vehicles.html.
[61] S. H. Lee, N. Britten, A. Block, A. Pandya, M. Jung, and P. Schmitt,
“Coming in! : Communicating lane change intent in autonomous
vehicles,” in Proceedings of the 2023 ACM/IEEE Intrl. Conf. on HRI.
[62] D. Moore, R. Currano, M. Shanks, and D. Sirkin, “Defense against
the dark cars: Design principles for griefing of autonomous vehicles,”
in Proceedings of the 2020 ACM/IEEE Intrl. Conf. on HRI.
[63] T. Mulder and N. E. Vellinga, “Exploring data protection challenges
of automated driving,” Computer Law Security Review, 2021.
[64] C. Bloom and J. Emery, “Privacy expectations for human-autonomous
vehicle interactions,” in 2022 31st IEEE International Conference on
Robot and Human Interactive Communication (RO-MAN), 2022.
[65] B. Mehler, “Considerations regarding introduction of optical, au-
dible signals intended to signal the status & intent of au-
tomated vehicle systems,” https://unece.org/sites/default/files/2021-
11/ECE-TRANS-WP1-2021-Presentation-21e.pdf, UNECE, 2021.
[66] J. Shutko and T. Bray, “Evaluation of AV external communication in
the wild,” https://wiki.unece.org/download/attachments/75531441/
AVSR-02-23e.pdf?api=v2, UNECE, 2018.
[67] “Automated driving system marker lamp j3134-201905,” Society of
Automotive Engineers,” Standard, 2019.
[68] “Road vehicles — ergonomic aspects of external visual
communication from automated vehicles to other road users,”
https://www.iso.org/standard/74397.html, ISO, Standard, 2018.
[69] ETSI, “Tr 102 638 intelligent transport systems; vehicular communi-
cations,” Jun 2009. [Online]. Available: https://www.etsi.org/standards
[70] S. Shingu, “Socially aware autonomous mobility lightning talks,”
https://www.massrobotics.org/socially-aware-autonomous-mobility-
lightning-talks/, May 2023.
[71] B. J. Zhang and N. T. Fitter, “Nonverbal sound in human-robot
interaction: A systematic review,” Transactions on HRI, 2023.
[72] F. A. Robinson, O. Bown, and M. Velonaki, “Designing sound for
social robots: Candidate design principles,” International Journal of
Social Robotics, 2022.
[73] S. Joshi, “Towards Community-Robot Interactions,” in Proceedings of
the 17th European Conference on Computer-Supported Cooperative
Work: The International Venue on Practice-centred Computing and
the Design of Cooperation Technologies, 2019.
