Peter Alexander Kopciak / Patrick Kolar / Daniela Hellmann / Shadja El Aeraky /Matthias Dollfuss
Virtual Reality Driving Simulator Prototype for
Teaching Situational Awareness in Traffic
103 - Recent Advances in Multimedia Processing, Organization and
Visualization beyond Domains and Disciplines
Modern-day driving schools in Austria are using theoretical tests and a practical driving exam to
ensure a person is fit to drive on the streets. Students have to learn these skills by driving with a
teacher and taking lessons in a class. But many topics can only be covered in theory and not
experienced in person beforehand. Those include very risky situations, like jaywalking, and situations
that occur either infrequently or suddenly, such as clearing a lane for emergency vehicles
(“Rettungsgasse”) on a highway. In other fields, like piloting an aeroplane, a simulator is used to teach
the needed skills in a cost efficient and safe way. Additionally, virtual reality is often used to fully
immerse the user in the real world and to prepare him for action in the world. This project explores the
possibilities of using a virtual reality driving simulator prototype to teach driving a car in a safe
environment using cognitive learning methods and gamification. In contrast to existing (virtual) driving
simulators, our prototype focuses on teaching situational awareness and is designed to be used in
addition to classical driving schools. The student gains control of a vehicle which they can use to drive
around a city. During this journey they have to reach a specific location and complete various
objectives. As they drive they may be faced with various situations inspired by real world events, like
fog or an unexpected encounter with a speeding vehicle. They have to quickly assess the situation
and react in a responsible way. Throughout the session the student is assisted by Movado, a voiced
computer system build into the virtual car. This assistant comments on the driving of the user, gives
them information on their objectives and supports the user with abilities like making the car translucent
or superimposing the worldview with arrows to help navigating the fictional city. We conducted user
tests to assess the user’s impression and experience of the simulator and how the user reacts during
the experience in regard to cyber sickness. User feedback improved both the overall experience and
acceptance of this new method.
virtual reality, driving simulator, driving school, gamification, education, edutainment, situational
Learning how to drive is a process which takes time and effort. People attend driving schools and pay
for theoretical lessons to learn traffic laws, road signs, and how to behave on the streets during
various situations. Afterwards, they pay for practical lessons to acquire the actual skill of driving a
vehicle, including how to brake, how to turn, and how to park. Afterwards, they must use their
theoretical knowledge in real world scenarios, react to other people’s actions and drive in an imperfect
world where mistakes can happen. Learning to drive well is a feat which comes from experience,
practice, and hours spent on streets and highways. Some situations, like a crossing pedestrian,
objects on the street, or a rescue alley require attention and a cool head. Billboards or fog can be
distracting. All these things may be no problem for a seasoned driver, but someone who has just
finished driving school may be overwhelmed at first. The majority of car accidents in Austria happen
due to negligence, inexperience, overconfidence, failure to yield traffic, and driving too fast (Dietl /
Pfeifer 2015: 14, 83). Similar to other fields, like flight training, we can use virtual reality to simulate
critical moments during driving without harming the user or bystanders (Eckstein 1999: 8). One cannot
learn everything in driving school lessons, and situational awareness can only be gained through
experience. For this, we have developed a virtual reality driving software which ought to be an addition
to established learning procedures in driving schools. With thorough simulation of different situations
one can train and experience these moments first hand without the fear of making a fatal mistake. This
new method, unlike classic approaches, provides interaction and enhances first person experience
instead of theoretical distance (Dörner et al. 2013: 189). The prototype is subject to continuous
change and development. Therefore, some things mentioned in this paper may change as this project
is further developed and improved.
2. The Virtual Reality Driving Simulator
Using a steering wheel, pedals, a gear shift, and an Oculus Rift, one can drive a car in a fictive city
and is able to look around freely employing virtual reality. The user is encouraged to adhere to traffic
regulations and drive in a responsible manner. Faced with an objective, like getting to a place in the
city or delivering items, the driver is set on a journey through the virtual world which they can explore
on their own accord. This corresponds to two basic kinds of movement control in the virtual world:
exploration or driving around without a specific aim, followed by a search assignment which is
reaching a goal or point in the virtual world (Dörner et al. 2013: 170). With this quest mechanic, known
from role-playing games, we enable virtual immersion and identification and, consequently we improve
the student’s sense of presence to make the experience as appealing as possible (Dörner et al. 2013:
60). Furthermore, immersion and personal engagement facilitate learning in virtual environments
which can be achieved by presenting the user with a story, a purpose and tasks (Dede 2006). Using
gamification metaphors we can lead the user and make the learning experience fun and entertaining.
The actual learning process is not inherently visible to the user and is thus pushed into the
background, following the principles of edutainment but leading to an efficient learning activity
(Jayasinghe / Dharmaratne 2013: 1). Another aspect regarding immersion and learning quality is the
employment of a spatial audio system to give the user detailed feedback about their surrounding
environment and to mimic real world auditory perception, which aids presence and hinders cyber
sickness (Warnecke / Bullinger 1993: 144). To provide the user with a familiar horizon and a reference
point for gazing we implemented a very detailed skybox with high-definition clouds. This reduces
visual conflict and increases comfort within the virtual world (Oculus VR, LLC 2016: 6). We refrained
from displaying information at screen depth, in front of the view at a fixed position in the form of a
heads-up display, to stay inside the narrative, and to prevent information overload, to reduce display
cluttering and the number of necessary depth jumps, and to increase immersion (Schild / Masucha
2012: 4, 10).
2.1. Situations and Scenarios
As the user drives in the city and is focused on his quest they may be faced with various scenarios
and situations. Like in the real world, one is not able to prepare for the onset of specific circumstances
but has to react properly and quickly. This leads to higher “replayability” and encourages exploration.
As an example, the user may find himself in a foggy part of the city (see Figure 1) and is warned to
pay attention and drive slower due to the reduced line of sight. They may pass a construction site and
if they, despite bad visual conditions, have driven too fast, they might crash or cause an accident. One
may hear the distant sirens of a fire brigade and later at an intersection a fire engine may drive past
the user, ignoring all traffic lights. This is a typical moment where immediate reaction is required to
avoid a collision.
2.2. Movado the Assistant
The user is assisted by Movado, a computer assistant system and driving instructor. Movado is
represented as a smiley face displayed on a monitor placed on the dashboard. It speaks to the user
and can change its facial expression. Movado introduces the user to the controls and to the
possibilities of virtual reality and makes the user aware of their mirrors and speedometer. Movado will
give feedback on mistakes and achievements or provide information on progress towards the main
goal. During situations it may offer assistance and information. By using Movado as an intradiegetic
device, we can share extradiegetic knowledge, instructions or advice, and activate various supporting
systems to aid the driver inside the narrative context of our world and thus hinder the breakdown of
immersion (Schild et al. 2013: 170). As an example, Movado can make the car translucent to offer
increased visibility of the surroundings and the exact positioning of the wheels (see Figure 2).
2.3. Navigation Through the City
Movado can help the user find a way through the city. Depending on the user’s preference, it can act
like a classic navigation system announcing the next turn and displaying an arrow on its screen (see
Figure 4). Alternatively, it may place floating arrows inside the cyber world. (see Figure 3). This can be
combined with a line on the street, which may be followed to the destination (Sherman / Craig 2002:
336). Checkpoints act as landmarks at specific places in the world, which fade when the player is in
proximity or passes the previous checkpoint (Oliveira / Araujo 2012: 326).
From Left to Right: Figure 1: A Fog Situation with reduced sight; Figure 2: Movado makes the car translucent;
Figure 3: Arrows are placed in the city; Figure 4: Movado shows the next turn
3. Technological Aspects and Development
One of our major challenges throughout the project was software performance and maintaining a
constant frame rate at 75 fps. A low frame rate could induce simulator sickness by not providing
enough frames per second and if the frame rate didn’t match the refresh rate (75Hz) of the Oculus Rift
DK2 we would have flickering and screen tearing (Kolasinski 1995: 31). During development we
applied various techniques to increase performance, reduce CPU load and the amount of micro
stutters. For instance, we were reducing memory allocations when instantiating new objects by pooling
and recycling objects for reuse, accessing objects with tag references created at the start of the game
and generally improving programming structures within Unity modules. We resorted to batching by
grouping geometry together so it may be drawn with just one API call; baking the meshes by
combining all static and immobile meshes to one object; baking textures, shadows and vertex colours
into the models; and occlusion culling to reduce rendering of objects which were obstructed by other
objects. With all these measures MotEx is now capable of running at 75 fps while simulating 20 AI
agents which behave in accordance to traffic rules and use PhysX WheelColliders and raycasts to
avoid collision with objects (running on Windows 7 with Intel Core i7-920 [Q4 2008], 3GB DDR3-1066
RAM and GeForce GTX 970 4GB).
4. The User Experience
We developed the software with an agile mind-set and in an iterative process. We conducted many
smaller scale tests on new features to gain insight into potential problems. We exhibited our software
at various locations and performed three user experience and usability tests. With the feedback from
our target group (15 – 25 years old), we improved our concept (Dörner et al. 2013: 178-186). We also
tested for the occurrence of cyber sickness and simulator sickness which could pose a problem for
some people (Dörner et al. 2013: 70). Testing began in an early stage of development, and we
continuously improved the software based on user feedback.
4.1. The First User Test – Acceptance of Virtual Reality
Stage of Development: The user drove in a fictional parking lot. The experience was timed at about 2
to 3 minutes. They collected coins, had to evade a sudden cloud of smoke, drove a zigzag course with
pylons and then maneuvered into a parking space.
Objectives: We wanted to recognize design flaws as early as possible, find out how users would react
to our idea, and how they would react to virtual reality in respect to dizziness and nausea.
Design and Participants: Participants (n=55) were questioned about their sociodemographic
background before being allowed to use the software. Their performance was measured during
driving. Afterwards, each participant was questioned about his experience. All questions were
conducted without intervention from any team member by handing the participant a tablet with a
questionnaire. 87% of participants were in our designated target group.
Results: 93% were very positive about the experience, and 67% would like or would have liked to use
MotEx during driving school. 9% experienced nausea and 18% experienced mild discomfort. 67% had
no problem in navigating or controlling the car, and 64% were actively scanning the screen. 14% did
not have a driving license, and 14% were in driving school at the time. During testing, users mentioned
that they missed mirrors and would like better sound feedback and more realistic acceleration. They
mentioned that the pylons were too small and that they “lagged” (performance issues).
4.2. The Second User Test – Virtual Driving Exam
Stage of Development: We reproduced the parking lot of an actual driving school with the
corresponding practical driving exam. Students were required to drive slalom, pass a narrow spot,
maneuver into and out of a parking space while looking (“Drei S Blick”) for other cars. Afterwards, they
were required to park in a garage, then make a U-turn and accelerate to 40 km/h, and finally stop
before a designated line. We implemented a spatial sound system, mirrors and a computer voice
(acting as a driving teacher) which told the user what to do next.
Objectives: We wanted to test how our implemented features and changes since the last test would
affect the user experience and how younger people would respond to our software.
Design and Participants: Similar to the previous test, participants (n=54) were questioned before and
after the experience through a questionnaire on a tablet without intervention of team members. Unlike
the previous test, participants wore headphones and were guided through the track by a computer
voice. The performance of the virtual driving exam was measured inside the software. 31% of the
participants were in the 15-25 year old age group, and 48% were in the 5-14 age group.
Results: 96% were very positive about the experience, and 74% would like to or would have liked to
use MotEx during driving school. 4% experienced nausea, and 11% experienced mild discomfort. 72%
had no problem in controlling the car or navigating the course. Participants were satisfied with sound
feedback but mentioned that the computer voice was talking too much and too slowly. Feedback
revealed that the mirrors needed to be adjusted and objects appeared too small; traffic signs and the
finishing line were not as visible as we had hoped. 36% did not have driver licenses, and 32% were in
driving school at the time. Acceleration was better, but the braking distance seemed off. This time
there were almost no performance issues.
4.3. The Third User Test – Navigating Through a City
Stage of Development: We replicated the city of St. Pölten with traffics lights, road markings,
environment, low poly buildings and trees. A majority of dynamic shadow had been removed for
performance reasons. We implemented an AI system with cars which navigate traffic, adhere to traffic
regulations and react to the player car. At this point we changed the concept of our software. Instead
of offering the same services as driving schools, we focused on providing an addition to existing
driving school learning procedures.
Objectives: Just as before, we wanted to test how users would react to our new features and if the
development of the project was focused in the right direction. Another important aspect of the user test
was to observe how players would drive in a city and how they would react to the AI. Participants
would drive approximately 10 minutes while guided by floating arrows.
Design and Participants: The participants (n=34) were, as before, questioned before and after the
experience in the simulator. 72% were in our designated target group of 15 – 25.
Results: 100% of our participants were very positive, and 78% would like to or would have liked to use
MotEx during driving school. 3% experienced nausea and 16% mild discomfort. 14% did not have a
driving license. Participants paid attention to surrounding simulated drivers, although they were still
speeding due to lack of awareness of their velocity. Overall they were not hindered or dissatisfied with
the presence of the AI cars. Participants liked to simply drive around while exploring the city.
5. ConclusionWe developed a virtual reality driving school simulator to help students learn how to
drive and prepare them for real world traffic and situations. At first we tried to simulate actual driving
schools and learning procedures, but we found out that there is a demand for learning specific skills
which could not be taught during traditional learning procedures. Our concept changed to teaching
experience and situational awareness as a complimentary learning offer to existing driving school
learning procedures. We focused more on the gamification and edutainment aspect of the software
experience. The users can learn how to behave during various situations while navigating a fictional
city and following a special quest. These situations happen at random without warning. The users are
accompanied by a virtual computer assistant inside the car which serves as a driving teacher,
feedback system and narrative device for storytelling and virtual reality display techniques (like
superimposing a path) without breaking immersion. With an agile mind-set and by staying in touch with
our target group’s trough user experience and usability testing we were continuously developing and
improving features. Like most virtual reality simulators before us, we ran into issues with cyber
sickness, motion sickness and simulator sickness. By improving programming structure and unity
module design, reducing CPU load, minimizing draw calls and baking meshes we could improve
performance to a point where our prototype would run at 75 fps while deploying 20 AI agents using
PhysX WheelColliders, raycasts while adhering to traffic rules and traffic priority. We introduced a
high-definition skybox, a spatial audio system and refrained from displaying too much information on
screen-depth to stay inside the narrative and to hinder the breaking of immersion. As we improved our
software, performance and concept, people were more likely to use and enjoy the experience. We
have now implemented a new navigational concept, inspired by traditional real life navigational
systems and combined it with arrows, checkpoints and lines, similar to those existing solutions in
virtual games. This feature and the general current state of development, including the assistant and
new situations, will need further tests for acceptance and usability. Furthermore, we plan to improve
the AI so that it may react better to the user car, initiate various manoeuvres and force specific
situations. This, combined with randomizing the occurrence of other situations, will yield a more
dynamic learning experience.
Dede C. (2006): How Mediated Immersion Shapes Learning. In: Virtual Reality Conference 2006, 13.
Dietl E. / Pfeifer E. (2015): Statistik Austria – Straßenverkehrsunfälle Jahresergebnisse 2014.
Dörner R. / Broll W. / Grimm P. / Jung B. (Eds.) (2013): Virtual und Augmented Reality (VR/AR):
Grundlagen und Methoden der Virtuellen und Augmentierten Realität. Berlin: Springer Verlag.
Eckstein J. (1999): Echtzeitfähige Kollisionserkennung für Virtual Reality Anwendungen. München:
Jayasinghe U. / Dharmaratne A. (2013): Game based learning vs gamification from the higher
education students’ perspective. In: Teaching, Assessment and Learning for Engineering (TALE),
2013 IEEE International Conference on 26 – 29 August 2013, 683 – 688.
Kolasinski E. M. (1995). Simulator Sickness in Virtual Environments. Virginia: U.S. Army Research
Oculus VR, LLC (2016): Introduction to VR – Oculus Best Practice.
Oliveira A. / Araujo R. B. (2012): Creation and visualization of context aware augmented reality
interfaces. In: AVI '12 Proceedings of the International Working Conference on Advanced Visual
Interfaces, 324 – 327.
Schild J. / Bölicke L. / LaViola Jr. J. J. / Masuch M. (2013): Creating and analysing stereoscopic 3D
graphical user interfaces in digital games. In: CHI '13 Proceedings of the SIGCHI Conference on
Human Factors in Computing Systems, 169 – 178.
Schild J. / Masuch M. (2012): Designing stereoscopic information visualization for 3D-TV: What can
we can learn from S3D gaming. In: SPIE Proceedings Vol. 8288: Stereoscopic Displays and
Applications XXIII, 91 – 103
Sherman W. R. / Craig A. B. (2002): Understanding Virtual Reality: Interface, Application, and Design.
San Francisco: Elsevier, Morgan Kaufmann Publishers.
Warnecke H.J. / Bullinger H.J. (Eds.) (1993): Virtual Reality: Anwendungen und Trends. Berlin: