Content uploaded by Grayson Bailey
Author content
All content in this area was uploaded by Grayson Bailey on Sep 10, 2022
Content may be subject to copyright.
Performing Immersive Virtual Environment User Studies
with VREVAL
A user study in wayfinding behavior in four train station design options
Grayson Bailey1, Olaf Kammler2, René Weiser3, Ekaterina Fuchkina4, Sven Schneider5
1,2,3,4,5Bauhaus-Universität Weimar
1,2,3,4,5{ grayson.daniel.bailey | olaf.kammler | rene.weiser | ekaterina.fuchkina |
sven.schneider }@uni-weimar.de
New advances in user studies within Immersive Virtual Environments (IVEs) have made
possible highly immersive Pre-Occupancy Evaluations (Pr-OEs) during the architectural
design process. However, there remains a lack of easy methods for integrating these IVE-
based user studies into design development phases while also providing reliable data-
driven results. In VREVAL, a framework for IVE-based architectural user studies, several
technical and interface solutions for architectural user studies have been addressed. In
the following paper, recent developments in the VREVAL hybrid web platform and
desktop application will be reviewed via the presentation of a partial Pr-OE of four
alternative train station designs. By examining the design process of a route choice user
study which compares four schematic designs, the VREVAL modular methods for creating
study tasks and questionnaires within a BIM-generated IVE will be demonstrated.
Keywords: Pre-Occupancy Evaluation, Immersive Virtual Environment, Wayfinding,
User Centered Design, Architectural Study Design
INTRODUCTION
With the increased use of Immersive Virtual
Environments (IVEs) in Architecture, there is an
opportunity for developing Pre-Occupancy
Evaluations (Pr-OEs) which integrate meaningful
user feedback into architectural design phases. In
the past few years there has been a focus on
developing various methods for designing and
executing IVE-based studies (Grübel et al 2017;
Moloney et al 2019; Agirachman and Shinozaki
2021), although there remain high levels of
specificity when setting up such studies, barriers of
programming and other technical knowledge sets,
and a low rate of reproducibility (Wölfel et al 2021).
To address these needs of easy and uniform study
design with reproducible results, significant
development of the VREVAL study framework
(Schneider et al 2017) has taken place. A new web-
based platform for modular study design builds
upon the BIM-oriented workflow of VREVAL,
allowing for easier integration into architectural
design phases using limited additional software.
Further, VREVAL uses a combination of web and
desktop applications to streamline the process of
designing, executing, and analyzing architectural
user studies, while also centralizing study results and
distribution methods for participant and researcher
access.
This paper will present an example of the
VREVAL workflow through a partial Pr-OE which
examines wayfinding and route choice in four
alternative train station design options. Study design
Volume 2 – Co-creating the Future – eCAADe 40 | 437
with VREVAL will be shown through a step-by-step
review of relevant study components and their
administration on the web platform, and present the
method of conducting the evaluation with the
VREVAL desktop application. Finally, results from the
study will be reviewed.
DESIGN-INTEGRATED USER STUDIES
Integrating user studies into early architectural
design phases opens the practical potentials of IVE-
based user studies. Rather than remaining purely a
source of scientific inquiry, well-integrated user
studies can help designers test designs in terms of
user-centered values, while also revealing
intransigent design problems. As in Figure 1, the
workflow of VREVAL prioritizes an easy transition
between digital models and data-driven output
which is helpful in the development of user-oriented
spaces.
At the center of VREVAL is a database which
stores study data and is accessed via two interfaces:
the VREVAL web platform and the VREVAL desktop
application [1]. As in Figure 2, the web platform is
used to design, administrate, and document user
studies from any device with a modern web-browser
and internet connection. The desktop application is
used to place study design elements, test study
modules, and perform user studies in either Head-
Mounted Display (HMD) Mode or Monitor-Mouse-
Keyboard (MMK) Mode. In order to conduct a user
study, the relevant study data is downloaded from
the database and visualized within the desktop
application. Results are then uploaded to the
database for each task individually. Once the study is
finished, study documentation is available to
download from the web platform.
STUDY CONCEPT
The following user study examines the wayfinding
behavior and route choice preferences of
participants within four train station building design
alternatives. Wayfinding analysis is a popular
application of IVE-based studies, with previous
research focusing on evacuation behavior (Lin et al
2020, Zhang et al 2021, Irshad et al 2021). However,
due to the difficulties of designing and
administering IVE-based user studies, almost no
research has been done on the integration of these
studies into the design process. Non-emergency
studies are largely based on already constructed
designs (Kuliga et al 2019, Ewart and Johnson 2021)
or focused purely on simulation (Gath-Morad et al
2022). While these focuses are also possible with
VREVAL, this study aims to resolve the integration of
user studies as a tool within the design process.
The context of this study is a train station
building in an anonymized city in Germany which
requires redevelopment due to a structure in
disrepair and outdated programming. One of the
major problems which must be addressed in the
design of a new station building is a public pathway
(PUB) to Platforms 2 and 3 which draws much of the
pedestrian traffic away from the station building
(STA). The clients have asked for design options
which preserve the pedestrian connection between
the sides of the station, but also entice engagement
with the station and its services. Via a between-
group study, the aim is to compare participant
wayfinding behavior and rates of path usage rates of
path usage in three new designs and the current
construction (Control), as seen in Figure 3. Through
a series of wayfinding tasks and questionnaires,
participants express their preference between STA
and PUB in each design and report qualities which
Figure 1
VREVAL workflow
Figure 2
VREVAL framework
overview
438 | eCAADe 40 – Volume 2 – Co-creating the Future
influenced their preference. Figure 4 shows a map of
the study and the locations of interaction during
three sections.
In the first section (1.1-1.2-1.3) the participant is
placed along the approach from the city center and
asked to find the Travel Center (1.2). Then, they are
asked two questions concerning the ease of finding
the Travel Center, and told to catch a train on
Platform 2 (1.3). In the second section (2.1-2.2-2.3),
the participant is placed on Platform 2 (2.1) and
asked to find the Kiosk (2.2). At 2.2 they are asked two
questions about finding the Kiosk, and then told to
find a car at 2.3. Section 1 and 2 aim to acclimate
participants to both pathways STA and PUB before
asking them to choose a
preferred route.
Finally, the third section (3.1-3.2A-3.2B) places
the participant once more on approach to the
Figure 3
Design alternatives
for Route Choice
comparison
Figure 4
Map of study
sections 1, 2, and 3
with important
locations
Volume 2 – Co-creating the Future – eCAADe 40 | 439
station and asks that they find their way to Platform
2 via whichever path they prefer. Once the
participant has declared their preference, they are
asked to select from a list which qualities were
important in making their choice.
STUDY DESIGN
Study content was organized using the VREVAL web
platform, and each layer of elements in VREVAL will
be described through the design process of the
current study. As in Figure 5, VREVAL involves a
hierarchy of elements which range from the most
basic elements (Environment Bundle, Markers, and
Forms) to organizational elements (Tasks, Scenarios,
and Playlists) which contain and order the basic
elements.
Beginning with the most central three elements
of a VREVAL user study, the researcher supplies the
study content: digital models used in the IVE
(Environment Bundle), locations within the IVE
(Markers), and instructional or questionnaire text
(forms). The Environment Bundle is easily exported
from various 3D modelling software, although this
study used Autodesk Revit [2] and exported with
Lumion [3]. Separate exports are made for urban
context, design options, and various entourage, and
then loaded into a Unity project [4], where they are
bundled under categories (urban context, design
options, etc.). Using the Asset Bundle Browser [5], the
Environment Bundle is created and then uploaded
to the web platform. Environment Bundles are
organized so that designers are able to limit the
amount of IVE information to strictly necessary
model elements used during each task (for
computation and loading time ease).
Next, the Markers used in the study (locations of
interaction) are created directly in the IVE by using
the design function in the VREVAL desktop
application. Once placed, these Markers are
uploaded to the web platform, where they can be
attached to different Tasks, and combined with
Forms. Forms are multi-page elements which
provide all textual information, including
instructional prompts, questions (single-answer,
multi-answer), and ratings, and are created on the
web platform.
Tasks
Once the most basic elements (Environmental
Bundle, Markers, and Forms) exist, study Tasks can be
created depending on the focus of the study. In this
example, three Wayfinding Tasks are created to
correspond with the study sections in Figure 4. All
types of Task in VREVAL (Default, Questionnaire,
Wayfinding, Placing, Annotation) require general
settings (position and orientation tracking, user
movement allowances, etc.), Environment Bundle
settings, and the linking of Markers and Forms as
required by the Task.
Figure 6 shows the starting marker for the first
Wayfinding Task is “StreetCorner” and the Form that
first opens at this marker is “RC1-A", which
introduces the task to the participant. The Travel
Center Marker and questionnaire Form are included
as a waypoint, and the final Marker in the task
(“Platform A”) is paired with an outro Form (“RC1-C”).
Figure 5
VREVAL element
hierarchy
440 | eCAADe 40 – Volume 2 – Co-creating the Future
In this way, section 1 from Figure 4 is prepared for
participant performance in VREVAL.
Scenarios and Playlists
The three Tasks created must be grouped together
as a Scenario. The VREVAL web platform uses
Scenarios as collections of Tasks within a study,
which the participant performs without any break
within the IVE. More complex studies might
involve the participant performing multiple
scenarios which address different user-centered
experiences of architectural space (such as spatial
relations ratings, option choice experiments,
visibility evaluations, etc.).
However, with a between-group study such as
this, each design option requires a separate scenario
in order to differentiate the Environment Bundle
settings and design-specific Markers. However,
these four nearly identical Scenarios can
utilize the same Markers (except for the Marker at
1.2) and Forms for identical experimental conditions.
Finally, a Playlist is a set of Scenarios, providing
another layer of organization for studies which
might investigate divergent variables or other
conditions that would require different sets of
Scenarios.
STUDY ADMINISTRATION
Once a study is fully developed, it can be made “live”
on the web platform and performed using the
desktop application. Unique user codes are
generated so that individual sessions can be
reviewed without identifying participants.
Study Sequence
This route choice study was a part of a larger user
study that additionally focused on waiting
preferences, spatial relations, and a design choice
experiment. A study sequence was provided by
researchers to be conducted by an attendant who
walked participants through the process and
monitored their progress.
The study sequence was broken into three
segments: [A] Pre-Study Phase, [B] Study Phase, [C]
Post-Study Phase. The Pre-Study Phase involved an
introduction, informed consent, and a VREVAL
interface tutorial. Following this, the Study Phase
was comprised of four studies and a follow up
questionnaire, which took approximately 25 to 35
minutes. The route choice study (seen in HMD mode
in Figure 7) was performed first, as to evaluate
wayfinding
Figure 6
VREVAL Scenario
with Wayfinding
Tasks created on
the VREVAL web
platform
Figure 7
HMD view during
the route choice
study
Volume 2 – Co-creating the Future – eCAADe 40 | 441
behavior before the participant formed a more
nuanced knowledge of the environment.
Afterwards, the participant was debriefed and
exited, after which the attendant sanitized the HMD
and other equipment using disinfectant wipes and
an ultraviolet light bath for 10 minutes.
In total, 62 participants from Bauhaus-
Universität Weimar performed the study, with either
15 or 16 participants per design option group. 47
self-reported as “feminine / weiblich”, 14 as
“masculine / männlich”, and 1 as “non-binary /
divers”. The study used the preliminary question “is
this your first year of University” to evenly distribute
first-semester students throughout each between-
group.
RESULTS
After the completion of the study (December 2021 -
January 2022), results for the route choice scenario
were downloaded from the VREVAL web platform.
Questionnaire and other self-reported participant
answers were analyzed statistical software, while IVE
positions and pathways were visualized in Autodesk
Revit using the VREVAL Dynamo package [1].
Route Choice Preferences
Participants showed a heavy preference for the
pathway PUB, although this is partially due to the
failure of the study design to ensure knowledge of
both paths. As shown in Figure 8, participant activity
during Tasks 1 and 2 demonstrates that most
participants in all design options avoided STA during
every task, which drastically undermined the
researcher’s method for demonstrating both
pathways before determining route choice.
Subsequently, only 4 out of the 62 participants
in all design options preferred STA during Task 3.
Control had the most participants use STA during
Task 1-2 (6/16), which is most likely due to the
proximity of the Travel Center Marker in Control to
the STA stairwell. However, Control also had the
lowest rate of STA preference (0/16). Meanwhile,
Option A had the highest rate of preference (2/15),
and both Option B and C had only 1 participant each
chose the pathway STA.
As for the ease of finding the Travel Center (1.2)
and the Kiosk (2.2), responses to the questionnaires
at each location showed that all design options were
rated neutrally (1 = very easy, 5 = very difficult) in
finding the Travel Center (MControl = 2.56, SDControl =
1.03; MA= 2.57, SDA = 1.09; MB = 2.94, SDB = 0.93; MC =
3.0, SDC = 1.07) and that Option B most easily found
the Kiosk (MControl = 2.56, SDControl = 1.46; MA= 3.27, SDA
= 1.28; MB = 1.94, SDB = 1.18; MC = 2.47, SDC = 0.99).
Additionally, Option A proved the most difficult to
navigate in Task 2-1, which is compatible with the
rate of pathway deviation for 2-1, as seen in Figure 8.
Generally, results shows that the influence of
pathway PUB is stronger than first assumed.
Therefore, the issue of attracting pedestrians
through any of the train station buildings must be
seen as a central aspect of further design
development in any potential station building
proposal, as no current design option performs
adequately in this regard.
Ease / Difficulty Qualities
In addition to rating the ease of navigation,
participants were asked at 1-2 and 2-2 to select
qualities that they felt made the location easier or
more difficult to find. Both 1-2 and 2-2 were
evaluated as difficult to find due to a lack of visibility,
as seen on Table 1. Also, in both locations most
participants throughout all design options rated the
Travel Center and Kiosk as having easy centrality.
However, participants reported low visibility at 1-2
more frequently in Options B and C, and lower
visibility at 2-1 more frequently in Control and
Option A.
Route Influence Qualities
The questionnaire performed by the participants in
Task 3 showed that the qualities of “Distance” (66%),
“Ease” (60%), and “Visibility” (61%) were the most
important when choosing between pathway STA
and PUB. The least chosen qualities were “I am not
sure…” (0%), “Aesthetics” (3%, “Other Reasons” (3%).
442 | eCAADe 40 – Volume 2 – Co-creating the Future
Figure 8
Participant
wayfinding
pathways and route
choices during each
study section.
Volume 2 – Co-creating the Future – eCAADe 40 | 443
Task 1.2: "Which qualities do you think made the Travel Center easy / difficult to find?"
Answers Total Control Group A Group B Group C
It w a s vis ible from a dis t a nc e ( Ea sy V is ibilit y ) 9
15%
5121
It w a s NOT visib le fro m a dist a nc e ( Diff ic ult V is ibilit y ) 44
71%
991313
It was marked well with signage ( Easy Signage ) 23
37%
7457
It was NOT marked well with signage ( Difficult Signage ) 22
35%
6763
It w a s loc a ted in a centra l loca t ion ( E a s y C e nt r a lity ) 36
58%
910 611
It was NOT located in a central location ( Difficult Centrality ) 13
21%
5134
Task 2.2: "Which qualities do you think made the Kiosk easy / difficult to find?"
Answers Total Control Group A Group B Group C
It w a s vis ible from a dis t a nc e ( Ea sy V is ibilit y ) 8
13%
1322
It w a s NOT visib le fro m a dist a nc e ( Diff ic ult V is ibilit y ) 39
63%
11 12 7 9
It was marked well with signage ( Easy Signage ) 23
37%
6656
It was NOT marked well with signage ( Difficult Signage ) 30
48%
8796
It w a s loc a ted in a centra l loca t ion ( E a s y C e nt r a lity ) 33
53%
11 6 9 7
It was NOT located in a central location ( Difficult Centrality ) 19
31%
5725
Task 3.2: "For which reasons did you choose this path to the train?"
Answers Total Control Group A Group B Group C
I am not sure, I was just wandering... 0
0%
0000
I expected this path to be the shortest (Distance) 51
66%
15 11 10 15
I expected this path to be the simplest (Ease) 37
60%
12 10 8 7
I c ould s e e th e pa t h t o P latf orm 2 this w ay ( V isibility) 38
61%
911 810
I expected this path to be the safest way (Safety) 6
10%
4110
I expected this path to be the most inviting (Welcomeness) 10
15%
1225
I expected this path to the most beautiful (Aesthetics) 4
3%
0112
I expected this path to be the most lively (Activity) 7
10%
3301
Other reasons... 3
3%
0120
Both preference groups from all design option
groups (STA = 4, PUB = 58) chose the same qualities
for influence on route choice. This shows that while
the qualities were preferred, further experiments
must clarify how participants evaluate distance and
ease of pathways. Also, the influence of primary
visual access should be tested, as the initial visibility
of PUB seems to have outweighed the delayed
visibility of STA in each design option.
DISCUSSION
VREVAL has shown the ability to integrate user
studies into the architectural design process with
relative ease, to organize study components with a
centralized and easily accessible method, and
standardize results in a manner that is potentially
reproducible. The use of Wayfinding tasks in a
VREVAL study shows the intricacies of behavior via
self-reported values and visualizations that provide
meaningful feedback for further design phases. The
ability to construct a multiple design option user
study starting with only digital models is a step
Table 1
Qualities
Questionnaire
responses for Tasks
1.2, 2.2,
and 3.
444 | eCAADe 40 – Volume 2 – Co-creating the Future
forward for both practical and scientifical use of IVE-
based studies.
The outcomes of the study presented here show
the utility of user studies as a design tool, insomuch
that the study highlights the inability of each design
to adequately attract pedestrians through the
station building and the need for continued design
development on the subject. Whereas the designers
assumed the success of several different methods for
providing an attractive path to the platforms, the
data-driven user study shows only nominal shifts
from current underperforming values.
Specifically in the field of wayfinding studies, the
utility of VREVAL is positioned well for producing
research studies which follow up the latest directions
of cross-cultural analysis (Zhu et al 2020), and the
differences between real-world and digitally
modelled studies (Arias et al 2022, Feng et al 2022),
and other extensions beyond documenting
evacuation behaviors.
Outlook
Future developments of VREVAL user studies will
require additional research in two directions: IVE
conditions, and study methodologies. First, IVE
conditions and the technical abilities for a full range
of tasks and IVE activities need additional
demonstration. For VREVAL, this means presenting
the full range of task types (Placing Tasks,
Annotation Tasks, Questionnaire Tasks), and further
developing activities which are required for
common research practices.
Additionally, the following issues require
general study in the manners that they effect
participant IVE experiences: Level of Detail (LoD) in
models and material representation, simulating
ranges of environmental conditions (weather, time
of day, etc.), and the presence of static or dynamic
entourage (human figures, furniture, etc.). Previous
research has shown that IVE studies are generally
comparable to real-world studies and that they offer
an opportunity for expanding topics and userships
of research (Krösl et al 2018), however more nuanced
understanding is required of the influence of various
elements within IVEs. The presence of crowds,
animated or otherwise, has also shown to influence
route choice (Yassin et al 2021), although little has
been tested in respect to other study methods than
wayfinding and route choice.
Finally, study methodologies must be further
tested by researchers in developing IVE common
practices. However, the development of these
practices and the avoidance of biases are not
centrally the responsibility of IVE frameworks like
VREVAL.
ACKNOWLEDGEMENTS
The study is part of the project OpenVREVAL, in
cooperation with Deutsche Bahn Station & Service
AG and funded by the Thüringer Ministerium für
Wirtschaft, Wissenschaft und Digitale Gesellschaft
(TMWWDG) under the funding number 5575/10-14.
Special thanks are given to Paul Leon Pollack for
assistance, and to the students from Bauhaus-
Universität Weimar for their creative input and
investigative spirit.
REFERENCES
Arias, S, Mossberg, A, Nilsson, D and Wahlqvist, J
2021, ‘A Study on Evacuation Behavior in
Physical and Virtual Reality Experiments’, Fire
Technology, 58, pp. 817-849.
Agirachman, FA and Shinozaki, M 2021, ‘VRDR: An
Attempt to Evaluate BIM-based Design Studio
Outcome Through Virtual Reality’, Proceedings
of the 26th International Conference of the
Association for Computer-Aided Architectural
Research in Asia (CAADRIA) 2021, Hong Kong, pp.
223-232.
Ewart, IJ and Johnson, H 2021, ‘Virtual Reality As a
Tool to Investigate and Predict Occupant
Behaviour in the Real World: The Example of
Wayfinding’, Journal of Information Technology
in Construction, 26, pp. 286-302.
Feng, Y, Duives, DC, Hoogendoorn, SP 2022,
‘Wayfinding behaviour in a multi-level building:
A comparative study of HMD VR and Desktop
Volume 2 – Co-creating the Future – eCAADe 40 | 445
VR’, Advanced Engineering Informatics,
51(101475), pp. 1-22.
Gath-Morad, M, Melgar, LEA, Conroy-Dalton, R and
Hölscher, C 2022, ‘Beyond the shortest-path:
Towards cognitive occupancy modelling in
BIM’, Automation in Construction, 135(104131),
pp. 1-14.
Grübel, J, Jiang, MH, Hölscher, C, Hackman, DA and
Schinazi, VR 2017, ‘EVE: A Framework for
Experiments in Virtual Environments’ in
Barkowsky, S, Burte, H, Hölscher, C and
Schultheis, H (eds.) Spatial Cognition X, Springer-
Verlag, Berlin, pp. 159-176.
Irshad, S, Perkis, A and Azam, W 2021, ‘Wayfinding
in Virtual Reality Serious Game: An Exploratory
Study in the Context of User Perceived
Experiences’, Applied Sciences, 11(7822), pp. 1-
19.
Krösl, K., Bauer, D., Schwärzler, M., Fuchs, H., Suter, G.
and Wimmer, M. 2018, ‘A VR-based user study
on the effects of vision impairments on
recognition distances of escape-route signs in
buildings’, The Visual Computer, 34, pp. 911-923.
Kuliga, S, Nelligan, B, Dalton, RC, Marchette, S,
Shelton, AL, Carlson, L and Hölscher, C 2019,
‘Exploring Individual Differences and Building
Complexity in Wayfinding: The Case of the
Seattle Central Library’, Environment and
Behavior, 51(5), pp. 622-665.
Lin, J, Zhu, R, Li, N, and Becerik-Gerber, B 2020, ‘Do
people follow the crowd in building emergency
evacuation? A cross-cultural immersive virtual
reality-based study’, Advanced Engineering
Informatics, 43(101040), pp. 1-13.
Moloney, J, Globa, A, Wang, R and Khoo, C 2019
‘Principles for the application of mixed reality as
pre-occupancy evaluation tools (P-OET) at the
early design stages’, Architecture Science Review,
63(5), pp. 441-450.
Schneider, S, Kuliga, S, Weiser, R, Kammler, O and
Fuchkina, E 2017, ‘VREVAL: A BIM-based
Framework for User-Centered Evaluation of
Complex Buildings in Virtual Environments’,
Proceedings of the 36th eCAADe Conference, Lodz,
pp. 833-842.
Wölfel, M, Hepperle, D, Purps, CF, Deuchler, J and
Hettmann, W 2021, ‘Entering a new Dimension
in Virtual Reality Research: An Overview of
Existing Toolkits, their Features and Challenges’,
2021 International Conference on Cyberworlds
(CW), Caen, pp.180-187.
Yassin, M, El Antably, A, Abou El-Ela, M A.S. 2021,
‘The others know the way: A study of the impact
of co-presence on wayfinding decisions in an
interior virtual environment’, Automation in
Construction, 128(103782), pp. 1-13.
Zhang, M, Ke, J, Tong, L and Luo, X 2021,
‘Investigating the influence of route turning
angle on compliance behaviors and evacuation
performance in a virtual-reality-based
experiment’, Advanced Engineering Informatics,
48(101259), pp. 1-15.
Zhu, R, Lin, J, Becerik-Gerber, B and Li, Nan 2020,
‘Influence of architectural visual access on
emergency wayfinding: A cross-cultural study in
China, United Kingdom and United States’, Fire
Safety Journal, 113(102963), pp. 1-16.
[1] https://database.architektur.uni-
weimar.de/dashboard
[2] https://www.autodesk.eu/products/revit/over
view?term=1-YEAR&tab=subscription
[3] https://www.lumion3d.de
[4] https://unity.com
[5] https://github.com/Unity-
Technologies/AssetBundles-Browser
446 | eCAADe 40 – Volume 2 – Co-creating the Future