Content uploaded by Jos P. van Leeuwen
Author content
All content in this area was uploaded by Jos P. van Leeuwen on Feb 08, 2019
Content may be subject to copyright.
Effectiveness of Virtual Reality in Participatory
Urban Planning
A Case Study
Jos P. van Leeuwen
The Hague University of Applied Sciences
The Hague, The Netherlands
j.p.vleeuwen@hhs.nl
Klaske Hermans
Municipality of The Hague
The Netherlands
klaske.hermans@denhaag.nl
Antti Jylhä
The Hague University of Applied Sciences
The Hague, The Netherlands
a.t.jylha@hhs.nl
Arnold Jan Quanjer
The Hague University of Applied Sciences
The Hague, The Netherlands
a.j.r.b.quanjer@hhs.nl
Hanke Nijman
Twynstra & Gudde
Utrecht, The Netherlands
hni@tg.nl
ABSTRACT
In urban planning, 3D modeling and virtual reality (VR) provide
new means for involving citizens in the planning process. For
municipal government, it is essential to know how effective these
means are, to justify investments. In this study, we present a case
of using VR in a municipal process of civic participation
concerning the redesign of a public park. The process included co-
design activities and involved citizens in decision-making through
a ballot, using 3D-rendered versions of competing designs. In co-
design, 3D-modeling tools were instrumental in empowering
citizens to negotiate design decisions, to discuss the quality of
designs with experts, and to collectively take decisions. This paper
demonstrates that, in a ballot on competing designs with 1302
citizens, VR headsets proved to be equally effective compared to
other display technologies in informing citizens during decision
making. The results of an additional, controlled experiment
indicate that VR headsets provide higher engagement and more
vivid memories than viewing the designs on non-immersive
displays.
By integrating research into a municipal process, we contribute
evidence of cognitive and engagement effects of using 3D
modeling and immersive VR technologies to empower citizens in
participatory urban planning. The case described in the paper
concerns a public park; a similar approach could be applied to the
design of public installations including media architecture.
Author Keywords
Participatory design; virtual reality; urban planning; civic
engagement.
CCS Concepts
• Human-centered computing ~ Virtual reality
• Human-centered computing ~ Participatory design
1 INTRODUCTION
In a participatory society, authorities share responsibilities with
citizens and civil communities [13,37] in many domains, such as
health care, safety, social security, and urban planning and the
organization of public spaces. City governments are looking for
optimal ways to improve the infrastructure for public
participation [6,10] using new technologies to support this. The
municipality of The Hague collaborated with The Hague
University of Applied Sciences in an activity of participatory
design to find evidence of the effectiveness of Virtual Reality
technology in participatory urban planning processes.
Virtual Reality (VR) devices such as 3D-rendering headsets and
smartphones have become accessible to a broad audience,
increasing their potential utility beyond specialist settings. Thanks
to their immersive nature, VR headsets can provide a tool for
supporting decision-making processes in architecture and urban
planning, by virtually placing the observer in the context of the
design. 3D rendering in such processes has evolved from expert
systems to participatory systems, in which the public is given the
opportunity to experience the envisioned design through an
immersive visualization [9,20,39,40]. Experimental research
suggests that VR can increase public participation in such
processes [9,17] and provide a sufficiently realistic experience to
make judgements on the quality of the presented content, instead
of paying attention to rendering artifacts.
The municipality of The Hague incorporated 3D rendering and VR
technology in the participatory re-design of a public park in a
neighborhood with approximately 13,500 residents. The
municipality defines four levels of participation: consultation,
advice, co-production, and co-decision [16]. These levels formed
the backbone for a co-creative process that, although initiated and
facilitated by the municipality, was owned by the residents. The
intention of the municipal district director and the neighborhood
manager was to stimulate a sense of ownership in residents and
Permission to make digital or hard copies of all or part of this work for personal or
classroom use is granted without fee provided that copies are not made or distributed
for profit or commercial advantage and that copies bear this notice and the full
citation on the first page. Copyrights for components of this work owned by others
than ACM must be honored. Abstracting with credit is permitted. To copy otherwise,
or republish, to post on servers or to redistribute to lists, requires prior specific
permission and/or a fee. Request permissions from Permissions@acm.org.
MAB18, November 13–16, 2018, Beijing, China
© 2018 Association for Computing Machinery.
ACM ISBN 978-1-4503-6478-2/18/11…$15.00
https://doi.org/10.1145/3284389.3284491
MAB'18, November, 2018, Beijing, China J.P. van Leeuwen et al.
engage them in a do-it-with-others (DIWO) activity regarding the
development of the park in their neighborhood [7].
The participation process can be summarized as follows, with the
participation levels shown in italics:
1. Call for participation – consultation
All residents in the neighborhood received an invitation to send
suggestions regarding the park to the neighborhood manager
and/or to partake in a workgroup for co-designing the revamp of
the park.
2. Co-creation workgroup – co-production
The workgroup of citizens engaged in a series of intensive co-
design sessions with experts from the municipality, making use of
3D modeling, leading to three high-quality designs for the park.
3. Public ballot – co-decision
The three variant designs were submitted to voting by all
neighborhood residents. The residents were allowed to view the
proposed designs using a variety of display technologies.
4. Final design – co-production
The city’s landscape architect and the workgroup co-produced the
final detailed design, based on the winning variant.
This paper reports on research tapped into phases 2 and 3 of the
process. These phases provided a vessel to investigate the
effectiveness of VR technology in participatory urban planning.
Specifically, we were interested in the following research
questions that mainly relate to phase 3, the public ballot:
RQ 1. How does VR technology affect the residents’
engagement with the decision-making process
regarding the park?
RQ 2. Does immersive VR technology provide a cognitive
benefit in decision making compared to non-immersive
display technologies?
Display technologies available in the ballot were a paper map, a
smartphone, a tablet, a personal computer, or a VR headset. Except
for the paper map that showed 2D plans, all technologies used the
same 3D model to display the designs.
Relating to RQ 1, we define engagement from two perspectives: 1)
citizens’ engagement with the overall participatory decision-
making process as conducted by the municipality; 2) citizens’
engagement with the presentation of the three designs for the
park. The first perspective, regarding the overall participation
process, was measured during the ballot. The latter perspective,
engagement with the presentations, gives an indication of how
well the medium captures citizens’ attention during their personal
decision-making process. In addition to the experiment during the
ballot, we conducted a controlled experiment to quantify the
experiential differences between immersive and non-immersive
presentations.
Relating to RQ 2, the cognitive benefits were examined by
studying the citizens’ perception of the differences between the
variants designed for the park. During the ballot, we collected data
about the voters’ confidence about their perception. In the
controlled experiment, we investigated the actual perception by
measuring the participants’ recollection of what they had seen.
The paper is structured as follows. Section 2 discusses related
work and section 3 introduces the co-design process. Section 4
then discusses the first experiment, executed during the ballot, and
section 5 reports on the second experiment in the controlled
laboratory study. Sections 6 & 7 present our discussion and
conclusion respectively.
2 RELATED WORK
In the past few decades, the role of citizens in relation to
government has evolved considerably from political
representation towards active, civic participation in policy-
making by the end of the 20th century. The ‘participation society’
from the beginning of this century is now evolving into a ‘do-
democracy’ where citizens’ initiatives shape government
responses and actions [19]. De Waal discusses political-
philosophical perspectives on citizenship and how citizenship is
changing as a consequence of the introduction of smart city
technology. He defines the republican perspective, that combines
individual freedom with collective responsibilities, as opposed to
the libertarian perspective, that focuses on individual rights and
minimal mutual responsibilities [37,36].
Arnstein, already in the 1960’s, introduced the concept of a citizen
participation ladder, as a form of classifying as well as critiquing
different styles of governance and participation settings [3]. Foth
defines four stages of evolution of the relationship between city
government and citizens, with government in roles evolving from
administrator to collaborator and citizens evolving from residents
to co-creators [13]. Creighton defines participation as “the process
by which public concerns, needs, and values are incorporated into
governmental and corporate decision-making” [8]. Engaging
citizens to participate in public decision-making processes can
foster creativity and generate fresh ideas [23], while resulting in
strong citizen commitment to related changes [3,21].
Practice teaches that effective collaboration between citizens and
professional decision-makers requires a middle-out approach [15].
Top-down approaches tend to place decision-makers at the center
of the process and do not generally lead to genuine engagement.
Bottom-up initiatives, e.g. taken by community groups, may be
seen by decision-makers as illegitimate or substandard and
therefore get disregarded. A middle-out process integrates top-
down objectives with bottom-up interests and involves
stakeholders in all stages of the decision-making process.
Visual communication can effectively support processes of
collective decision-making and communicating about such
decisions. It can offer a common language in an easily accessible
medium and thus reduce barriers for public engagement, allowing
participants to become more literate about planning processes
[38]. 3D visualizations in urban design have long targeted
professional users, such as architects, urban planners, and
landscape designers [25,35], but are increasingly used for
participatory urban planning as well [1,2]. Visualizations in 3D
and VR are used in public consultation processes [20,39], allowing
the public to interactively access information in 3D models and
leave comments and suggestions for modifications. VR has been
Effectiveness of Virtual Reality in Participatory Urban Planning MAB'18, November, 2018, Beijing, China
demonstrated a suitable means, technologically, to facilitate public
participation in urban decision-making [20]. Visualizations also
offer opportunities to bring evidence into the decision-making
process, informing citizens about environmental issues [40] and
consequences for, e.g., daylight and shadow [9].
The present technology of smartphones allows for sufficiently
realistic VR experiences, making the technology suitable for large-
scale participatory design and assessment of design proposals
[5,22,27]. Experimental applications have suggested that VR and
interactive 3D visualizations on mobile devices help improve
citizens’ understanding and increase public engagement [17].
Immersive VR has been suggested to be beneficial for recalling
items seen during the virtual experience, especially in
multisensory VR environments [11,26]. As an example, Harman,
Brown, and Johnson [18] made an experiment to compare recall
ability between a VR headset and a computer monitor, and found
that people were able to remember more information if they were
using the VR headset. However, also contradictory evidence has
been presented [4,14], suggesting that increased immersion may
have a counterproductive effect for recall. Possible explanations
are limited cognitive capacity and mediated arousal, which have
been hypothesized to affect recall negatively [4].
3 CO-DESIGN PROCESS
In the beginning of the process, the municipality published a call
to the residents for attending an information session with
municipal experts. Over 70 residents attended the event, where
municipal experts explained possibilities and limitations of what
could be achieved in the park, both technically and budgetary. The
residents were invited to write up their ideas for the park, which
resulted in over 60 written suggestions.
A workgroup of 25 residents from the neighborhood was formed
in the event, including males and females from various ethnic
backgrounds and age groups, reasonably representative of the
neighborhood’s population. The workgroup collaborated with
municipal experts to generate three designs for the park in three
co-design sessions.
In the first session, the workgroup transformed the residents’
written suggestions into a comprehensive list of ideas. Based on
this list, the neighborhood manager made a set of puzzle pieces for
the different zones of the park. Characteristics for each zone were
derived from the list of ideas.
In the second session, the workgroup used the pieces of the puzzle
to create five compositions for the park. The workgroup members
individually prioritized the list of ideas by distributing 20 points
between the listed items. These activities enabled the workgroup
members to develop a vision for their ideas and choices. The group
then formed three teams that each created a design draft for the
park. The three drafts were then elaborated by a municipal
landscape architect to generate more detailed initial designs.
In the third co-design session, each team discussed their design for
the park (see Figure 1) and improved it using a digital 3D
modelling system, Sketchup, which was operated by an expert (see
Figure 2). This enabled the three teams to comment on and modify
the initial designs and instantly visualize their new ideas in the 3D
modelling environment.
Figure 1. One of the three teams discussing their design
using 2D plans.
Figure 2. One of the three teams discussing their design
with the landscape architect (left) and interacting with an
expert (sitting behind the computer) as he was modelling
their design in 3D.
The co-design sessions resulted in three variant proposals for the
revamp of the park. The differences between the three proposals
concerned the layout of the pathways in the park and the kind and
density of vegetation (e.g., flower beds, hedges). The differences
also concerned larger elements, such as playgrounds, activity
areas (e.g., skate park, basketball ground, fitness equipment), and
outdoor furniture. Two of the three proposals removed an existing
service building; one proposal added a new bandstand in the park.
4 EXPERIMENT 1: PUBLIC BALLOT
4.1 Method
With the three designs, a public ballot was conducted among the
residents of the neighborhood. The ballot period spanned 3 weeks.
The main aim of the municipality was to let residents vote for their
favorite among the three designs. This also provided a platform to
investigate the effects of using VR technology in the decision-
making process.
4.1.1 Participants
The municipality sent a written invitation for the ballot to all
residents of the area (approximately 13,500) and advertised it
online and through a poster campaign. As a result, 1302 residents
participated in the ballot. The voting was entirely anonymous and
MAB'18, November, 2018, Beijing, China J.P. van Leeuwen et al.
demographic data of the participants was not collected in order to
minimize any barriers for participating in the voting. The only
personal information recorded was postcode; this was recorded in
order to calculate the distance of the park from the home of the
resident.
4.1.2 Setup and Technology
Photorealistic 3D rendering was used to visualize the three
prospective design variants for the park. The rendering was done
for three fixed locations in the park and the rendering engine
supported navigation similar to Google Street View (see Figure 3),
i.e., supporting rotation of the point of view and discrete
transitions from one vantage point to another.
The 3D-rendered park variants were viewable with a range of
devices. A Web-based interface allowed participants to view the
designs on their personal computer, smartphone, or tablet. This
possibility was enabled to support remote voting, e.g., from home.
A voting-support team was active during the ballot period in over
twenty public locations in the neighborhood in and around the
park. The voting-support team presented the designs either with
a VR headset or using a 2D paper map detailing the design
variants. The paper map was used as a baseline in the study.
4.1.3 Procedure
The procedure was slightly different for participation through the
voting-support team vs. participating independently using
personal devices. With the voting-support team, the three variants
were presented in a fixed order (A-B-C), whereas for unassisted
voting the participants were free to display the variants in any
order as many times as they wanted on their own device. The team
assisted in two types of voting sessions, offered to residents as a
choice: using paper or using the VR headset. The team was
instructed to guide the viewing of the three variants, by operating
the VR headset for navigation between vantage points and
variants, and by answering questions that residents might have.
An assisted voting session lasted on average approximately 6
minutes using the VR headset and 3 minutes using the paper map.
After viewing the variants, either unassisted or assisted by the
team, the residents could vote for their preferred proposal. Voting
took place through a form on the district’s official website, where
voters could first view the three variants – using their preferred
device – and then cast their vote. The voting-support team also
carried tablets to allow residents to vote immediately after viewing
the designs via the VR headsets or on paper (see Figure 4).
In addition to the voting, the participants were asked to answer a
short questionnaire after reviewing all the variants. The
questionnaire contained questions related to the confidence of the
vote, perceiving differences between the design variants,
importance of the park for the voter, and willingness to
recommend the voting to others. It was a strategic decision to keep
the questionnaire short in the ballot in order to maximize the
number of responses in the participatory decision-making process.
4.1.4 Data
Table 1 summarizes the main data collected in the ballot. Voters’
willingness to recommend others to vote was used to determine
the Net Promoter Score (NPS) for the voting procedure. The NPS
is a measure, often used in marketing research, for customers’
engagement with a brand or product [32]. We used this measure
to determine citizens’ engagement with the overall decision-
making process regarding the park, as initiated by the
municipality.
The questionnaire also gave the voter the opportunity to write a
motivation with their vote, which informed the district officials
about the perceived pros and cons of the three variants for the
park.
Figure 3. Visualization and navigation of the designs
as shown on computer screens.
Figure 4. Residents viewing the variants of the park using
the VR headset (left) and paper plans (right), with the
voting-support team.
Variable
Scale
Confidence of the vote
0 – 10 (not at all confident –
100% certain)
Confidence about having
perceived the differences
between the variants
0 – 10 (very few differences –
differences very clear)
The importance of the park to
the voter
0 – 10 (not at all important –
very important)
The voter’s willingness to
recommend others to vote
0 – 10 (not at all willing –
certainly willing)
Table 1. Main variables for the questionnaire in the ballot.
Effectiveness of Virtual Reality in Participatory Urban Planning MAB'18, November, 2018, Beijing, China
4.2 Results
4.2.1 Engagement with the process (RQ1)
At the end of the voting period, 1302 residents had submitted valid
votes and completed forms. In earlier work, the research results
regarding the NPS were published [24] and showed that the
engagement with the voting-process in this case was in line with
the average appreciation for municipalities in The Netherlands.
No correlation was found between the importance of the park to
voters and their engagement with the process. The data showed
that voters using the 2D paper maps were significantly less willing
to ask others to vote as well, compared to users of smartphones
and users of the VR headset. Voters using their own computer
were significantly less willing to recommend than smartphone
users, but no significant difference was found between the VR
headset and the smartphone [24].
These results from the ballot questionnaire answer RQ1 from the
first perspective we mentioned earlier: regarding citizens’
engagement with the overall decision-making process. The second
perspective, citizens’ engagement with the presentation, using VR
as opposed to other display techniques, required an additional,
more detailed and controlled study in experiment 2.
4.2.2 Cognitive benefits (RQ2)
Analyzing the data from the ballot questionnaire, the voters’
confidence of their vote did not relate to the vote they casted (for
design variant A, B, or C), nor to the device they used to view the
variants. Voters’ confidence of being able to perceive the
differences between the variants for the park was also not related
to their actual vote.
(I) Device
(J) Device
Mean
Difference
(I-J)
Std.
Error
Sig.
95%
Confidence
Interval
Paper
Computer
.568
.251
.157
-.12
1.25
Tablet/iPad
.764
.354
.198
-.21
1.73
Smartphone
1.342*
.255
.000
.65
2.04
VR headset
-.200
.228
.905
-.82
.42
VR
headset
Paper
.200
.228
.905
-.42
.82
Computer
.768*
.206
.002
.21
1.33
Tablet/iPad
.964*
.324
.025
.08
1.85
Smartphone
1.543*
.211
.000
.97
2.12
*. The mean difference is significant at the 0.05 level.
Table 2. Mean differences for the variable
‘Differences perceived’ in the ballot.
There were significant differences between the devices used, with
respect to the voters’ self-reported ability to see the differences
between the variants (see Table 2). Voters that viewed the plan
with the VR headset or with the 2D paper maps – thus voters that
were approached and assisted by the voting-support team – were
significantly more confident of having been able to see the
difference than voters viewing the plans unassisted using their
own devices. The assistance by the voting-support team, that was
instructed to answer questions about the different designs, seems
to be the factor of importance here, not the type of device used for
viewing the designs. The mean difference for the variable
‘Differences perceived,’ between assisted and unassisted voting
sessions was relevant and significant at -1.056, BCa 95% CI [-1.360,
-.752], p = .000.
Regarding the perceived differences between the three designs,
the short questionnaire in the ballot only addressed the voters’
confidence about perceiving the differences. Further study was
needed to examine the actual perception of differences, which we
did in experiment 2.
5 EXPERIMENT 2: LABORATORY STUDY
5.1 Method
Due to the unassisted voting opportunity, it was not possible to
control the contextual circumstances during the ballot, which
might have had an effect on the results. A laboratory study was
conducted to assess in detail the differences between immersive
and non-immersive VR regarding perceived differences between
the presented designs and engagement with the presentation
medium. The controlled study was conducted in two public
locations in the city.
5.1.1 Participants
76 participants (32 female) were recruited for the experiment. The
participants were recruited randomly from the location of the
experiment. 42 of these participants used the VR headset to display
the park designs, while the other 34 viewed the designs on a laptop
computer. None of the participants had participated in the public
ballot.
5.1.2 Setup and Technology
The same VR headset that was used in the ballot was used in the
laboratory study. For comparison, a laptop computer was used to
display the 3D rendering on a 2D screen. Other devices such as
smartphones, tablets, and paper maps were not addressed in the
laboratory study. The main aim was to compare an immersive VR
headset with a non-immersive representation, and it was deemed
sufficient to do this comparison between the headset and a
conventional laptop screen.
The study was conducted in two locations in the city, relatively far
from the park itself in order to eliminate the importance of and
familiarity with the park from the experiment. The locations were
selected to be public indoor spaces with other people present in
the environment. The purpose of having background noise and
activity around the participant was to study the immersive quality
of the VR headset as well as potential distraction effects.
5.1.3 Procedure
The procedure was similar to that in the ballot with the voting-
support team. Each participant was instructed to view the
different designs either with the VR headset or with a laptop. For
the VR headset, the experiment conductor controlled the vantage
point in a fixed order. The participant was allowed to view the VR-
rendered park from each vantage point as long as they wanted and
MAB'18, November, 2018, Beijing, China J.P. van Leeuwen et al.
were encouraged to rotate their head to see the entire park. When
they were ready to move to the next vantage point and
subsequently to the next variant design, they expressed this to the
experiment conductor. Participants who viewed the designs on a
laptop were instructed to follow the same fixed order, but this was
not enforced and these participants switched more freely between
the vantage points and variants. The experiment conductor did not
offer any further clarifications or answers to questions about the
park design variants.
After the viewing, the participants were presented with an
extended questionnaire. The questionnaire was tuned to measure
memory and recall of objects in the park and differences between
the variants. The first part of the questionnaire asked the
participants to write down as many objects as they remembered
seeing (Free recall). The second part contained questions about the
voting experience and the voting itself. The third part presented
the participants with a checklist of 22 items (verbally described,
e.g., ‘a gazebo’); the task was to check whether the participant had
seen each item in variant A, B, and/or C (Recall accuracy).
Variable
Scale
Free recall
number of objects listed by the
participant
Recall accuracy
0 – 10 (10 = perfect recall)
Confidence of the vote
1 – 5 (not confident at all – 100%
certain)
Confidence about having
perceived the differences
between the variants
1 – 5 (very few differences –
differences very clear)
Immersion in the 3D
environment
1 – 5 (not immersed at all – very
immersed)
Translocation to the park
1 – 5 (no translocation at all – felt
like in the park completely)
Concentration
1 – 5 (not concentrated at all –
completely concentrated)
Table 3. Main variables for the laboratory study.
5.1.4 Data
Table 3 presents an overview of the main data collected during the
laboratory study. The participants’ capability to remember what
they had seen was measured in two ways: (1) by counting the
number of objects they were able to write down (Free recall); and
(2) by a Recall accuracy score calculated from the checklist of 22
items. This score is representative for how accurately the
participant remembered the presence of each item in each of the
variants. Mistakes, such as remembering the item but not the
variant in which it was present, led to a lower score. A score of 10
represents perfect recall of all items in all variants (which never
occurred).
The questionnaire also included questions for measuring the level
of immersion experienced by participants and their feeling of
having been translocated to the park as opposed to being at the
experimentation location. Finally, the questionnaire asked
participants how well they could concentrate on viewing the park
designs.
5.2 Results
5.2.1 Engagement with the decision-making (RQ1)
The data from the lab study clearly shows that participants were
significantly more engaged by the VR experience than by
navigating the 3D renderings on the computer screen. For each of
the three variables ‘Immersion,’ ‘Translocation,’ and
‘Concentration,’ there is a significant advantage for the VR
headset (see Table 4). None of these variables could be related to
gender or age groups (under 25, 25-49, 50+).
Variable
Mean difference
Immersion
-.697, 95% CI [-1.250, -.145], p = .014
Translocation
-1.755, 95% CI [-2.326, -1.184], p = .000
Concentration
-.821, 95% CI [-1.243, -.398], p = .000
Table 4. Mean differences between Laptop and VR headset,
for variables Immersion, Translocation, and
Concentration.
Regarding session duration, there was no significant difference
between the VR headset and the computer (299 and 275 seconds,
respectively). On average, the sessions with the laptop were not
significantly overestimated, but those with the VR headset were.
Interestingly, the overestimation was found only in male
participants: the mean overestimation of the VR session duration
by males was 32.7% CI [15.214, 51.026], p = -.024; by females it was
3.7% and not significant.
In terms of engagement, the results are clear: in all the
engagement dimensions, the VR headset outperformed the laptop
computer.
5.2.2 Cognitive benefits (RQ2)
The laboratory study confirmed that the VR headset did not
significantly induce more self-reported confidence in the
participants’ decision, compared to the laptop. Participants’
confidence of being able to perceive the differences between the
variants for the park was also not related to their actual vote.
No significant mean difference in the recall accuracy was observed
between the laptop and the VR headset. However, the mean of the
number of objects participants listed in the free recall task was 6.24
for the laptop and 8.02 for the VR headset. The mean difference is
significant: -1.789, BCa 95% CI [-3.237, -.340], p = .016. In other
words, compared to using a laptop for viewing the 360˚ images of
the park, the VR headset allowed participants in the lab
experiment to better recount what they had seen, but when
measuring their recall accuracy in the recall test, there is no
significant difference between the two devices.
6 DISCUSSION
The results of this research provide insight into the effectiveness
of VR technology in involving the larger public in participatory
urban planning. Regarding the first research question about
citizens’ engagement with the decision-making process, two
Effectiveness of Virtual Reality in Participatory Urban Planning MAB'18, November, 2018, Beijing, China
conclusions can be drawn. First, in the co-design process, the
application of 3D modeling was instrumental for the workgroup
to see and reflect upon their designs, resulting in instant design
iterations. Instant visualization in 3D brings ideas to life and fuels
creativity, both in professional designers/planners and untrained
participant citizens, particularly so when designs can be
interactively changed during the co-design sessions. The activity
of visualization helps participants to assess and reflect deeper on
the spatial properties and qualities of their ideas. It facilitates
comparisons of alternative designs and places these, literally, into
the larger urban context. Being able to immediately review design
alternatives in 3D contributed positively to the engagement of the
workgroup.
Second, the results of the controlled laboratory study indicate that
the participants using immersive VR experienced higher levels of
engagement than the participants with non-immersive VR when
viewing alternative designs for the park. This result is very
important for the municipality in deciding whether to invest in
using immersive VR technology to stimulate citizen participation:
engaged voters can be argued to be less prone to external
distractions during the decision making and to be more committed
to make informed decisions. We can conclude that using 3D
rendering and VR technologies is fruitful to enhance civic
participation, as previously demonstrated [24] and discussed also
by Gill and Lange [17].
The data analysis showed a significantly higher engagement in
voters that used a Smartphone to view and assess the designs. This
may either indicate that the device used influences the
engagement level or that Smartphones users are more likely to
engage with local policy. Both possibilities are hard to substantiate
from the research results. The data shows no indications that the
outcome of the voting was biased by the device used to view and
assess the designs.
Interestingly, regarding the second research question about
cognitive benefits of immersive VR, increased engagement did not
result in higher confidence of the vote nor better ability to account
for differences between competing designs. The results of the
laboratory study show that people who used the VR headset were
able to remember more items in the park than people who used
the laptop. There was, however, no difference in the recall
accuracy between the two devices. This indicates that viewing the
designs on the VR headset resulted in more vivid, but not more
accurate, memory than viewing the designs on the laptop.
Considering the fact that the VR headset outperformed the laptop
in terms of immersion, translocation, and concentration, it seems
that the higher level of immersion with the VR headset might have
helped the participants with the free recall of items, which is in
line with previous results [18]. However, based on the results,
immersion was not helpful regarding the recall accuracy. The list
of 22 items to recall was, admittedly, rather long, so the high
cognitive demand of the task combined with the immersive
experience might have hindered the recall accuracy [4,14].
The controlled experiment corroborates the finding from the
ballot regarding the effect of the voting-support team on
perceiving differences between different designs. In the ballot, the
VR headset and paper maps were used with the voting-support
team, who were not only assisting the voting but were also
available to answer any questions the voters had. It is possible that
this discussion influenced the confidence in perceiving differences
between the designs. Since no significant difference between the
VR headset and the laptop was found in the controlled experiment,
the assistance by the team is a probable explanation for these
differences in the ballot. This result suggests that while
technology can be utilized for informed decision making in a
participatory process, interacting with municipal representatives
during the decision making can provide further insight into the
decision to be made.
6.1 Limitations
There are four main methodological limitations in the study, First,
as discussed above, the presence of the voting-support team
during the ballot seemed to influence the comparison between the
technologies. While this can be seen as biasing the voters in terms
of their confidence in perceiving the differences between the
designs, it is important to remember that the main aim of the
municipality was to get as many votes in as possible. We rectified
this bias by conducting the controlled laboratory experiment,
which eliminated the effect of the voting-support team from the
comparison.
Second, the questionnaire during the ballot was limited in scope
due to the aim of easy voting. This implied that we were not able
to include questions related to immersion, demographics, and
details of perceived differences between the designs. While this
data could have provided further insight into how people
perceived the VR technology in the voting, we were still able to
sufficiently investigate immersion and perceived differences in the
controlled experiment.
Third, to simplify the viewing of the three designs with the VR
headset, they were always presented in the same order (A-B-C).
Thus, the results do not account for possible order effects. We did
not control in which order people viewed the variants in the
unassisted voting, nor how many times they looked at each variant
from each vantage point. However, this approach did not have a
statistically decisive influence on the outcome of the ballot itself
and it is unlikely that it had an effect on the key measures of our
study. Willingness to recommend the voting, perceived
differences between the variants, and level of immersion can be
argued to be relatively robust against order effects.
Finally, while qualitative data was collected through interviews
and observations during the sessions of the workgroup in phase 2
of the participation process, the research presented in this paper
focused solely on collecting quantitative data in the public ballot
in phase 3. Interactions between the voting-support team and the
public were observed, but not methodically enough to allow
conclusions to be drawn.
7 CONCLUSION
The presented case study investigated the effectiveness of VR
technology in participatory urban planning, in the context of a
municipal decision-making process. The results suggest that there
are several benefits in using VR headsets with 3D rendering in
such a process. First, immersive VR provides higher engagement
MAB'18, November, 2018, Beijing, China J.P. van Leeuwen et al.
than using 2D presentation technologies, indicating stronger
potential in eliciting participation. Second, immersive VR results
in more vivid memory of the viewed content than computer
monitors, which may be important in terms of making informed
decisions. Third, the effect of human interaction should not be
neglected in decision-making processes harnessing engaging
technologies. Finally, the experiments provide the kind of
evidence that municipal governments need to decide about
investments and the design of participatory planning processes.
The participatory design project had an important social impact
on the neighborhood. The resulting design closely reflects the
needs and wishes of the inhabitants near the park. The
municipality’s decision to involve all inhabitants, in response to
discontentment about the park, has also contributed to the
relationship between the neighborhood and the municipal
government. Citizens showed an increased and genuine interest in
the local urban development and felt invited to partake in the
decision-making. In this way, the renewed park and the process of
its realization contribute to the social cohesion in the
neighborhood and a shared responsibility to maintain its social
and physical qualities. Did the participatory design and the voting
procedure lead to a better design for the park? In the opinion of
the municipal officials involved, the actual value in a project like
this is determined not merely by the quality of the resulting
design, but by the quality of the public support for its outcome.
The social and managerial impact of the participation project is
discussed in more detail in earlier work [24].
Due to the nature of the municipal decision-making process, there
were some methodological limitations in the large-scale field
study during the ballot. These limitations were rectified through
an additional controlled experiment to ensure experimental
validity. The field study and the controlled experiment
complemented each other and resulted in findings that could not
have been obtained without conducting both experiments.
Therefore, we can conclude that when conducting research as an
additional component in a design process of a municipality, it is
important to conduct additional experiments to shed more light
on the obtained insights.
7.1 Future work
From our observations and evaluations with the voting-support
team, we learned that many voters said, after voting, that they
would like to be able to participate in a more nuanced manner than
just choosing between the three options given. They had
suggestions to make and ideas to explore for modifications of the
proposed designs. Ball et al. [5] stress that mutual understanding
between planners and stakeholders is a prerequisite for successful
participatory design. For mutual understanding to happen, there
must be a public dialogue about urban plans, stakeholder needs,
and consequences of design decisions [5]. Facilitating such
dialogues and co-design activities for the larger public calls for a
much subtler approach, where the effectiveness of visual
communication is combined with tools for dialogue, design
exploration, collective construction of meaning, and of shared
understanding. Future work on this topic would have to address
the design of such tools and to explore the delicate balance
between meaningful interactions and public accessibility of what
could potentially become a very complex collaborative design
environment. Prior to this kind of development, more qualitative
research is needed to gain insight into the social factors that bind
participants, both professional and untrained, in co-design
projects.
The increasing availability of VR and AR technology, including on
common Smartphones, is lowering the threshold for setting up
and managing this technology in projects with public
participation. For municipalities it is essential to increase the level
of citizen-engagement in decision-making processes and media
technology proves to be a very appealing way to do this. The
municipalities’ interest in using VR and other digital media for this
purpose, leads to two speculations regarding media architecture.
First, in addition – or alternative – to VR, other digital media can
be utilized to facilitate citizen engagement and offer valuable
capabilities. For instance, using public displays as discussed by,
e.g. [12,28–31,33,34], may be suitable to prompt public debates
about alternatives in decision-making processes. Second,
participatory processes will play an increasing role in decision-
making, also regarding the design and realization of media
installations in public spaces. The process presented in this paper
can be applied in such cases and our conclusions may help
optimize the social and managerial effects and the acceptance of
the outcome.
REFERENCES
1. Kheir Al-Kodmany. 1999. Using visualization techniques for
enhancing public participation in planning and design:
process, implementation, and evaluation. Landscape and Urban
Planning 45, 1: 37–45.
2. Kheir Al-Kodmany. 2001. Visualization Tools and Methods for
Participatory Planning and Design. Journal of Urban
Technology 8, 2: 1–37.
3. Sherry R. Arnstein. 1969. A Ladder of Citizen Participation.
Journal of the American Planning Association 35, 4: 216–224.
4. Jakki Bailey, Jeremy N Bailenson, Andrea Stevenson Won,
June Flora, and K Carrie Armel. 2012. Presence and Memory:
Immersive Virtual Reality Effects on Cued Recall. Proceedings
of the International Society for Presence Research Annual
Conference: 24–26.
5. J Ball, N Capanni, and S Watt. 2007. Virtual Reality for Mutual
Understanding in Landscape Planning. International Journal of
Social, Behavioral, Educational, Economic, Business and
Industrial Engineering 1, 11: 661–671.
6. Erling Björgvinsson, Pelle Ehn, and Per-Anders Hillgren. 2010.
Participatory design and “democratizing innovation.” In
Proceedings of the Participatory Design Conference - PDC ’10,
41–50.
7. Glenda Amayo Caldwell and Marcus Foth. 2014. DIY media
architecture: Open and participatory approaches to
community engagement. In Proceedings of the 2014 Media
Architecture Biennale, 1–10.
8. J.L. Creighton. 2005. The public participation handbook: Making
Better Decisions Through Citizen Involvement. John Wiley &
Sons, Inc., Hoboken, NJ.
9. Jens Dambruch and Michel Krämer. 2014. Leveraging public
participation in urban planning with 3D web technology.
Proceedings of the Nineteenth International ACM Conference on
3D Web Technologies - Web3D ’14: 117–124.
10. Christopher A. le Dantec and Carl DiSalvo. 2013.
Effectiveness of Virtual Reality in Participatory Urban Planning MAB'18, November, 2018, Beijing, China
Infrastructuring and the formation of publics in participatory
design. Social Studies of Science 43, 2: 241–264.
11. H.Q. Dinh, N. Walker, L.F. Hodges, Chang Song, and A.
Kobayashi. 1999. Evaluating the importance of multi-sensory
input on memory and the sense of presence in virtual
environments. In Proceedings IEEE Virtual Reality 1999, IEEE,
Houston, TX, 222–228.
12. Marcus Foth, Martin Tomitsch, Laura Forlano, M. Hank
Haeusler, and Christine Satchell. 2016. Citizens breaking out
of filter bubbles. In Proceedings of the 5th ACM International
Symposium on Pervasive Displays - PerDis ’16, 140–147.
13. Marcus Foth. 2017. Participation, Co-Creation, and Public
Space. The Journal of Public Space 2, 4: 21–36.
14. Jesse Fox, Jeremy Bailenson, and Joseph Binney. 2009. Virtual
experiences, physical behaviors: The effect of presence on
imitation of an eating avatar. Presence: Teleoperators and
Virtual Environments 18, 4: 294–303.
15. Joel Fredericks, Glenda Amayo Caldwell, and Martin
Tomitsch. 2016. Middle-out design: collaborative community
engagement in urban HCI. In Proceedings of the 28th Australian
Conference on Computer-Human Interaction, 200–204.
16. Gemeenteraad Den Haag. 2012. Inspraak- en
participatieverordening gemeente Den Haag 2012 (The Hague
municipal bill on civic participation, in Dutch). Gemeente Den
Haag.
17. Lewis Gill and Eckart Lange. 2015. Getting virtual 3D
landscapes out of the lab. Computers, Environment and Urban
Systems 54: 356–362.
18. Joel Harman, Ross Brown, and Daniel Johnson. 2017. Improved
Memory Elicitation in Virtual Reality: New Experimental
Results and Insights. In INTERACT 2017, Springer, 128–146.
19. Pepijn van Houwelingen, Anita Boele, and Paul Dekker. 2014.
Burgermacht op eigen kracht? Een brede verkenning van
ontwikkelingen in burgerparticipatie (in Dutch). Sociaal en
Cultureel Planbureau, Den Haag.
20. T.L.J. Howard and N. Gaborit. 2007. Using Virtual
Environment Technology to Improve Public Participation in
the Urban Planning Process. Journal of Urban Planning and
Development 133, 4: 233–241.
21. Helena Karasti. 2014. Infrastructuring in participatory design.
In Proceedings of the Participatory Design Conference - PDC ’14,
141–150.
22. S. F. Kuliga, T. Thrash, R. C. Dalton, and C. Hölscher. 2015.
Virtual reality as an empirical research tool - Exploring user
experience in a real building and a corresponding virtual
model. Computers, Environment and Urban Systems 54: 363–
375.
23. Antje Kunze, Jan Halatsch, Carlos Vanegas, Martina Maldaner
Jacobi, Benamy Turkienicz, and Gerhard Schmitt. 2011. A
Conceptual Participatory Design Framework for Urban
Planning. In Proceedings of eCAADe 2011: Respecting Fragile
Places, Ljubljana, Slovenia, 895–903.
24. Jos P. van Leeuwen, Klaske Hermans, Arnold Jan Quanjer,
Antti Jylh
ä
, and Hanke Nijman. 2018. Using Virtual Reality to
Increase Civic Participation in Designing Public Spaces. In
Proceedings of the European Conference on Digital Government
2018, Santiago de Compostela, Spain.
25. Jos P. van Leeuwen and Harry J.P. Timmermans (eds.). 2006.
Innovations in Design & Decision Support Systems in
Architecture and Urban Planning. Springer, Dordrecht, NL.
26. James J.W. Lin, Henry B.L. Duh, Donald E. Parker, Habib Abi-
Rached, and Thomas A. Furness. 2002. Effects of Field of View
on Presence, Enjoyment, Memory, and Simulator Sickness in a
Virtual Environment. In Proceedings IEEE Virtual Reality
2002(VR), IEEE, Orlando, FL, 164.
27. Luigi Maffei, Massimiliano Masullo, Aniello Pascale, Gennaro
Ruggiero, and Virginia Puyana Romero. 2016. Immersive
virtual reality in community planning: Acoustic and visual
congruence of simulated vs real world. Sustainable Cities and
Society 27: 338–345.
28. Nemanja Memarovic, Ivan Elhart, Andrea Michelotti, Elisa
Rubegni, and Marc Langheinrich. 2013. Social Networked
Displays: Integrating Networked Public Displays with Social
Media. In Proceedings of the International Joint Conference on
Pervasive and Ubiquitous Computing - UbiComp’13, Zurich,
Switzerland, 55–58.
29. Nemanja Memarovic, Marc Langheinrich, Florian Alt, Ivan
Elhart, Simo Hosio, and Elisa Rubegni. 2012. Using public
displays to stimulate passive engagement, active engagement,
and discovery in public spaces. In Proceedings of the 4th Media
Architecture Biennale Conference on Participation - MAB ’12,
Aarhus, Denmark, 55–64.
30. Nemanja Memarovic, Marc Langheinrich, and Ava Fatah gen.
Schieck. 2014. Community is the Message - Viewing
Networked Public Displays Through McLuhan’s Lens of
Figure and Ground. In Proceedings of the Media Architecture
Biennale - MAB’14, Aarhus, Denmark, 30–33.
31. Timo Ojala, Vassilis Kostakos, Hannu Kukka, Tommi
Heikkinen, Tomas Lind
é
n, Marko Jurmu, Simo Hosio, Fabio
Kruger, and Daniele Zanni. 2012. Multipurpose Interactive
Public Displays in the Wild: Three Years Later. Computer: 42–
49.
32. FF Reichheld. 2003. The one number you need to grow.
Harvard business review 81, 12: 46–55.
33. Mettina Veenstra, Niels Wouters, Marije Kanis, Stephan
Brandenburg, Kevin te Raa, Bart Wigger, and Andrew Vande
Moere. 2015. Should Public Displays be Interactive? Evaluating
the Impact of Interactivity on Audience Engagement. In
Proceedings of the 4th International Symposium on Pervasive
Displays - PerDis ’15, Saarbruecken, Germany, 15–21.
34. Leena Vent
ä
-Olkkonen, Marianne Kinnula, Graham Dean,
Tobias Stockinger, and Claudia Zu
ñ
iga. 2013. Who’s There?
Experience-Driven Design of Urban Interaction Using a
Tangible User Interface. In Proceedings of the 12th Int’l Conf. on
Mobile and Ubiquitous Multimedia - MUM ’13, Lule
å
, Sweden.
35. Bauke de Vries, Jos van Leeuwen, and Henri Achten (eds.).
2001. Computer Aided Architectural Design Futures 2001.
Kluwer Academic Publishers, Dordrecht, NL.
36. Martijn de Waal. 2014. The City as Interface. Nai010 Publishers,
Rotterdam, NL.
37. Martijn de Waal and Marloes Dignum. 2017. The citizen in the
smart city. How the smart city could transform citizenship.
Information Technology 59, 6: 263–273.
38. Sarah Williams. 2016. Data Visualizations Break Down
Knowledge Barriers in Public Engagement. In Civic Media:
Technology, Design, Practice, Eric Gordon and Paul Mihailidis
(eds.). The MIT Press, Cambridge, MA, 165–197.
39. Ulrike Wissen Hayek. 2011. Which is the appropriate 3D
visualization type for participatory landscape planning
workshops? A portfolio of their effectiveness. Environment
and Planning B: Urban Analytics and City Science 38, 5: 921–
939.
40. Ulrike Wissen Hayek, Noemi Neuenschwander, and Adrienne
Grêt-Regamey. 2012. Facilitating well-informed trade-off
decision making on land use change: Integrating rules and
indicators of ecosystem service provision into procedural 3D
visualization. In 6th International Congress on Environmental
Modelling and Software - Leipzig, Germany, 2235–2242.
