Content uploaded by William Shelstad
Author content
All content in this area was uploaded by William Shelstad on Sep 27, 2018
Content may be subject to copyright.
Gaming on the Rift: How Virtual Reality Affects
Game User Satisfaction
William J. Shelstad, Dustin C. Smith, Barbara S. Chaparro
Wichita State University
Virtual reality (VR) headsets like the Oculus Rift, HTC Vive, and PlayStation VR can
provide a unique experience different from traditional computer monitors. Research
demonstrates some support for VR resulting in more immersive gaming than traditional
games using a computer or TV monitor. This study investigates how VR technology
impacts game user satisfaction. Participants played the same strategy video game using
the Oculus Rift, a VR headset, and a computer monitor. Game user satisfaction was
measured by the psychometrically validated Game User Experience Satisfaction Scale
(GUESS) which consists of nine constructs. Results from this study showed that VR
enhanced overall satisfaction, enjoyment, engrossment, creativity, sound, and graphics
quality.
INTRODUCTION
An estimated 30.4 billion in consumer
spending has gone toward the video game industry
in 2016 (ESA, 2017). According to the
Entertainment Software Association (2017) 65% of
United States households are home to at least one
person who plays games regularly (3 hours or more
per week) and about 67% of U.S. households own a
dedicated game console. The age of an average
video game player is 35 and 37% of game players
are female. Clearly, there is a large and diverse
audience of people that are willing to buy and play
video games.
PC gaming has evolved over time with 92%
of game purchases being through digital distribution
in 2013 (Chalk, 2014). Steam, for example, one of
the largest digital distribution platforms for video
games, released 2,964 games in 2015 and 4,240
games in 2016 (Sarkar, 2016). One explanation for
the higher number of new game releases is due to
multiple virtual reality (VR) headsets that have
recently been released. About 16% of the video
games released on Steam in 2016 support a VR
headset and 83% of those that support VR require a
VR headset to play (Sarkar, 2016).
Virtual Reality Gaming Research
In the past year, multiple VR headsets have
been released including the Oculus Rift, HTC Vive,
and PlayStation VR. Researchers have examined
how VR headsets affect the gaming experience of
video game players. Game user experience of
participants that played the videogame Half-Life 2
using a traditional computer monitor and the Oculus
Rift DK1 was compared (Tan, Leong, Shen,
Dubravs, & Si, 2015). After gameplay, participants
completed surveys investigating immersion and
flow. Most participants reported a more ‘intense’
experience playing on the Oculus Rift and reported
higher scores of immersion and flow than playing
on the computer monitor. Martel, Su, Gerroir,
Hassan, Girouard, and Muldner (2015) examined
how head tracking as a game input method in VR
would affect immersion and performance.
Different
control schemes were compared with varying levels
of control using VR and a non-VR control scheme.
Participants completed surveys after using each
control scheme and all VR control schemes were
rated higher in immersion. Performance measures
for kills and fewest errors during gameplay were
highest in the non-VR control.
What is a Good Game and How to Measure it?
Video game developers are tasked with
creating games that are ‘better’ than the
competition. What is considered a “better” or
“good” game is debatable. Certainly, concepts like
enjoyment, ease of use, and the ability to make
interesting choices are considered to be necessary in
a game (Shelley, 2001, Totilo, 2012). In some
cases, players report a higher sense of reward when
they are able to make their own choices in a game,
especially when it is considered a good choice.
Copyright 2017 by Human Factors and Ergonomics Society. DOI 10.1177/1541931213602001
Proceedings of the Human Factors and Ergonomics Society 2017 Annual Meeting 2072
Playtesting (Davis, Steury, & Pagulayan,
2005; Fulton, 2002) is a common method used to
evaluate games during development. Playtesting is
similar to usability testing but tends to be more free
form and usually includes a questionnaire to get
feedback from the player post play. There are a
variety of survey instruments that evaluate different
aspects of video games like engagement,
immersion, or flow. Many of the scales are limited
in that they may only measure one or two aspects
that contribute to game user satisfaction (See Phan,
Keebler & Chaparro, 2016 for a comprehensive
review of the strengths and limitations of current
gaming scales). The Game User Experience
Satisfaction Scale (GUESS) is a recently developed
comprehensive and psychometrically validated
game scale that includes nine subscales contributing
to video game satisfaction (Phan et al., 2016). Table
1 lists a short description of each subscale. The
question arises if gaming using VR results in a
better overall experience for players, especially as it
compares to traditional play on a computer monitor.
Using the GUESS, key constructs can be compared
across environments to better understand the impact
of VR to the gaming experience.
While there has
been research into how VR headsets affect different
aspects of video game experience, there has not yet
been research using a psychometrically validated
scale like the GUESS to evaluate how VR affects
game user satisfaction. It is expected that if VR
provides a richer, more immersive experience then
GUESS scores should vary for a game played using
VR and no VR. Based on previous literature, we
predicted that participants would rate the VR
version of the game higher in engrossment (Tan et
al., 2015). Higher ratings for visual aesthetics for
the VR version also were predicted because the
Oculus provides a higher quality 3D environment
for users to see while playing. The impact of VR on
other GUESS scales such as ease of use, enjoyment,
creative freedom, social, audio aesthetic or narrative
are unknown.
Current Study
The current study examines how the use of
VR technology affects user satisfaction of a video
game. Specifically, player satisfaction was assessed
using the GUESS after playing the same game on
the Oculus Rift and a traditional computer monitor.
Table 1: A Short Description of GUESS Subscales
Subscale Description
Usability/Playability The ease in which the game can be played with
clear goals/objectives in mind and with minimal
cognitive interferences or obstructions from the
user interfaces and controls
Narratives The story aspects of the game (e.g., events and
characters) and their abilities to capture the
player’s interest and shape the player’s emotions
Play Engrossment The degree to which the game can hold the
player’s attention and interest
Enjoyment The amount of pleasure and delight that was
perceived by the player as a result of playing the
game
Creative Freedom The extent to which the game is able to foster the
player’s creativity and curiosity and allows the
player to freely express his or her individuality
while playing the game
Audio Aesthetics The different auditory aspects of the game (e.g.,
sound effects) and how much they enrich the
gaming experience
Personal Gratification The motivational aspects of the game (e.g.,
challenge) that promote the player’s sense of
accomplishment and the desire to succeed and
continue playing the game
Social Connectivity The degree to which the game facilitates social
connection between players through its tools and
features
Visual Aesthetics The graphics of the game and how attractive they
appeared to the player
Note. A Short Description of Each Subscale.
Reprinted from “The Development and Validation
of the Game User Experience Satisfaction Scale
(GUESS),” by M. H. Phan, J. R. Keebler, and B. S.
Chaparro, 2016, Human Factors, 58, p. 1238.
Copyright 2016 by Human Factors and Ergonomics
Society. Reprinted with permission.
METHOD
Participants
Forty undergraduate students were recruited
using the university’s online research pool. The
participants consisted of 16 males and 24 females
with ages ranging from 18 to 40 (M = 21.53, SD =
4.32). One participant owned a smartphone VR
headset. Six participants have used a VR headset
before and five of those have played games on a VR
headset. Only one participant had used the Oculus
Rift.
Measures
Game User Experience Satisfaction Scale (GUESS).
The GUESS consists of 55 questions rated
with a 7-point Likert scale (1 = Strongly Disagree to
Proceedings of the Human Factors and Ergonomics Society 2017 Annual Meeting 2073
7 = Strongly Agree). Subscale scores are calculated
by averaging the items in that scale. The overall
score is calculated by summing the average score of
each subscale together for a total possible score of
63.
Materials
DG2: Defense Grid 2 and Defense Grid 2:
Enhanced VR Edition.
Defense Grid 2 is a tower-defense strategy
game developed by Hidden Path Entertainment.
Players view the game from a top-down perspective
as they take on the role of a military commander
defending a base from waves of aliens. Players
acquire resources over time and from killing aliens.
Resources are used to build towers that shoot at the
aliens as they approach their base. There are
different types of towers like gun or laser, each with
different strengths and weaknesses. In addition,
there are a variety of aliens; some are very slow and
difficult to kill while others are weak and fast.
Towers can be upgraded to improve strength and to
adjust the aliens’ path. For example, one level
provides the player the option to divert aliens’ path
with tower placement and another provides the
player a space to build towers that creates the path
the aliens take.
Participants played the game on a 24-inch
computer monitor and the consumer version of the
Oculus Rift VR headset.
The Oculus Rift is a VR
headset that has features like stereoscopic 3D,
positional tracking, an accelerometer, and a
gyroscope (Nield, 2016). The Oculus Rift provides
high graphical fidelity to create a more compelling
virtual environment.
Gamers can use head
movements to look around and immerse themselves
in the virtual world. DG2: Defense Grid 2 is the
game that was played on the monitor and Defense
Grid 2: Enhanced VR Edition was the version
played on the Oculus Rift. Defense Grid 2 was
chosen as the game for this study as there are VR
and non-VR versions of the game. The versions of
the game are almost identical except that the VR
edition included minor changes to the user interface.
For example, the VR edition displayed the build
turret menu as a grid of symbols for each tower type
above the selected turret location. The monitor
version displayed the build turret menu on the left
side of the screen with details about tower attack
damage, range and rate of fire. The similarities
between the two versions were one reason for
choosing it as the game to use, as some games that
have a VR version change to only motion controls
or are not similar enough to compare. An Xbox One
controller was used to play both versions of the
game. There were slight differences in controls
between the two versions. In the VR edition, players
used their head to aim and inspect the map. In the
monitor version players used the left and right
control sticks on the Xbox One controller to move
the camera about.
Procedure
Participants signed up for the study on the
university’s online research management system.
After providing informed consent, they played the
video game Defense Grid 2 using either the Oculus
Rift or a traditional computer monitor. Once the
first portion of play was completed, participants
completed the GUESS. Participants then continued
to play Defense Grid 2 using the other device and
completed the GUESS again. A within-design for
this experiment was chosen to provide a comparison
between the two versions of the games. Participants
would continue play instead of restarting from the
beginning as playing that section a second time
could be boring or too easy for the participants and
could affect their satisfaction. The study took
approximately one hour to complete. The order of
conditions was counterbalanced across participants.
None of the participants were familiar with the
game prior to the study.
RESULTS
To examine the differences in GUESS
scores when users played Defense Grid 2 on either
the Oculus Rift or by using a traditional computer
monitor, paired-samples t-tests were conducted with
Bonferroni correction (.0025). There was a
significant difference in the overall satisfaction
scores between the Oculus Rift (M = 48.46, SD =
4.67) and traditional computer monitor conditions
(M = 45.53, SD = 4.76) with users rating higher
satisfaction on the Oculus Rift t (39) = -5.53, p <
.001. The Cohen’s d was .87 indicating a large
effect size.
Proceedings of the Human Factors and Ergonomics Society 2017 Annual Meeting 2074
Further analysis of the GUESS subscales
revealed that playing the game with the Oculus Rift
resulted in higher perceptions of engrossment,
enjoyment, creative freedom, audio aesthetic, and
visual aesthetic (Figure 1 and Table 2). No
significant differences were found for usability,
narrative, personal gratification, or social
connectivity.
Figure 1. Subscales of the GUESS that were more
satisfying for the Oculus Rift gameplay (7 = more
satisfied).
Table 2:
Overall GUESS and subscale scores for
Oculus Rift and Monitor.
Subscale Monitor Oculus Rift t Cohen’s d
Usability M = 5.85
SD = .68
M = 5.86
SD = .73
-.075 .01
Narrative M = 4.58
SD = .75
M = 4.77
SD = .67
-2.23 .35
Engrossment M = 4.76
SD = 1.01
M = 5.52
SD = .79
-5.30* .83
Enjoyment M = 5.38
SD = .94
M = 5.93
SD = .74
-4.50* .71
Creative
Freedom
M = 4.38
SD = .98
M = 4.95
SD = .86
-5.01* .79
Audio
Aesthetic
M = 5.58
SD = .80
M = 5.89
SD = .69
-3.89* .61
Personal
Gratification
M = 5.78
SD = .62
M = 5.89
SD = .61
-1.26 .20
Social
Connectivity
M = 3.82
SD = 1.05
M = 3.87
SD = 1.06
-2.79 .05
Visual
Aesthetic
M = 5.71
SD = .64
M = 6.20
SD = .63
-5.89* .93
Overall
Satisfaction
M = 45.53
SD = 4.76
M = 48.46
SD = 4.67
-5.53* .87
* p < .001 (two-tailed).
DISCUSSION
As the development of VR gaming
advances, game developers will want to consider
how VR affects player satisfaction. Some
developers note that game development for VR
requires more time to create and test since the
technology is still new and constantly changing
(Radak, 2016). The market for high quality VR
games is small when compared to traditional
gaming markets so making big budget large games
is a risk. The VR gaming market is new, quite
small, and high quality VR devices are expensive to
purchase. Developers are still discovering new
techniques and design guides that worked in
traditional mediums do not carry over well (Radak,
2016).
The results of this study demonstrate that,
for the strategy game tested, VR resulted in a more
satisfying experience. Users rated playing Defense
Grid 2 on the Oculus Rift as more satisfying than
when playing the same game on a traditional
computer monitor. In particular, they found it more
engrossing, enjoyable, open to creative freedom,
and had better audio and visual aesthetics. This may
be because the Oculus Rift provides a number of
features that are not available when using a regular
computer monitor. For example, users are able to
move their head and direct their aim just by
looking; this could lead to increased feelings of
immersion than just using a controller to “look”
about the world. This also may contribute to why
creative freedom was reported higher in the VR
version of the game. Players may feel like they
could explore more of the map and have more
freedom to look around the world as they are the
ones controlling movement with their head.
The results of this study contribute to the
literature in two ways: 1) they demonstrate how the
addition of VR to a standard PC-monitor video
game impacts player satisfaction, and 2) they
provide practical validation to the GUESS as a tool
to analyze player satisfaction across two visual
platforms. The GUESS could be helpful to video
game developers when deciding whether to enhance
a game with VR. While some differences were
found between the Oculus Rift and monitor
conditions for this game, it is possible that the same
may not be true for other games that vary in genre
1
2
3
4
5
6
7
Satisfaction Score
GUESS Subscales
Satisfaction Subscales Oculus Rift
Monitor
Proceedings of the Human Factors and Ergonomics Society 2017 Annual Meeting 2075
and quality. In addition, it is interesting to note the
GUESS subscales that were not different across
conditions (Usability, Narrative, Personal
Gratification, and Social Connection). The game
has other characters that spoke to the player about
the plot of the game and the most interaction with
other characters was associated with building turrets
to shoot at enemies. The lack of difference could be
due to the nature of the game played but should be
studied further with a variety of game types.
This study was limited in that only one
video game in one genre type was evaluated.
Players continued their play from one condition to
the next so one half of the game content could have
been more satisfying for users or they believed
using VR was more engaging due to it being a more
novel experience. Instead, a between subjects
design would limit participants to playing only one
version of the game. Future research could examine
how virtual reality affects other video games or
genres like racing or first person shooters.
Research
was conducted using the Oculus Rift, which is only
one of the available high-end VR headsets.
Researchers could investigate if the other high-end
VR headsets like the HTC Vive or PlayStation VR
produce similar results.
REFERENCES
Chalk, A. (2014, August). Analyst says digital sales
made up 92 percent of PC game market in
2013. PC Gamer. Retrieved from
http://www.pcgamer.com/analyst-says-
digital-sales-made-up-92-percent-of-pc-
game-market-in-2013/
Davis, J. P., Steury, K., & Pagulayan, R. (2005). A
survey method for assessing perceptions of a
game: The consumer playtest in game
design. Game Studies, 5(1).
Entertainment Software Association. (2017, April).
Essential facts about the computer and video
game industry. Retrieved from
http://essentialfacts.theesa.com/mobile/
Fulton, B. (2002, March 21). Beyond psychological
theory: Getting data that improves games.
Gamasutra. Retrieved from
http://www.gamasutra.com/view/feature/131
412/beyond_psychological_theory_.php
Martel, E., Su, F., Gerroir, J., Hassan, A., Girouard,
A., & Muldner, K. (2015, June). Diving
head-first into virtual reality: Evaluating
HMD control schemes for VR games. In
Proceedings of the 10th International
Conference on the Foundations of Digital
Games. New York, NY: ACM.
Nield, D. (2016). How Oculus Rift works:
Everything you need to know about the VR
sensation. Wearable. Retrieved from
https://www.wareable.com/oculus-rift/how-
oculus-rift-works
Phan, M. H., Keebler, J. R., & Chaparro, B. S.
(2016). The development and validation of
the game user experience satisfaction scale
(GUESS). Human Factors, 58(8), 1217-
1247.
Radak, J. (2016). VR & game development is not a
grocery store. Medium. Retrieved from
https://medium.com/@Fr0z3n/vr-game-
development-is-not-a-grocery-store-
f288da369efd#.j5sdd7gce
Sarkar, 2016. (2016). Almost two-fifths of steam’s
entire library was released in 2016. Polygon.
Retrieved from
http://www.polygon.com/2016/12/1/138079
04/steam-releases-2016-growth
Shelley, B. (2001, August 15). Guidelines for
developing successful games. Gamasutra.
Retrieved from
http://www.gamasutra.com/view/feature/131
450/
Tan, C. T., Leong, T. W., Shen, S., Dubravs, C., &
Si, C. (2015, October). Exploring gameplay
experiences on the Oculus Rift. In
Proceedings of the 2015 Annual Symposium
on Computer-Human Interaction in Play
(pp. 253-263). ACM.
Totilo, S. (2012, July 9). The difference between a
good game and a bad one. Kotaku. Retrieved
from http://kotaku.com/5924387/the-
difference-between-a-good-video-game-and-
a-bad-one
Proceedings of the Human Factors and Ergonomics Society 2017 Annual Meeting 2076
