ArticlePDF Available

An Empirical Study of Trends of Popular Virtual Reality Games and Their Complaints

Authors:
  • Huawei Canada

Abstract

The market for virtual reality (VR) games is growing rapidly, and is expected to grow from $3.3B in 2018 to $ 13.7B in 2022. Due to the immersive nature of such games and the use of VR headsets, players may have complaints about VR games which are distinct from those about traditional computer games, and an understanding of those complaints could enable developers to better take advantage of the growing VR market. We conduct an empirical study of 750 popular VR games and 17,653 user reviews on Steam in order to understand trends in VR games and their complaints. We find that the VR games market is maturing. Fewer VR games are released each month but their quality appears to be improving over time. Most games support multiple headsets and play areas, and support for smaller-scale play areas is increasing. Complaints of cybersickness are rare and declining, indicating that players are generally more concerned with other issues. Recently, complaints about game-specific issues have become the most frequent type of complaint, and VR game developers can now focus on these issues and worry less about VR-comfort issues such as cybersickness.
IEEE TRANSACTIONS ON GAMES, VOL. XX, NO. Y, FEB 2020 1
An Empirical Study of Trends of Popular Virtual
Reality Games and Their Complaints
Rain Epp, Dayi Lin, and Cor-Paul Bezemer
Abstract—The market for virtual reality (VR) games is growing
rapidly, and is expected to grow from $3.3B in 2018 to $13.7B
in 2022. Due to the immersive nature of such games and the
use of VR headsets, players may have complaints about VR
games which are distinct from those about traditional computer
games, and an understanding of those complaints could enable
developers to better take advantage of the growing VR market.
We conduct an empirical study of 750 popular VR games and
17,635 user reviews on Steam in order to understand trends in VR
games and their complaints. We find that the VR games market
is maturing. Fewer VR games are released each month but their
quality appears to be improving over time. Most games support
multiple headsets and play areas, and support for smaller-scale
play areas is increasing. Complaints of cybersickness are rare and
declining, indicating that players are generally more concerned
with other issues. Recently, complaints about game-specific issues
have become the most frequent type of complaint, and VR game
developers can now focus on these issues and worry less about
VR-comfort issues such as cybersickness.
Index Terms—Virtual Reality games, gamer complaints
I. INTRODUCTION
The releases in 2016 of consumer virtual reality (VR)
headsets such as the HTC Vive and Oculus Rift has afforded
developers a large and growing audience for VR computer
games. The VR market is rapidly growing in size: the install
base of VR is expected to grow from 14 million in 2018 to
51 million in 2022, and the VR game market is expected to
grow from $3.3B to $13.7B in the same period [45].
The unique display and input hardware used in VR games
offers new and different experiences and means of interaction
for players of VR games. When comparing the VR and
non-VR modes of individual games, VR has been shown to
increase immersion and enjoyment, but also may be associated
with worse performance in completing in-game tasks and a
perceived loss of control [27, 37, 46]. Moreover, use of VR in
games may lead to cybersickness [39], an issue typically ab-
sent from non-VR games. Reflecting these differences, players
may have complaints about VR games which are distinct from
complaints about ‘traditional’ computer games. An awareness
of such complaints would enable developers to take greater
advantage of the growing VR market as it continues to evolve.
While many aspects of VR computer games have been
studied (e.g., their medical [15, 26] or educational [7, 48]
potential, and comparisons with non-VR games [27, 37, 46]),
R. Epp and C. Bezemer are with the Analytics of Software, Games and
Repository Data (ASGAARD) lab, University of Alberta, Canada. E-mail:
{mdepp, bezemer}@ualberta.ca
D. Lin is with the Center for Software Excellence, Huawei, ON Canada.
E-mail: dayi.lin@huawei.com
to the best of our knowledge our study is the first large-scale
examination of user complaints of such games.
In this paper, we conduct a large-scale study of 750 VR
games and 17,635 reviews on the Steam platform, one of the
most popular distribution platforms for computer games. Our
study provides a broad view of complaints among players
of VR games. We first conduct a preliminary analysis of
the metadata of VR games on Steam. We then examine
the negative reviews of those games and classify them into
categories to identify common complaints. We focus on the
following two research questions.
RQ1: What are the characteristics of VR games on
Steam? The games have a median of 3 updates each,
and a median price of $10.99, though this price has
increased over time. Cross-platform support for multiple
VR headsets is increasing, although support for Windows
Mixed Reality headsets lags behind. Support for smaller-
scale play areas is getting increasingly common.
RQ2: What do players of VR games complain about?
We identify seven complaint categories. The most com-
mon complaint is that games are overpriced or lack
content; however, the total proportion of such complaints
is rapidly decreasing over time. Complaints reflecting
issues that are most related to VR-comfort, such as cy-
bersickness and VR control schemes, are rare. Recently,
game-specific complaints have become the most common
type of complaint.
This paper is organized as follows. Section II provides back-
ground information on VR technology and Steam. Section III
presents the methodology of our study. Sections IV and V
present the results of our investigation into VR game charac-
teristics and user complaints respectively. Section VI discusses
the implications of our findings. Section VII discusses related
work. Section VIII discusses threats to the validity of our
study. Finally, Section IX concludes the paper.
II. BACKGROU ND
In this section we provide a brief background on the VR
technology used by the games we study. We also describe
aspects of the Steam Store and Community which are relevant
to our data mining processes.
A. VR technology for games
The main device needed to play VR games is a VR headset,
which creates the illusion of a physical world by showing each
eye a separate render of the virtual game world. Orientation
and position tracking are used to update the virtual world
IEEE TRANSACTIONS ON GAMES, VOL. XX, NO. Y, FEB 2020 2
TABLE I
AN OVE RVIE W OF TH E FO UR MO ST P OPU LA R VR HE AD SET S US ED TO P LAY STEAM GAMES
Name Tracking Controllers Controller Tracking
Oculus Rift External sensors (outside-in) Oculus Touch Same as headset
HTC Vive External markers (inside-out) Vive controllers Same as headset
Windows Mixed Reality Markerless inside-out WMR controllers Sensors on headset
HTC Vive Pro External markers (inside-out) Vive controllers Same as headset
based on the user’s movements. According to the June 2019
Steam survey [47], the most popular VR headsets used with
Steam games are the Oculus Rift, HTC Vive, Windows Mixed
Reality1, and HTC Vive Pro. The capabilities of these headsets
are summarized in Table I.
Several headsets use external apparatus to aid position and
orientation tracking, such as external sensors for the Oculus
Rift and external markers for the HTC Vive and HTC Vive
Pro. However, some headsets, especially those released later,
implement tracking using only cameras and sensors on the VR
headset itself (markerless inside-out tracking). All Windows
Mixed Reality headsets use this method of tracking.
VR headsets can be roughly grouped into three categories.
PC-based headsets require a tethered connection to an external
desktop or laptop computer in order to offload processing
and rendering from the headset. Smartphone-based headsets
instead require a connection to a smartphone that is used for
processing and display. Finally, standalone headsets require
no connection to any external device in order to function. It
is worth noting that various means exist to use a wireless
connection instead of a tether for PC-based headsets, such as
the Vive Wireless Adapter for the Vive and Vive Pro. Also,
some standalone headsets (such as the Oculus Quest) can be
connected through a wire to a PC as well.
All four of the headsets in Table Iare PC-based. However,
playing Steam games on other types of headsets is also
possible, and various projects exist for this purpose, such as
ALVR2which acts as a remote display for the smartphone-
based Samsung Gear VR and the standalone Oculus Go and
Oculus Quest headsets.
While some VR games can be controlled using standard in-
put devices such as keyboards, mice, and game controllers, VR
gaming systems typically have dedicated motion controllers.
In most cases two controllers are used, one held in or attached
to each hand. Like the headset, their position and orientation
is tracked to provide a more natural and intuitive input than
traditional control schemes. These controllers may be tracked
using the same external markers or sensors as the headset, or
by sensors on the headset itself. Controller support is generally
limited to a single headset or series of headsets, but exceptions
exist (e.g., 3rd-party software exists to use the Vive controllers
with Windows Mixed Reality headsets).
Although game engines such as Unity and Unreal attempt
to abstract away the differences in controller and headset
functionality, such differences still have implications for game
1Windows Mixed Reality is not actually a discrete headset but instead refers
to any VR headset following Microsoft’s Windows Mixed Reality standard,
known as Windows Mixed Reality immersive headsets.
2https://github.com/polygraphene/ALVR
developers. For example, using the headset to track controllers
(as Windows Mixed Reality systems do) can lead to less
reliable controller tracking than other methods, potentially
breaking games which need 360tracking of controllers.
B. VR games on Steam
Steam is a digital distribution platform maintained by Valve
Corporation. It is the largest digital distribution platforms for
PC games, containing over 30,000 games3and over 90 million
active users monthly4.
Steam is also a major source of VR games, containing
over 3,000 games which support VR in some capacity. By
contrast, the official Oculus Rift store “Rift Experiences” had
792 games, and the official HTC Vive store “Viveport” had
986 games, as of 2019-05-13. Valve provides a set of tools
and services for users and developers of VR games known
collectively as SteamVR5. Applications can interface with
SteamVR using the OpenVR SDK. However, to publish a VR
game on Steam, it is not mandatory to use SteamVR. For
example, a game might use the Oculus SDK to support the
Oculus Rift and the OpenVR SDK to support other headsets.
VR games on Steam are searchable through two separate
filters: VR Only and VR Supported. The former describes
games which require VR hardware in order to play; in the
latter, use of VR is supported but optional. VR Only is almost
entirely a subset of VR Supported (we observed during our
data analysis that the four exceptions are likely an error by
Valve or the games’ developers).
For each game, developers can indicate officially supported
categories of headsets and controllers as seen by users. Headset
support is indicated to users as any combination of Oculus Rift
(e.g., Oculus Rift, Oculus Rift S), HTC Vive (e.g., HTC Vive,
HTC Vive Pro), and Windows Mixed Reality (all Windows
Mixed Reality immersive headsets)6; controller support by
Gamepad,Keyboard/Mouse, and Tracked Motion Controllers.
While in theory all SteamVR games are compatible with all
headsets among the possible categories (i.e., they will run), as
noted previously there are still differences in practice, which
make testing on individual devices important. In addition to
controllers and headsets, developers can specify supported
play areas for their games, indicated to users as Seated,
Standing, and Room-Scale. Valve defines a Standing play area
3https://store.steampowered.com/search/?category1=998
4https://partner.steamgames.com/
5https://www.steamvr.com/en/
6There are also labels for the Oculus Rift developer kits, but these are rarely
used, hence we ignore them. Additionally, after our data was collected Steam
added a label for the Valve Index headset.
IEEE TRANSACTIONS ON GAMES, VOL. XX, NO. Y, FEB 2020 3
as a 1-meter diameter circle and a Room-Scale play area as a
rectangle of at least 2×1.5meters.7
C. Steam Community
Once a user has recorded playtime8for a game on Steam,
they are able to post a user review for that game to Steam
Community. Instead of a star rating system, Steam reviews
are designated either Recommended or Not Recommended,
corresponding with positive and negative ratings respectively.
For each game, these ratings are summarized on the store page
in categories (e.g., Mostly Positive or Mixed), over all reviews
and over only reviews posted within the last 30 days.
Through Steam Community, developers and journalists are
able to post news updates, such as promotions or update
announcements, on a per-game basis. These news posts are
organized into channels denoting their origin or intended
purpose, such as PC Gamer or Product Releases.
III. METHODOLOGY
In this section we describe the methodology of our empirical
study. Figure 1gives a visual overview of our data collection
and filtering processes. In the remainder of this section, we
describe the processes in greater detail.
A. Collecting basic game data
We adapt a customized crawler from our prior work [23]
to extract information for all 31,177 games on Steam as of
April 4, 2019. For each game, the crawler gathers the title,
release date, publisher, developer(s), price (undiscounted, in
Canadian dollars), supported headsets, supported input de-
vices, supported playing areas, VR support (the VR Supported,
VR Only properties), and number of reviews. Vague or non-
specific release dates (e.g. January 2018) are ignored; only
release dates with daily precision are gathered.9
We select the games for our study according to the following
criteria:
VR support: The game must be a VR game.
Popularity: The game must have 25 or more reviews.
We wish to investigate which complaints players have with
VR games. This makes gathering reviews for games with
optional VR support somewhat problematic, since we do not
know whether a review describes the experience with or
without VR. Unfortunately, Steam provides no mechanism to
determine if a review is of the VR mode or non-VR mode of
a game (or both). Therefore, we only consider games marked
VR Only as “VR games” in this study.
There are a few instances in the paper in which it is helpful
to consider the full set of games (including the non-VR and
less popular ones); these cases are noted specifically when
they occur.
For each studied game, we also collected all Steam Commu-
nity news posts. As described in Section IV, we use these news
7https://steamcommunity.com/app/358720/discussions/0/
350532536103514259/
8https://partner.steamgames.com/doc/store/reviews
9Only the release date is ignored. The rest of the game data is still crawled
and processed as usual.
TABLE II
ABR IEF S UM MARY O F THE D ATA COLL ECT ED F ROM T HE STE AM S TOR E
AN D STEA M COMMUNITY
# of collected games in total 31,117
# of VR games 2,710
# of VR games with 25 reviews 750
# of collected reviews 150,887
# of unique review authors 74,899
# of English reviews 117,012
# of English, negative reviews 17,635
# of collected news posts 5,436
posts to estimate the number of updates for each VR game.
Additionally, we created a crawler to collect historical pricing
information of Steam games from the website SteamDB10. We
ran this crawler for all non-free, popular VR games to capture
how their prices changed over time.
B. Collecting reviews of VR games
We use another custom crawler to collect all 150,887
user reviews of the studied VR games. This crawler was
executed directly after the basic characteristics crawler and
also performed its data collection on April 4, 2019. For
each user review, we gather the title of the game under
review, the review’s content, the rating (Recommended or Not
Recommended), and whether the review was an early access
review.
C. Dataset description
Table II shows a brief description of the collected data.
As noted in prior sections, basic characteristics were crawled
for every game on Steam, but reviews and news posts, and
historical prices were only collected for VR games with 25 or
more reviews.
IV. RQ 1: W HAT ARE THE CHARACTERISTICS OF VR
GA ME S ON ST EA M?
Motivation: Before analyzing user reviews, we examine the
basic characteristics of VR games. As our paper is the first
large-scale examination of VR games on Steam, it is important
to understand the dataset. We examine trends within the data to
understand the evolution of VR games over time. The findings
in this section contextualize the remainder of the study.
Approach: We examined the release dates, developers,
prices, and updates of the collected VR games. We also
examined the level of support over time for the categories
of headsets, controllers, and play areas.
To estimate the number of updates for each VR game, we
used a method from our prior work which extracts release
notes from collected Steam Community news posts [21]. This
method was evaluated in that work to have a precision of
89% and a recall of 87% for early access games. Only news
posts from the Product Updates,Product Releases,Client Up-
dates, and Steam Community Announcements channels were
considered. A news post from these channels was considered
to contain release notes if:
10https://steamdb.info
IEEE TRANSACTIONS ON GAMES, VOL. XX, NO. Y, FEB 2020 4
Steam Store
Extract game
metadata
Filter by VR support,
popularity
Metadata of popular
VR games
Steam
Community
Extract reviews of
popular VR games
Reviews of popular
VR games
Extract news posts of
popular VR games
Estimate number of
updates from news
posts
Steam
Community
Collecting basic game data
Collecting reviews of VR games
Metadata of the
entire set of games
SteamDB
Extract historical
prices of popular VR
games
Fig. 1. An overview of the methodology of our study.
it was published on the Product Updates or Product
Releases channel; or
its title contained the word update,release,patch,hot-
fix, or change log, ignoring case, and using the case-
insensitive regex patterns hot.*fix and change.*log
for hotfix and change log respectively; or
its title contained a version number, using one of the
following case-insensitive regex patterns:
(version|alpha|beta|gamma|build) ?[0-9]+ , or
\bv\.? ?[0-9][0-9.a-z]*\b .
The number of updates per game was then considered to be the
number of news posts for that game which contained release
notes.
To compare the price and update distributions, we used the
Wilcoxon rank-sum test, a non-parametric test which can be
used to compare two independent distributions. If p < 0.05
we consider the distributions to be significantly different. To
quantify this difference, we use Cliff’s delta (d) [24], with the
thresholds provided by Romano et al. [40]:
Effect size is
negligible if |d| ∈ [0,0.147]
small if |d| ∈ (0.147,0.33]
medium if |d| ∈ (0.33,0.474]
large if |d| ∈ (0.474,1]
Early access games were identified as in our prior work [21],
using the presence of either an early access banner or an early
access review.
We estimated the release price of the paid games by com-
bining the SteamDB historical pricing data with the game’s
release date from Steam. If SteamDB recorded the price before
or at the release date, we used the last recorded price as
the release price. Otherwise, we estimate the release price
using the earliest recorded price for the game, provided it was
recorded no more than two days after the game’s release date.
Results: The rate at which VR games are released is
declining. Figure 2shows the number of games released per
week from 2016 to 2019. Throughout the year 2018, 158
of the studied games were released, corresponding with 3.04
releases per week. The release rate of VR games peaked at
the beginning of 2017 and thereafter declined. However, since
only games with 25 or more reviews were considered, a slight
decline is to be expected nearer the present, as recent games
have had less time to accumulate reviews. For this reason
Figure 2also shows the number of games per week for all VR
games, including those with few reviews. When considering
all VR games, the release rate peaked mid 2017, but again
declined after that time. The initial spike coincides with the
release of the HTC Vive in June 2016. The next spike, at the
end of 2016, occurs in the first three-quarters of December in
that year and is likely a result of developers rushing to release
their games by Christmas 2016.
The median price of a VR game was $10.99, and $14.24
when considering paid VR games only. Paid VR games
were more expensive than paid non-VR games. Figure 3
shows the price distribution of the games. The median price of
the studied games was $10.99. 184 of 750 (25%) of the studied
IEEE TRANSACTIONS ON GAMES, VOL. XX, NO. Y, FEB 2020 5
VR games with 25 reviews
All VR games
2016 2017 2018 2019 2016 2017 2018 2019
0
10
20
30
40
50
Year
Number of games released per week
Fig. 2. Release dates of Steam VR games. The plot on the left shows the
release dates of the VR games with 25 reviews; the plot on the right shows
all VR games, including those with fewer reviews. Games with release dates
in the future are excluded, as are 6 games with release dates before 2016.
Additionally, 151 games with no release date given are not shown (including
1 with 25 or more reviews). The trend line is a LOESS trend line which
provides a smoothed estimate of the rate of releases at each point in time.
110 100
VR
Non−VR
Fig. 3. Price distribution of the studied VR and non-VR games. The x-axis
is 1 + the price of the game, in Canadian dollars. The black vertical lines
indicate the median values of each distribution.
games were free or free to play. Excluding these games, the
median price (of non-free games) was $14.24. The two most
costly games, with a price of $80.00, were Fallout 4 VR11 and
The Elder Scrolls V: Skyrim VR12, both developed by Bethesda
Game Studios. The median price of a non-VR game (with 25
or more reviews) was $7.79 and $8.99 when considering paid
games only. A Wilcoxon rank-sum test showed that the price
distribution of the VR games was significantly different than
that of (popular) non-VR games, both in the overall and paid
case. The effect size was negligible (|d|= 0.06) overall and
small (|d|= 0.30) for paid games.
The release price of paid VR games has increased over
time. Figure 4shows the trend of the release price of the paid
studied VR games over time. This is the undiscounted price
of each game at its release. The price increases for two years
until just after 2018, when it levels off and even decreases
slightly. Within the final six months, the median price for a
paid VR game was $22.14, compared with $10.99 within the
first six months.
11https://store.steampowered.com/app/611660/
12https://store.steampowered.com/app/611670/
10
15
20
25
2017 2018 2019
Year
Median price of paid VR games
Fig. 4. Trends for the release prices of the paid VR games. The Year axis is
grouped into months. Each data point is the median price of paid VR games
released in that month. The line is a LOESS trend line, indicating a smoothed
estimate of the median price of paid games over time. Games with release
dates before 2016 or in the future are excluded, as are 22 games with no
estimated release price from SteamDB.
● ●
1 10 100
1 + Number of updates per game
Fig. 5. Distribution of the number of updates to VR games.
The studied VR games received a median of 3 updates.
Figure 5shows the distribution of the number of updates per
studied VR game. Of these games, 176 of 750 (23%) received
no updates at all. In addition, 439 of 574 games with updates
(76%) received only 10 or fewer updates. Early access games
received a median of 8 updates, while non-early access games
received only 2 (this difference was statistically significant as
determined with a Wilcoxon rank-sum test, with |d|= 0.61
indicating a large effect size). These figures suggest that there
is a low level of ongoing developer support for many of the
studied games.
396 of 715 VR developers (55%) published only a single
game on Steam, and 77% exclusively published VR games.
These numbers take into account all games (including non-
VR or less popular ones) created by the VR developers.13
Of the developers with multiple games, 153 of 319 (48%)
have exclusively created VR games. In addition, among the
developers of the studied games with both VR and non-
VR games on Steam, 72% published the non-VR game
first. Again, these numbers take into account all games by
the VR developers, including non-VR or less popular ones. A
total of 166 developers published both VR and non-VR games
on Steam. 156 of these developers had at least one game in
each category (VR or non-VR) with a well-defined release
date. Among these 156 developers, 113 (72%) released their
first non-VR game before their first VR game. Therefore while
only a small proportion of VR developers created both VR and
13By VR developer we refer to developers which created at least one of
the 750 studied VR games. Developers of only less-popular and/or non-VR
games are not considered, since we wish to characterize the developers of the
studied games only. However, all games created by the VR developers are
taken into account, in order to properly characterize their output.
IEEE TRANSACTIONS ON GAMES, VOL. XX, NO. Y, FEB 2020 6
● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●
0.00
0.25
0.50
0.75
1.00
2016 2017 2018 2019
Year
Fraction VR games for headset
HTC Vive Oculus Rift Windows Mixed Reality
Fig. 6. Trends for headset support among the studied games. The Year axis
is grouped into months. Each data point is the fraction of games released in
the corresponding month which support a VR headset as of 2019-04-04. The
lines are LOESS trend lines, indicating for each headset a smoothed estimate
of the fraction of VR games released supporting that headset.
non-VR games, most developers among that group had prior
experience making non-VR games before publishing their first
virtual reality game on Steam.
In addition, we manually examined a random sample of
50 of these developers and determined that only 6 of their
collective 199 games (3.0%) are sequels, spinoffs, or remakes
of existing non-VR titles that were released by those devel-
opers on Steam. This indicates that, generally speaking, the
studied virtual reality games are new material likely created
specifically for VR instead of adaptations of existing games.
The HTC Vive is the most widely supported headset on
Steam with more than 99% support. Among the studied
games, 749 of 750 support the HTC Vive. In contrast, only
77% and 31% of the studied games support the Oculus Rift
and Windows Mixed Reality categories, respectively. The
high support for the HTC Vive is likely related to Valve’s
close association with the HTC Vive headsets, which were
developed by a co-operation between HTC and Valve.
Support for Oculus Rift and Windows Mixed Reality
headsets is increasing. Figure 6shows the fraction of games
released each month with official support for each VR headset
category. Support for Oculus Rift and Windows Mixed Reality
headsets grows throughout the entire timeline. Within the final
6 months, support for Oculus Rift headsets has reached 97%,
indicating it has essentially caught up with the Vive. In that
time period, Windows Mixed Reality support still lags behind
at 67%, although it continues to increase at least up until the
time of data collection.
SteamVR gained support for Windows Mixed Reality head-
sets in November 15, 2017 [25], a month after the first launch
of Windows Mixed Reality headsets. At the time, indicated
support for Windows Mixed Reality headsets was disabled
by default, requiring developers to manually specify that
their games were compatible with Windows Mixed Reality.
Therefore it is likely that the 116 games with support for these
headsets prior to November 15, 2017 have had support added
after their release.
712 of 750 (95%) games support tracked motion con-
trollers, and support for other control schemes is de-
creasing. Figure 7shows the proportion of games released
● ● ●
● ●
● ● ●
0.00
0.25
0.50
0.75
1.00
2016 2017 2018 2019
Year
Fraction VR games for controller
Gamepad Keyboard / Mouse Tracked Motion Controllers
Fig. 7. Proportion of games released each month with support for the three
controller categories. This graph is constructed identically to Figure 6, except
it displays controllers instead of headsets.
with support for each category of controller. 98% of the
studied games released in the final six months support tracked
motion controllers, and only 3 of 58 games (5.2%) released
in that time period support other control schemes (i.e. key-
board/mouse or gamepad). By contrast, only 87% of the VR
games that were released within the first 6 months of 2016
had support for tracked motion controllers, and a full 25%
supported other control schemes. A possible explanation could
be the lack of a dedicated controller for the Oculus Rift at
launch. Not only has support for dedicated VR controllers
increased to near-ubiquity, but recent releases have largely
stopped supporting traditional control schemes at all.
The most common play areas are room-scale and stand-
ing, supported by 79% and 76% of games respectively.
Support for the seated play area is increasing. As seen
in Figure 9, the proportion of games released with support
for the “room-scale” and “standing” play areas has remained
approximately constant since the year 2017. However, support
for the “seated” play area has increased throughout the time-
line from 40% of games released in the first six months to
59% in the final six months. Within the final six months the
support for room-scale and standing play areas is 78% and
85% respectively, indicating that support for “standing” has
also increased. The increasing support for smaller-scale play
areas is likely an attempt by developers to make their games
appeal to a wider audience, since the larger-scale VR setups
require greater user commitment in terms of equipment and
physical space, and thus may be unavailable or unappealing for
some users. In addition, a smaller required play area would
make the game more portable. As standalone VR headsets
grow in popularity, this would allow the game to be played
while travelling, such as on an airplane.
V. RQ 2: W HAT DO PLAYERS OF VR GAMES COMPLAIN
ABOUT?
Motivation: User reviews are a visible indication of the user-
perceived quality or enjoyment of a game. It is therefore im-
portant to developers to obtain positive reviews for their games
in order to maximize their chances of success. Reviews also
provide valuable feedback for developers on which aspects
of their games have been positively received and which need
IEEE TRANSACTIONS ON GAMES, VOL. XX, NO. Y, FEB 2020 7
17,635 negative reviews of popular VR games
Preprocessing the data
Convert to lower case
Remove punctuation
Remove stop words
Stem review text
Preprocessed
reviews
Extracting topics
Run Twitter-LDA
Extracted topics
Rank topics based on
number of reviews
Top 10 topics
Repeat 5 times
Categorizing complaints
Read sample of 20
reviews for each topic
Categorize
complaints in topics
Consolidate
complaint categories
Categorize topics
again
Complaints
7 complaint
categories
5x10 categorized
topics (by two
authors)
Fig. 8. An overview of our approach for extracting complaints from the negative reviews.
● ●
0.00
0.25
0.50
0.75
1.00
2016 2017 2018 2019
Year
Fraction VR games for play area
Seated Standing Room−Scale
Fig. 9. Proportion of games released each month with support for the three
play area categories. This graph is constructed identically to Figure 6, except
it displays play areas instead of headsets.
improvement. Although both positive and negative reviews can
provide useful feedback, our prior work shows that negative
reviews do so more often [23]. Here we analyse negative user
reviews to extract the most commonly-occurring complaints
for virtual reality games. Such an overall understanding of
users’ complaints of VR games is useful for developers and
could enable them to tailor their development and testing
efforts to most effectively address players’ concerns.
Approach: Figure 8give an overview of our approach for
extracting complaints from reviews. We detail each step below.
Preprocessing the data: All 17,635 negative, English re-
views for the studied VR games were automatically classi-
fied using the Twitter-LDA [51] algorithm. Twitter-LDA is a
variation of LDA that is designed to assign topics to Twitter
“tweets”, and as such is capable of generating meaningful
topics from smaller amounts of text than traditional LDA. One
topic was assigned to each document (review). Prior to running
the algorithm, each review was preprocessed as follows:
1) The review text was converted to lowercase and punc-
tuation was removed.
2) Using the Python package nltk [3], English stopwords
were removed from the review text.
3) The review text was stemmed using the Porter stemmer,
as implemented by nltk.
Extracting topics: After running Twitter-LDA on the re-
views, the generated topics were ranked according to the
number of reviews assigned to each topic. Twitter-LDA is
nondeterministic, and its results will vary by run. Therefore
we ran the algorithm five times and combined the results by
grouping the top 10 topics of each run (by number of reviews)
into manually-classified complaint categories.
Categorizing complaints: Each topic was assigned one com-
plaint category as follows:
1) For each run, the first and third authors were each given
a random sample of 20 reviews for each of the top 10
topics of that run. Each author was therefore assigned
a total of 1000 reviews over 50 topics. Using these
reviews, both authors independently identified categories
for each topic. Each topic was assigned exactly one
category.
2) The categories identified by each author were consoli-
dated into a new set of 7 complaint categories agreed
upon by both authors. These complaint categories are
shown in Table III.
3) The topics were classified by the first and third authors
once more (using the same set of reviews as in step 1),
this time into the predefined complaint categories. The
authors disagreed on 3 of the 50 classified topics, with
Cohen’s kappa κ= 0.929, indicating a strong agreement
between categorizations. The remaining differences were
resolved through discussion.
After this process was complete, each review had been
automatically classified into a topic once in each run, and
each of the top 10 topics for each run had been grouped
into a complaint category. We therefore considered a review
to belong to a complaint category if it was assigned a topic
which was grouped into that category. For each run, we ranked
the complaint categories by number of reviews, from 1 to
7 for most to least reviews. For each complaint category,
we calculated the median rank across runs to summarize its
relative occurrence among the reviews.
Results: Table III shows the median rank of each category
across all runs. Categories with rank 1 had the most reviews
within a run; categories with rank 7 had the least. Figure 10
IEEE TRANSACTIONS ON GAMES, VOL. XX, NO. Y, FEB 2020 8
TABLE III
MAN UALLY I DEN TI FIED C OM PLA IN T CATEG OR IES F OR N EGATI VE R EVI EW S OF STE AM VR GAM ES . THE E XA MPL E FOR E ACH C OM PLA IN T CATEG ORY I S
A FR AGME NT O F TEX T FRO M A RE VIE W WI TH TH AT CATEG ORY (TH E RE VIE W TE XT HA S BE EN MI NI MAL LY EDI TED F OR C LAR IT Y).
Complaint category Description and Example Median rank
lacks content The game is too short, lacks content, or is overpriced. “it’s good but not worth the money” 1
game-specific A significant number of reviews in the topic are for a single game only, and contain complaints about game-
specific information such as gameplay. “Skyrim is cool and all, but . . . it’s basically ‘Crouch-Simulator with
dragons’ ”
3
community The game has been abandoned by either players or developers, or has multiplayer issues. “I couldn’t find
anyone online to play with”
4
crashes The game crashes, freezes, cannot start, or otherwise has major bugs rendering it unplayable. “it crashes
within minutes of starting”
4
nausea The game causes nausea or headaches. “made me ill to the point I have to lie down” 5
controls The game has various issues with controls (locomotion and/or manipulation) or with headset. “can’t correctly
use the oculus touch controllers”
6
optimization The game has optimization issues such as stutters or a low framerate. “starts out smooth then gets progressively
choppier”
7
shows the relative frequency of each complaint category over
time. The most common complaint was that games lack
content or are overpriced. The frequency of this complaint
is decreasing. These complaints correspond with the lacks
content complaint category. As can be seen in Figure 10,
the relative frequency of this complaint category has decreased
over time from 2016 to 2019. Interestingly, while one might
expect early reviews to be more tolerant of such concerns given
the early nature of the technology, this does not appear to be
the case. The decreasing frequency of this complaint suggests
that that the VR games market has become more mature over
time, with developers as a whole moving away from smaller,
“tech demo”-like experiences typical of early efforts. Another
explanation could be that gamers are now used to VR games
being smaller than non-VR games and hence compare newer
VR games to older VR games, instead of to non-VR games.
It is interesting to note that the release price of VR games has
increased over time (see Figure 4) even as the frequency of
lacks content complaints have decreased. This suggests
that as time goes on, players of VR games have felt they are
getting better value for their money.
A major complaint was with lack of developer or player
community. The corresponding complaint, community, had
median rank 4 and is thus tied for second most-occurring
(aside from the game-specific complaint category). The
relative frequency of this complaint has largely stayed constant
over time. This suggests that abandoned games (both by the
developer and the players) have been and continue to be an
issue for players of VR games.
Crashes, freezes, and game-breaking bugs were com-
plained about less frequently than game design- and
business-oriented issues. The category crashes, cor-
responding with such bugs, had median rank 4. The
game design- and business-oriented lacks content and
community complaint categories, with ranks 1 and 4 respec-
tively, collectively occurred more frequently than complaints
about bugs and crashes. This observation is consistent with
our prior work [23], which was not specific to VR games and
which showed that, especially among negative reviews, com-
plaints about game design are more common than complaints
about bugs (including crashes).
The relative frequency of game-specific complaints
jumped in early 2018. These complaints (belonging to the
game-specific category), are from topics with a large
percentage of reviews that complaint about a single game.
This increase and another in early 2019 was large enough to
make these complaints the second-most-frequently occuring.
This could indicate a shift to complaints about specific game
content instead of other issues. As with the decline in the
lacks content complaint category, this shift likely reflects
the growing maturity of the VR games market.
The complaints reflecting issues that are most unique
to VR occurred the least often. The complaint categories
nausea,controls, and optimization had the lowest
median ranks assigned, at 5, 6, and 7 respectively. These
categories correspond most closely with issues specific to
VR-comfort: nausea since it describes symptoms of cy-
bersickness; controls due to the unique control schemes
in VR; and optimization since VR games require extra
processing power to render scenes to each eye, and since issues
with unoptimized games (e.g. stuttering or reduced framerate)
can induce cybersickness. While the absence of these issues
does not guarantee user enjoyment, such issues can quickly
render a game unplayable. The low rank assigned to the
categories suggests that these issues were considered by users
to be less important than the others which are less intrinsic
to VR platforms, suggesting that developers and hardware
designers have largely succeeded in reducing VR-comfort
issues to an acceptable level for computer games.
VI. IMPLICATIONS
The VR games market is maturing. Complaints that
games are overpriced for their content have decreased sharply,
even as the price of VR games has risen over time. The median
price of a paid VR game has risen from $10.99 in the first
six months of 2016 to $22.78 in the latest six months of
data collection, levelling off after 2018. Despite this increase,
complaints about games which are overpriced or lack content
have decreased in frequency from roughly 50% to 20% of all
complaints. This decrease suggests that the quality of new
games has generally increased, since players have become
more satisfied with the amount of content in newer games even
IEEE TRANSACTIONS ON GAMES, VOL. XX, NO. Y, FEB 2020 9
lacks content
nausea
optimization
community
controls
crashes
game−specific
2016 2017 2018 2019 2016 2017 2018 2019 2016 2017 2018 2019
2016 2017 2018 2019
0.0
0.2
0.4
0.6
0.8
0.0
0.2
0.4
0.6
0.8
Year
Fraction complaints with topic
Fig. 10. The fraction of categorized negative reviews belonging to each category over time. The Year axis is grouped into months. Each point is the fraction
of categorized complaints published in that month belonging to the given category. This fraction is calculated using the combined topics and categories of all
five runs. The lines are LOESS trend lines, providing a smoothed estimate of the Fraction of categorized complaints with each topic over time.
as they are more expensive. Hence, while less VR games are
released each month, their quality seems to be improving.
Developers should be prepared to deliver cross-platform
experiences which support multiple play areas. The ma-
jority of recent releases have support for multiple headset
categories and play areas, likely creating an expectation of
such support among players. Within the final six months of
our study period, 97% of the released games supported both
Oculus Rift and HTC Vive headsets, with Windows Mixed
Reality support at 67% and increasing. Similarily, within this
time period 85% of the releases supported Standing and 78%
of the releases supported Room-Scale play areas. Seated play
areas have seen increasing support, reaching 59% overall
within the final six months and climbing steadily, a trend
developers should keep in mind when creating their VR games.
It is also worth noting that nearly all recent releases have
supported tracked motion controllers: 98% of games released
in the final six months have support for tracked motion
controllers, and only 5.2% supported other control schemes.
Therefore developers should not worry about including key-
board or gamepad support in their games.
Cybersickness is not a major issue among players of
current VR games. The frequency of complaints reflecting
cybersickness has decreased steadily over time, likely reflect-
ing efforts of developers and hardware makers to mitigate the
issue. However, cybersickness has never comprised a large
proportion of the complaints. It was initially dwarfed by
complaints about price and content, and recently by complaints
about game-specific issues. Overall, it is ranked the 5th-most
common complaint category. While Rangelova et al. con-
cluded from an online survey that cybersickness is widespread
among gamers [38], our results suggest that cybersickness is
not a major issue with players compared with other concerns.
Developers should focus on delivering high-quality gam-
ing experiences rather than further improving VR-comfort
aspects of their games. Since approximately 2018, the most
common complaint among user reviews has been with game-
specific issues. Complaints of cybersickness, controls, and
other elements reflecting VR-comfort issues have low and
declining frequency. In order to address the largest number of
complaints, developers should focus on further improving the
value of their VR experiences as games rather than addressing
VR-comfort issues. Since the frequency of complaints about
lack of content and overpriced games has decreased sharply
over time, developers may consider improving the quality of
game content, for example by fixing bugs and crashes, or by
building game communities, both corresponding to complaint
categories ranked above VR-comfort issues.
VII. REL ATED W OR K
In this section, we discuss related work about (1) virtual
reality complaints, (2) game repository mining, and (3) games
and software engineering.
Virtual reality complaints: Most of the work which focuses
on what issues users of current VR technology have with
the technology is concerned with cybersickness. Porter III
et al. [36] studied discussions on Reddit about cybersickness
and immersion on the HTC Vive. Similar to our study, they
conclude that the VR games market is maturing and that the
concerns of gamers have evolved over time.
A few studies have investigated the severity of VR sickness
among current VR games. Munafo et al. [30] reported 22% and
56% incidence of motion sickness when participants played
two VR games (Balance Rift and Affected) on an Oculus Rift
DK1. Shafer et al. [41] conducted an experiment across three
commercial VR games and two headsets (Oculus Rift DK1 and
CV1), and concluded that factors important to the enjoyment
of other types of games were also important for VR games,
and that the different headsets had no impact on the severity
of cybersickness. Walch et al. [49] compared a racing game
played on either flat screens or an HTC Vive headset. They
found that the visualization method had no significant effect
on SSQ scores (a measure of cybersickness), although users of
the VR setup felt a significantly greater amount of discomfort.
Tan et al. [46] explored gameplay experiences from the game
Half-Life 2 on the Oculus Rift DK1. While 8 of 10 participants
experienced cybersickness at some point, it largely did not
effect immersion. Rangelova et al. [38] conducted the only
IEEE TRANSACTIONS ON GAMES, VOL. XX, NO. Y, FEB 2020 10
large-scale examination of cybersickness among gamers of
which we are aware. Using an online survey, they concluded
that cybersickness is widespread among players of current VR
games. While our study does not measure the incidence or
severity of cybersickness among players, we do examine user
reviews complaining of its symptoms, and are thus able to
rank its perceived severity relative to other player complaints.
Several studies have leveraged user reviews to study com-
plaints for other platforms such as mobile apps. Khalid
et al. [17] studied 6,390 low-rated reviews for 20 free iOS
apps. They found that the most frequent complaints were of
functional errors, feature requests, and crashes, while the most
negatively-impacting complaints were of ethical and privacy
issues. Mcilroy et al. [28] found that up to 30% of low-
rated reviews contained multiple complaints, and presented
a classifier to identify complaints from user reviews. Fu
et al. [8] presented a system which analyzed how user com-
plaints evolved over time. Mujahid et al. [29] investigated
complaints in 2,667 reviews of 19 Android wearable apps. The
most frequent complaints were functional errors and cost, and
the most negatively-impacting complaints were installation,
incompatibility, and ethics issues. Hassan et al. [12] studied
the top 250 bad updates (updates with the highest increase
in the percentage of negative reviews following the update)
of 2,526 free apps on the Google Play Store. They found
that feature removal and user interface issues caused the
highest increase in negative reviews but that developers were
most likely to fix crashes and functional issues. Our study
investigates PC-based VR games, and is the first to show
that the occurrence of complaints about cybersickness is low
amongst players of popular VR games on Steam.
Game repository mining: Several studies have mined data
from digital game distribution platforms. Sifa et al. [43]
analyzed the playtime distributions of over 3,000 games on
the Steam platform, grouped them into four archetypes, and
described the main types of games within each archetype.
Poretski and Arazy [35] collected data for 45 games from
the Nexus Mods game mod distribution platform to investigate
the value added by modding communities. Our prior work
has collected game and review data from Steam in order to
analyze early access games [21], urgent updates [20], and user
reviews [23]. In our prior work on user reviews, we studied
reviews of all types of games. In addition, we did not focus on
complaints of gamers. In this paper, we focus on complaints
from VR gamers in particular. VR games and their complaints
must be studied separately from traditional games due to the
presence of hardware that may affect the gamers’ complaints.
Many studies focused on the social media aspects of Steam
Community and on characterizing player behaviour. O’Neill
et al. [32] collected data from 109 million user accounts and
384 million owned games on Steam. They reported a low
number of friendships compared to other social networks,
among other findings, and emphasized the diversity of player
behaviour within their results. Becker et al. [2] analyzed
the Steam Community social network, user groups, and the
evolution of the network over time. Sifa et al. [44] used the
Steam web API to analyze cross-game behaviour of 6 million
players across more than 3,000 games. Blackburn et al. [4]
observed “cheaters” within Steam Community and found that
cheating behaviour spreads with a “contagion”-like effect, with
players having cheating friends being more likely to later
become cheaters themselves. Li et al. [19] collected user
profile for 60 thousand users within Steam Community and
extracted eight factors characterizing user attributes. Baumann
et al. [1] analyzed so-called “hard-core” gamers.
We collect game data and reviews for VR games on Steam,
and our work is the first to focus specifically on VR games.
Games and software engineering: There is a large amount of
work studying computer games and various aspects of software
engineering. Much of this is work concerned with the game
creation process, such game as architecture [9, 18, 31] or
development practices [16, 33, 34].
Numerous studies have applied data mining techniques to
compute game data. Many of these have used in-game teleme-
try, public APIs, or game logs to analyze player behavior for
a single game or small number of games, e.g. [10, 11, 13, 14,
42, 50]. Among the studies which consider a large number of
games as we do, a few gather data primarily from platforms
other than digital distribution platforms. Chambers et al. [5]
examined a collection of 550 online games using data from
GameSpy.com (now defunct) to investigate game workloads
and the potential for shared infrastructure. Cheung et al. [6]
gathered 200 reviews from a combination of Amazon and
long-form reviews from gaming websites, as well as interviews
from industry professionals, and recommended a focus on
engagement rather than fun for the first playing hour of games.
In our prior work [22], we collected metadata of Youtube game
videos and trained a random forest classifier to identify videos
showcasing game bugs.
VIII. THR EATS TO VALIDITY
Internal validity: A potential threat to the validity of our
study is that we filtered out games with fewer than 25 reviews.
Recent games could be underrepresented since they have not
had much time to accumulate enough reviews. Future studies
should repeat our analysis in several years to understand how
trends have changed since then.
Another threat is that user reviews are prone to review
bombs, i.e., large numbers of negative reviews to discredit
a game. While we did not observe any indications of review
bombs in our data, future studies should further sanitize the
collected data to reveal the impact of such bombs.
In order to categorize user complaints, we only studied
reviews written in English. We also examined only negative
reviews, even though our prior work has shown that positive
reviews also frequently contain complaints [23]. Because of
the small percentage of positive reviews which contain com-
plaints, it is difficult to automatically extract complaints from
these reviews. Future studies should investigate the complaint
categories in positive reviews of VR games.
The HTC Vive headset was developed by HTC in partner-
ship with Valve, the owner of Steam. This close relationship
with Valve likely explains the large fraction of games which
support that headset.
Construct validity: We estimated trends in headset, con-
troller, and play area support by combining the release date
IEEE TRANSACTIONS ON GAMES, VOL. XX, NO. Y, FEB 2020 11
of each game with the stated support on Steam. However,
developers may update these values at any time, and so the
stated headset, controller, and play area support among earlier
games may not reflect their values at launch. In particular,
we found 116 games which were released before the launch
of Windows Mixed Reality headsets, but which officially
supported those headsets at the time our data was collected.
We estimated the number of updates to each game by
filtering the game’s news posts for release notes. In our prior
work with early access games, we evaluated this method to
identify release notes with a precision of 89% and a recall of
87% [21]. Since posting such release notes is not mandatory,
our method could underestimate the number of updates.
Twitter-LDA only assigns one topic per review, even though
a review might contain multiple complaints. Our manual
categorization also assigned only one category per topic.
Assigning only one topic per review might cause complaints of
secondary importance within user reviews to be disregarded.
To assess the impact of this threat, we manually studied a
randomly selected sample of 70 negative reviews to verify
what portion of the reviews contains multiple complaints. We
found that approximately only 23% of the reviews contained
complaints from more than one category. However, most of
these reviews were categorized into the game-specific and the
controls category. During the manual categorization in the
rest of the paper, we always favoured the other complaint
categories over the game-specific category. Hence, we do not
expect that the fact that we assigned only one category per
topic influences the eventual outcome of our study. Regardless,
future studies should further investigate the impact of allowing
multiple complaint categories per review.
External validity: We only examine games with mandatory
VR support. This excludes games which have both VR and
non-VR modes, and so our results may not generalize to those
games. Unfortunately, this restriction was unavoidable since
games with optional VR support may contain reviews both
of the VR and non-VR content, with no method to determine
which is which. Likewise, we only study popular PC-based
VR games on Steam and our results may not generalize to
VR games for other platforms, such as PlayStation VR, or
PC-based VR games that are not available on Steam.
Our findings may not hold for the entire potential audience
for VR games. For example, players sensitive to cybersickness
may be less likely to own a VR headset and to review VR
games on Steam, possibly making them underrepresented in
our results. Finally, our results apply to PC-based headsets
since, as explained in Section II, these comprise the majority of
VR headsets used with Steam. Our results may not generalize
to other types of VR headsets, such as standalone headsets.
IX. CONCLUSIONS
The unique display and input hardware used by VR games
allows these games to offer different experiences than non-VR
games. Understanding what complaints players have about VR
games may allow developers to tailor their development and
testing efforts to most effectively address user concerns.
We performed an empirical study of 750 PC-based VR
games and 17,635 user reviews on the Steam platform. We ex-
amined trends in release frequency, price, headsets, controllers,
and play areas, and also examined the games’ developers
and updates. We extracted seven common categories from the
negative user reviews in order to understand player complaints,
and examined the frequency of these complaint categories over
time. Our most notable findings are:
1) The majority of recent releases have support for multiple
headset categories and play areas. In particular, support
for smaller-scale playing areas has increased.
2) Complaints that games are overpriced for their content
have decreased sharply, even though the median price
of VR games has more than doubled over time.
3) The frequency of complaints reflecting cybersickness is
low and has decreased steadily over time.
4) Since approximately 2018, the most common complaint
among user reviews has been with game-specific rather
than VR-comfort issues.
Our findings show that the PC-based VR games market
is maturing. In particular, we observe that VR-comfort is-
sues which have traditionally received widespread attention
in research, such as cybersickness, are uncommon within
complaints and have become less prevalent. There are several
possible explanations for this reduction in complaints about
VR-comfort. The first possible explanation is that PC-based
VR technology and design patterns for VR have greatly
improved over the years, allowing developers to create VR ex-
periences that are more comfortable and less nausea-inducing.
A second possible explanation is that players who suffer from
VR-comfort issues when playing VR games simply no longer
play these games, thereby reducing such complaints (even
though the issues may still exist). Future studies should further
investigate these possible explanations for the reduction in
complaints about VR-comfort.
Overall, our findings are an indication that developers of
PC-based VR games now no longer should have improving
VR-comfort as their sole focus. Instead, they can start focusing
more on improving the design and gameplay of VR games to
attract a wider range of players (while, of course, maintaining
the same level of VR-comfort).
REFERENCES
[1] F. Baumann, D. Emmert, H. Baumgartl, and R. Buettner, “Hardcore
gamer profiling: Results from an unsupervised learning approach to
playing behavior on the Steam platform,” in Knowledge-Based and
Intelligent Information & Engineering Systems: Proc. of the 22nd
International Conference KES-2018, Belgrade, Serbia, 3-5 September
2018., 2018, pp. 1289–1297.
[2] R. Becker, Y. Chernihov, Y. Shavitt, and N. Zilberman, “An analysis of
the Steam community network evolution,” in IEEE 27th Convention of
Electrical and Electronics Engineers in Israel, Nov 2012, pp. 1–5.
[3] S. Bird, E. Klein, and E. Loper, Natural Language Processing with
Python. O’Reilly Media Inc., 01 2009.
[4] J. Blackburn, N. Kourtellis, J. Skvoretz, M. Ripeanu, and A. Iamnitchi,
“Cheating in online games: A social network perspective,ACM Trans.
Internet Technol., vol. 13, no. 3, pp. 9:1–9:25, May 2014.
[5] C. Chambers, W.-c. Feng, S. Sahu, and D. Saha, “Measurement-based
characterization of a collection of on-line games,” in Proc. of the 5th
ACM SIGCOMM Conference on Internet Measurement. Berkeley, CA,
USA: USENIX Association, 2005, pp. 1–1.
[6] G. K. Cheung, T. Zimmermann, and N. Nagappan, “The first hour
experience: How the initial play can engage (or lose) new players,” in
Proc. of the First ACM SIGCHI Annual Symposium on Computer-human
Interaction in Play, 2014, pp. 57–66.
IEEE TRANSACTIONS ON GAMES, VOL. XX, NO. Y, FEB 2020 12
[7] L. Freina and M. Ott, “A literature review on immersive virtual reality
in education: State of the art and perspectives,” 04 2015.
[8] B. Fu, J. Lin, L. Li, C. Faloutsos, J. Hong, and N. Sadeh, “Why people
hate your app: Making sense of user feedback in a mobile app store,” in
Proc. of the 19th ACM SIGKDD International Conference on Knowledge
Discovery and Data Mining. ACM, 2013, pp. 1276–1284.
[9] A. W. B. Furtado, A. L. M. Santos, G. L. Ramalho, and E. S. de
Almeida, “Improving digital game development with software product
lines,” IEEE Software, vol. 28, no. 5, pp. 30–37, Sep. 2011.
[10] A. R. Gagn´
e, M. Seif El-Nasr, and C. D. Shaw, “Analysis of telemetry
data from a real-time strategy game: A case study,Comput. Entertain.,
vol. 10, no. 1, pp. 2:1–2:25, Dec. 2012.
[11] E. Harpstead, T. Zimmermann, N. Nagapan, J. J. Guajardo, R. Cooper,
T. Solberg, and D. Greenawalt, “What drives people: Creating engage-
ment profiles of players from game log data,” in Proc. of the Annual
Symposium on Computer-Human Interaction in Play. ACM, 2015, pp.
369–379.
[12] S. Hassan, C. Bezemer, and A. E. Hassan, “Studying bad updates of top
free-to-download apps in the Google Play store,IEEE Transactions on
Software Engineering, pp. 1–1, 2018.
[13] J. Huang, T. Zimmermann, N. Nagappan, C. Harrison, and B. Phillips,
“Mastering the art of war: How patterns of gameplay influence skill in
Halo,” in Proc. of the International Conference on Human Factors in
Computing Systems. ACM, April 2013.
[14] K. Hullett, N. Nagappan, E. Schuh, and J. Hopson, “Data analytics for
game development (NIER Track),” in Proc. of the 33rd International
Conference on Software Engineering. ACM, 2011, pp. 940–943.
[15] D. Jack, R. Boian, A. S. Merians, M. Tremaine, G. C. Burdea, S. V.
Adamovich, M. Recce, and H. Poizner, “Virtual reality-enhanced stroke
rehabilitation,” IEEE Transactions on Neural Systems and Rehabilitation
Engineering, vol. 9, no. 3, pp. 308–318, Sep. 2001.
[16] J. Kasurinen, M. Palacin-Silva, and E. Vanhala, “What concerns game
developers?: A study on game development processes, sustainability and
metrics,” in Proc. of the 8th Workshop on Emerging Trends in Software
Metrics. IEEE, 2017, pp. 15–21.
[17] H. Khalid, E. Shihab, M. Nagappan, and A. E. Hassan, “What do mobile
app users complain about?” IEEE Software, vol. 32, no. 3, pp. 70–77,
May 2015.
[18] J. Kienzle, A. Denault, and H. Vangheluwe, “Model-based design of
computer-controlled game character behavior,” in Proc. of the 10th
International Conference on Model Driven Engineering Languages and
Systems. Berlin, Heidelberg: Springer-Verlag, 2007, pp. 650–665.
[19] X. Li, C. J. Lu, J. Peltonen, and Z. Zhang, “A statistical analysis of
Steam user profiles towards personalized gamification,” in GamiFIN,
2019.
[20] D. Lin, C.-P. Bezemer, and A. E. Hassan, “Studying the urgent updates of
popular games on the Steam platform,” Empirical Softw. Engg., vol. 22,
no. 4, pp. 2095–2126, Aug. 2017.
[21] ——, “An empirical study of early access games on the Steam platform,
Empirical Softw. Engg., vol. 23, no. 2, pp. 771–799, Apr. 2018.
[22] ——, “Identifying gameplay videos that exhibit bugs in computer
games,” Empirical Softw. Engg., Jun 2019.
[23] D. Lin, C.-P. Bezemer, Y. Zou, and A. E. Hassan, “An empirical study of
game reviews on the Steam platform,Empirical Softw. Engg., vol. 24,
no. 1, pp. 170–207, Feb. 2019.
[24] J. D. Long, D. Feng, and N. Cliff, Ordinal Analysis of Behavioral Data.
American Cancer Society, 2003, ch. 25, pp. 635–661.
[25] J. Lyons, “New today: Experience SteamVR on Windows Mixed Real-
ity,” 2017, https://steamcommunity.com/games/719950/announcements/
detail/1466347451617826043, Accessed: 2019-05-13.
[26] K. M. Malloy and L. S. Milling, “The effectiveness of virtual reality
distraction for pain reduction: A systematic review,” Clinical Psychology
Review, vol. 30, no. 8, pp. 1011 – 1018, 2010.
[27] E. Martel, F. Su, J. Gerroir, A. Hassan, A. Girouard, and K. Muldner,
“Diving head-first into virtual reality: Evaluating HMD control schemes
for VR games,” in Proc. of the 10th International Conference on the
Foundations of Digital Games, 2015.
[28] S. Mcilroy, N. Ali, H. Khalid, and A. E. Hassan, “Analyzing and
automatically labelling the types of user issues that are raised in mobile
app reviews,Empirical Softw. Engg., vol. 21, no. 3, pp. 1067–1106,
Jun. 2016.
[29] S. Mujahid, G. Sierra, R. Abdalkareem, E. Shihab, and W. Shang, “An
empirical study of Android Wear user complaints,Empirical Softw.
Engg., vol. 23, no. 6, pp. 3476–3502, Dec 2018.
[30] J. Munafo, M. Diedrick, and T. A. Stoffregen, “The virtual reality head-
mounted display Oculus Rift induces motion sickness and is sexist in
its effects,Experimental Brain Research, vol. 235, no. 3, pp. 889–901,
Mar 2017.
[31] T. Olsson, D. Toll, A. Wingkvist, and M. Ericsson, “Evolution and eval-
uation of the model-view-controller architecture in games,” in Proc. of
the Fourth International Workshop on Games and Software Engineering.
Piscataway, NJ, USA: IEEE Press, 2015, pp. 8–14.
[32] M. O’Neill, E. Vaziripour, J. Wu, and D. Zappala, “Condensing Steam:
Distilling the diversity of gamer behavior,” in Proc. of the Internet
Measurement Conference. ACM, 2016, pp. 81–95.
[33] A. Osborne O’Hagan, G. Coleman, and R. V. O’Connor, “Software
development processes for games: A systematic literature review,” in
Systems, Software and Services Process Improvement. Springer Berlin
Heidelberg, 2014, pp. 182–193.
[34] C. Politowski, L. Fontoura, F. Petrillo, and Y. Gu´
eh´
eneuc, “Are the
old days gone? a survey on actual software engineering processes in
video game industry,” in 2016 IEEE/ACM 5th International Workshop
on Games and Software Engineering (GAS), May 2016, pp. 22–28.
[35] L. Poretski and O. Arazy, “Placing value on community co-creations: A
study of a video game ‘modding’ community,” in Proc. of the Conference
on Computer Supported Cooperative Work and Social Computing.
ACM, 2017, pp. 480–491.
[36] J. Porter III and A. Robb, “An analysis of longitudinal trends in
consumer thoughts on presence and simulator sickness in VR games,” in
Proceedings of the Annual Symposium on Computer-Human Interaction
in Play (CHI PLAY). ACM, 2019, pp. 277–285.
[37] J. Porter III, M. Boyer, and A. Robb, “Guidelines on successfully porting
non-immersive games to virtual reality: A case study in Minecraft,” in
Proc. of the 2018 Annual Symposium on Computer-Human Interaction
in Play. ACM, 2018, pp. 405–415.
[38] S. Rangelova, D. Motus, and E. Andr´
e, “Cybersickness among gamers:
An online survey,” in Advances in Human Factors in Wearable Technolo-
gies and Game Design, T. Ahram, Ed. Cham: Springer International
Publishing, 2020, pp. 192–201.
[39] L. Rebenitsch and C. Owen, “Review on cybersickness in applications
and visual displays,” Virtual Real., vol. 20, no. 2, pp. 101–125, Jun.
2016.
[40] J. Romano, J. D. Kromrey, J. Coraggio, J. Skowronek, and L. Devine,
“Exploring methods for evaluating group differences on the NSSE and
other surveys: Are the t-test and Cohen’s d indices the most appropriate
choices,” in Annual Meeting of the Southern Association for Institutional
Research, 10 2006.
[41] D. Shafer, C. Carbonara, and M. Korpi, “Factors affecting enjoyment of
virtual reality games: A comparison involving consumer-grade virtual
reality technology,Games for Health Journal, vol. 8, 09 2018.
[42] K. B. Shores, Y. He, K. L. Swanenburg, R. Kraut, and J. Riedl, “The
identification of deviance and its impact on retention in a multiplayer
game,” in Proc. of the 17th Conference on Computer Supported Coop-
erative Work & Social Computing. ACM, 2014, pp. 1356–1365.
[43] R. Sifa, C. Bauckhage, and A. Drachen, “The playtime principle: Large-
scale cross-games interest modeling,” in Conference on Computational
Intelligence and Games, Aug 2014, pp. 1–8.
[44] R. Sifa, A. Drachen, and C. Bauckhage, “Large-scale cross-game player
behavior analysis on Steam,” in Proc. of the Eleventh AAAI Conference
on Artificial Intelligence and Interactive Digital Entertainment, 2015,
pp. 198–204.
[45] SuperData Research Holdings, Inc., “State of the consumer XR market,”
2019, https://www.superdataresearch.com/state-of-consumer-xr-2019/,
Accessed: 2019-06-11.
[46] C. T. Tan, T. W. Leong, S. Shen, C. Dubravs, and C. Si, “Exploring
gameplay experiences on the Oculus Rift,” in Proc. of the Annual
Symposium on Computer-Human Interaction in Play. ACM, 2015,
pp. 253–263.
[47] Valve Corporation, “Steam hardware & software survey: June 2019,”
2019, https://store.steampowered.com/hwsurvey, Accessed: 2019-07-19.
[48] M. Virvou and G. Katsionis, “On the usability and likeability of virtual
reality games for education: The case of VR-ENGAGE,Comput. Educ.,
vol. 50, no. 1, pp. 154–178, Jan. 2008.
[49] M. Walch, J. Frommel, K. Rogers, F. Sch¨
ussel, P. Hock, D. Dobbelstein,
and M. Weber, “Evaluating VR driving simulation from a player
experience perspective,” in Proc. of the CHI Conference Ext. Abstracts
on Human Factors in Computing Systems. ACM, 2017, pp. 2982–2989.
[50] G. Wallner, “Sequential analysis of player behavior,” in Proc. of the
Annual Symposium on Computer-Human Interaction in Play. ACM,
2015, pp. 349–358.
[51] W. X. Zhao, J. Jiang, J. Weng, J. He, E.-P. Lim, H. Yan, and X. Li,
“Comparing Twitter and traditional media using topic models,” in Proc.
of the 33rd European Conference on Advances in Information Retrieval.
Berlin, Heidelberg: Springer-Verlag, 2011, pp. 338–349.
... Over the years, numerous studies related to web-based digital game distribution platforms have been published, including the social aspects of the gaming community [10], mining sentiment analysis [11], gamer's archetypes profiling [12], user reviews on virtual reality games [13], purchasing patterns [14], and game metadata information [9]. However, one of the overlooked aspects is the impact of digital game visual styles on user satisfaction when browsing game products. ...
Article
Full-text available
The digital gaming community appreciates the visual style classification system to search for information about a game product. However, scholars have discovered that the applied visual style classification system frequently leads to frustration and dissatisfaction among users owing to inaccurate search results. Moreover, there are yet any research studies that accurately measure user satisfaction during game searching activity in the digital game library based on the current visual style classification system. Therefore, this study was performed to investigate the influence of information quality components (Accuracy, Content, Ease of Use, Format, and Timeliness) on User Satisfaction during game searching activity in the digital game library. A cross-sectional study was conducted by distributing self-administered digital questionnaires to 239 game players in Malaysia. The 12-item instrument survey questionnaire, which consists of four main sections, adopted the End-user Computing Satisfaction (EUCS) model to examine the relationships between user satisfaction and information quality components. The results were then analysed using descriptive statistics, preliminary data analysis, and Confirmatory Factor Analysis (CFA). Based on the findings, more than half of the respondents were male (n = 139, 58.2%) with an age range between 19 and 24 years old, and 79.1% (n = 189) have experience in searching digital games based on visual styles. The EUCS model revealed that user satisfaction during game searching activity using the visual style information highly correlated with Content, Accuracy, and Ease of Use. Users were satisfied when they receive accurate, precise, and sufficient information. In addition, a user-friendly and simplified navigation interface improved the searching experience and stimulate users to further their searching activity. In contrast, the Format and Timeliness showed a weak correlation. Providing visual classification and appeal format has less impact on user satisfaction. In addition, the fast speed of information retrieval and up-to-date information showed an insignificant contribution to user satisfaction. Overall, this research demonstrated that information quality components, namely Content, Accuracy, and Ease of Use, influenced User Satisfaction when searching for games based on the visual style. Future studies should explore and evaluate the effectiveness of visual style information systems for digital game distribution platforms in Malaysia.
... Epp concludes common VR headsets are widely adopted in VR games, such as Oculus Rift, HTC Vive, and Windows Mixed Reality. They could help enter the virtual games more easily and improve the quality of VR games to a large extent [27]. Considering the advances in interactive technologies, it would be much more convenient for users to enter the Metaverse and enjoy fantastic services in the coming days. ...
Preprint
The Metaverse depicts a parallel digitalized world where virtuality and reality are fused. It has economic and social systems like those in the real world and provides intelligent services and applications. In this paper, we introduce the Metaverse from a new technology perspective, including its essence, corresponding technical framework, and potential technical challenges. Specifically, we analyze the essence of the Metaverse from its etymology and point out breakthroughs promising to be made in time, space, and contents of the Metaverse by citing Maslow's Hierarchy of Needs. Subsequently, we conclude four pillars of the Metaverse, named ubiquitous connections, space convergence, virtuality and reality interaction, and human-centered communication, and establish a corresponding technical framework. Additionally, we envision open issues and challenges of the Metaverse in the technical aspect. The work proposes a new technology perspective of the Metaverse and will provide further guidance for its technology development in the future.
... A standard minimum space for a comfortable experience is two meters by one and a half meters (Lynch, 2021;Liu, 2015). According to (Epp et al., 2021) "The most common play areas are room-scale and standing, supported by 79% and 76% of games respectively. Support for the seated play area is increasing." ...
Thesis
Full-text available
Tactical first-person shooter games are trending in the world of virtual reality. Virtual reality offers different movement and view possibilities compared to the traditional personal computer first-person shooter games. Therefore, virtual reality games require different design choices, which create new playing experiences. These different design choices raised questions about their effectiveness; hence, a comparison was made on how players play a tactical first-person shooter on the personal computer and in virtual reality. Two separate groups of participants played ten modified situations on the same map in the game Counter Strike: Global Offensive and Pavlov virtual reality. A mixed-method approach was utilized where the gameplay and real-life movements in virtual reality were recorded and analysed according to Mirweis Sangin in the book Games User Research by Drachen et al. (2018) and a semi-structured post-interview was conducted and thematically analysed to gather an in-depth explanation. The data resulted in a clear difference in the movement of the body in virtual reality compared to the personal computer. Aiming with virtual reality controllers resulted in complications compared to aiming with the mouse on a personal computer. Reloading proved to be a harder skill to master for virtual reality players compared to personal computer players. Furthermore, reloading and switching had some disadvantages for new virtual reality players compared to personal computer players. Suggestions were offered to improve the design in virtual reality, which can be used by game designers when developing a tactical first-person shooter game or simulation for virtual reality.
... Virtual reality (VR) has become increasingly established in the everyday lives of private persons in recent years (Epp et al., 2021). However, VR technology is also becoming more popular in industrial applications, such as in construction (e.g. ...
Conference Paper
Full-text available
The advent of sophisticated virtual reality (VR) technologies has the potential to transform the way organizations design their workplaces and processes. The evaluation of artifacts for this use case is mostly limited to short-term effects at the task level. This paper therefore aims to unfold the benefits of using VR for workplace and process design in the long run at the division and corporate level. For this purpose, we applied a mixed methods approach. First, we identified benefits by using a VR software artifact for workplace and process design in field studies. Second, we extended the initial benefits by applying utility effect chains resulting in benefits at the division and corporate level. The resultant utility effect chain model reveals that, in addition to functional benefits, the hedonistic user experience of VR can also provide economic benefits for organizations in the long term.
... Notably, Foxman et al. [8] analyzed common game genres among virtual reality (VR) games and their user ratings by crawling for the number of owners and ratings from a third-party API and the Steam platform. Similarly, Epp et al. [7] gathered information such as price history and reviews from Steam and third-party websites to study the trends from VR games and their associated reviews. Li and Zhang [18] crawled Steam to present a new approach to video game genre classification and to analyze connections between different game genres. ...
Preprint
The gaming industry is rapidly expanding. With over 2.7 billion players worldwide, game development has become increasingly challenging. To meet the ever-changing demands and expectations of players and due to unforeseen hindrances in the development process, game developers may require to delay the release of their game. We conducted an empirical study of 23,485 games on the Steam platform to analyze how often, and which games delayed their release date. We find that delaying a release is common: 48% of the studied games had a delayed initial release. Games delayed their release by a median of 14 days. Games for which a release date range (e.g., "Q1 2019") was specified, rather than a specific date were more likely to release within that range. Across different game genres, the percentage of games that delay release is similar (ranging from 48% to 52%). Finally, games with a delayed release are rated lower than games that release on time, but the difference is negligible.
... Such add-ons do not neatly fit into most content-based classification systems. And this discrepancy can affect public perception; for instance, Epp et al. 's investigation of consumer VR complaints highlights that most experiences did not live up to expectations regarding content [15]. Opinions may shift with more comprehensive classification systems and standards. ...
Article
Full-text available
Although virtual reality (VR) shares common features with video games, it offers different experiences for users based on the mediums unique affordances. Still, there is no cohesive classification system for commercial VR applications. To account for this deficiency, the paper presents an exploratory comparative analysis of titles from the Steam digital store across three platform types: VR only, VR supported, and non-VR. We analyzed data from a subset of the most popular applications within each (N=141, 93, and 1217, respectively). These three classification types were academic game genres, developer defined categories, and user-denoted tags. Results identify the most common content classifications (e.g., Action and Shooter within VR only applications), the relative availability of each classification between platforms (e.g., Casual is more common in VR only than VR supported or non-VR), general platform popularity (e.g., VR only received less positive ratings than VR supported and non-VR), and which content classifications are associated with higher user ratings across platforms (e.g., Action and Music/Rhythm are most positively rated in VR only). These findings provide a novel snapshot of the state of VR applications and lay the foundation for future theoretical constructions of classification systems based on content, market, interactivity, sociality, and service dependencies.
... For example, Ray et al. (Ray et al. 2014) used the topics extracted from feature descriptions on GitHub to identify project domains, while Lukins et al. (Lukins et al. 2010) used an LDA-based approach for automatic bug localization in open-source software. Other have used LDA to categorize and detect duplicate bug reports (Somasundaram and Murphy 2012; Budhiraja et al. 2018), and analyze reviews for mobile apps (Hu et al. 2019) and games (Epp et al. 2021). ...
Article
Full-text available
The game development industry is growing, and training new developers in game development-specific abilities is essential to satisfying its need for skilled game developers. These developers require effective learning resources to acquire the information they need and improve their game development skills. Question and Answer (Q&A) websites stand out as some of the most used online learning resources in software development. Many studies have investigated how Q&A websites help software developers become more experienced. However, no studies have explored Q&A websites aimed at game development, and there is little information about how game developers use and interact with these websites. In this paper, we study four Q&A communities by analyzing game development data we collected from their websites and the 347 responses received on a survey we ran with game developers. We observe that the communities have declined over the past few years and identify factors that correlate to these changes. Using a Latent Dirichlet Allocation (LDA) model, we characterize the topics discussed in the communities. We also analyze how topics differ across communities and identify the most discussed topics. Furthermore, we find that survey respondents have a mostly negative view of the communities and tended to stop using the websites once they became more experienced. Finally, we provide recommendations on where game developers should post their questions, which can help mitigate the websites’ declines and improve their effectiveness.
Conference Paper
Full-text available
Gamification is widely used as motivational design towards enhancing the engagement and performance of its users. Many commonly adopted game design elements have been verified to be effective in various domains. However, the designs of such elements in the majority of the target systems are similar. Due to inevitable differences between users, gamification systems can perform more effectively when users are provided with differently and personally designed features according to their preferences. Many studies have suggested such requirements towards personalizing gamified systems based on the users' preferences, with categorizing gamification users and identifying their preferences as the initial step. This study proposes a preliminary analysis of the factors that categorize user preference in a game community, based on the user profiles data of the Steam platform. It shall not only facilitate understanding of players' preferences in a game community but also lay the groundwork for the potential personalized gamification design.
Conference Paper
Full-text available
Since the release of the Oculus Rift CV1 in 2016, millions of VR headsets have found their way into consumer homes. In this paper, we sought to understand what shifts have taken place within the two years since consumer VR became available. In this paper, we consider what can be learned about long-term use of consumer VR through an analysis of discussions in online forums devoted to VR. We gathered posts made on the /r/Vive subreddit from the first two years after the HTC Vive's release. We present the results from an in-depth qualitative analysis concerning immersion, presence, and simulator sickness. Over time, as users moved from passive to active, their attitudes and expectations towards immersion and simulator sickness matured. Major trends of interest found were game design implementation and locomotion techniques.
Article
Full-text available
With the rapid growing market and competition in the gaming industry, it is challenging to develop a successful game, making the quality of games very important. To improve the quality of games, developers commonly use gamer-submitted bug reports to locate bugs in games. Recently, gameplay videos have become popular in the gaming community. A few of these videos showcase a bug, offering developers a new opportunity to collect context-rich bug information. In this paper, we investigate whether videos that showcase a bug can automatically be identified from the metadata of gameplay videos that are readily available online. Such bug videos could then be used as a supplemental source of bug information for game developers. We studied the number of gameplay videos on the Steam platform, one of the most popular digital game distribution platforms, and the difficulty of identifying bug videos from these gameplay videos. We show that naive approaches such as using keywords to search for bug videos are time-consuming and imprecise. We propose an approach which uses a random forest classifier to rank gameplay videos based on their likelihood of being a bug video. Our proposed approach achieves a precision that is 43% higher than that of the naive keyword searching approach on a manually labelled dataset of 96 videos. In addition, by evaluating 1,400 videos that are identified by our approach as bug videos, we calculated that our approach has both a mean average precision at 10 and a mean average precision at 100 of 0.91. Our study demonstrates that it is feasible to automatically identify gameplay videos that showcase a bug.
Conference Paper
Full-text available
Virtual reality games have grown rapidly in popularity since the first consumer VR head-mounted displays were released in 2016, however comparatively little research has explored how this new medium impacts the experience of players. In this paper, we present a study exploring how user experience changes when playing Minecraft on the desktop and in immersive virtual reality. Fourteen players completed six 45 minute sessions, three played on the desktop and three in VR. The Gaming Experience Questionnaire, the i-Group presence questionnaire, and the Simulator Sickness Questionnaire were administered after each session, and players were interviewed at the end of the experiment. Participants strongly preferred playing Minecraft in VR, despite frustrations with using teleporation as a travel technique and feelings of simulator sickness. Players enjoyed using motion controls, but still continued to use indirect input under certain circumstances. This did not appear to negatively impact feelings of presence. We conclude with four lessons for game developers interested in porting their games to virtual reality.
Article
Full-text available
Objective: This study investigated psychological responses to playing videogames using a virtual reality (VR) head-mounted display (HMD). We also investigated how cybersickness impacts the sense of presence one feels in the virtual environment, as well as how cybersickness affects enjoyment. Materials and methods: Participants played randomly assigned VR games that varied in the level of sensory conflict they provided: "Lucky's Tale," "Elite: Dangerous," and "Minecraft." Results were compared based on two headset conditions-the Oculus DK2 and the recently released Oculus Rift Consumer Version (CV1). Results: Cybersickness was not reduced by playing games with a VR HMD of higher technological quality-the Oculus Rift CV1. Furthermore, cybersickness responses were significantly different based on the level of sensory conflict in the games. Games with less sensory conflict, such as third-person platformer games, or space and flight simulator games, produce less cybersickness in players than first-person games. Enjoyment of VR games was shown to be directly influenced by a sense of spatial presence, which was affected by interactivity and realism. Findings suggest that the variables that are important to the enjoyment of console, mobile, or motion-based games are consistently important to enjoyment of VR games. Conclusion: Better technology does not affect the frequency or severity of cybersickness for players; but sensory conflict has a significant impact on how sick users become. Additionally, we present a model that indicates how enjoyment is produced in VR gaming experiences. These findings bear further investigation as new methods of interacting with VR games are developed.
Article
Full-text available
Developers always focus on delivering high-quality updates to improve, or maintain the rating of their apps. Prior work has studied user reviews by analyzing all reviews of an app. However, this app-level analysis misses the point that users post reviews to provide their feedback on a certain update. For example, two bad updates of an app with a history of good updates would not be spotted using app-level analysis. In this paper, we examine reviews at the update-level to better understand how users perceive bad updates. We focus our study on the top 250 bad updates (i.e., updates with the highest increase in the percentage of negative reviews relative to the prior updates of the app) from 26,726 updates of 2,526 top free-to-download apps in the Google Play Store. We find that feature removal and UI issues have the highest increase in the percentage of negative reviews. Bad updates with crashes and functional issues are the most likely to be fixed by a later update. However, developers often do not mention these fixes in the release notes. Our work demonstrates the necessity of an update-level analysis of reviews to capture the feelings of an app's user-base about a particular update.
Article
Full-text available
Based on a very large dataset of over 100 million Steam platform users we present the first comprehensive analysis of hardcore gamer profiles by over 700,000 hardcore players (users playing more than 20 hours per week) covering more than 3,300 games. Using an unsupervised machine learning approach we reveal the specific behavioral categories of hardcore players, i.e. First Person Shooter, Team Fortress 2 player, Action game player, Dota 2 player, Strategy and action combiner, Genre-switching player. Subsequently we derive individual patterns of hardcore gamers in the categories found, such as strategy-action games combiner or game switching players. Our results are useful for computer science and information systems scholars interested in individual differences in user behavior as well as practitioners interested in game-designing.
Article
Full-text available
The steadily increasing popularity of computer games has led to the rise of a multi-billion dollar industry. Due to the scale of the computer game industry, developing a successful game is challenging. In addition, prior studies show that gamers are extremely hard to please, making the quality of games an important issue. Most online game stores allow users to review a game that they bought. Such reviews can make or break a game, as other potential buyers often base their purchasing decisions on the reviews of a game. Hence, studying game reviews can help game developers better understand user concerns, and further improve the user-perceived quality of games. In this paper, we perform an empirical study of the reviews of 6,224 games on the Steam platform, one of the most popular digital game delivery platforms, to better understand if game reviews share similar characteristics with mobile app reviews, and thereby understand whether the conclusions and tools from mobile app review studies can be leveraged by game developers. In addition, new insights from game reviews could possibly open up new research directions for research of mobile app reviews. We first conduct a preliminary study to understand the number of game reviews and the complexity to read through them. In addition, we study the relation between several game-specific characteristics and the fluctuations of the number of reviews that are received on a daily basis. We then focus on the useful information that can be acquired from reviews by studying the major concerns that users express in their reviews, and the amount of play time before players post a review. We find that game reviews are different from mobile app reviews along several aspects. Additionally , the number of playing hours before posting a review is a unique and helpful attribute for developers that is not found in mobile app reviews. Future longitudinal studies should be conducted to help developers and researchers leverage this information. Although negative reviews contain more valuable information about the negative aspects of the game, such as mentioned complaints and bug reports, developers and researchers should also not ignore the potentially useful information in positive reviews. Our study on game reviews serves as a starting point for other game review researchers, and suggests that prior studies on mobile app reviews may need to be revisited.
Chapter
In recent years a trend of head-mounted displays (HMDs) appears on the consumer market and it expands the entertainment dimension by adding a new segment to it called virtual reality (VR) gaming. However, VR games not only bring enjoyment to the players, but they also bring discomfort called cybersickness. In this study, an online survey among gamers was used to identify whether and which symptoms of cybersickness occur while playing VR games. The results showed that the most experienced symptoms during or after VR exposure regarding gaming are nausea, fatigue, and general discomfort. Additionally, the paper aims to give insights about what gamers do against cybersickness associated with VR games. The findings showed that the side effects of VR games are widespread among gamers and an appropriate solution has not been found yet. This paper is a starting point for more extended research on cybersickness induced by VR games.