Content uploaded by Dayi Lin
Author content
All content in this area was uploaded by Dayi Lin on Jun 16, 2017
Content may be subject to copyright.
Noname manuscript No.
(will be inserted by the editor)
An Empirical Study of Early Access Games on the Steam
Platform
Dayi Lin ·Cor-Paul Bezemer ·Ahmed E.
Hassan
Received: date / Accepted: date
Abstract “Early access” is a release strategy for software that allows consumers to
purchase an unfinished version of the software. In turn, consumers can influence the
software development process by giving developers early feedback. This early access
model has become increasingly popular through digital distribution platforms, such
as Steam which is the most popular distribution platform for games. The plethora
of options offered by Steam to communicate between developers and game players
contribute to the popularity of the early access model. The model is considered a
success by the game development community as several games using this approach
have gained a large user base (i.e., owners) and high sales. On the other hand, the
benefits of the early access model have been questioned as well.
In this paper, we conduct an empirical study on 1,182 Early Access Games (EAGs)
on the Steam platform to understand the characteristics, advantages and limitations
of the early access model. We find that 15% of the games on Steam make use of the
early access model, with the most popular EAG having as many as 29 million own-
ers. 88% of the EAGs are classified by their developers as so-called “indie” games,
indicating that most EAGs are developed by individual developers or small studios.
We study the interaction between players and developers of EAGs and the Steam
platform. We observe that on the one hand, developers update their games more fre-
quently in the early access stage. On the other hand, the percentage of players that
review a game during its early access stage is lower than the percentage of players
that review the game after it leaves the early access stage. However, the average rating
of the reviews is much higher during the early access stage, suggesting that players
are more tolerant of imperfections in the early access stage. The positive review rate
does not correlate with the length or the game update frequency of the early access
stage.
Based on our findings, we suggest game developers to use the early access model
as a method for eliciting early feedback and more positive reviews to attract additional
Dayi Lin ·Cor-Paul Bezemer ·Ahmed E. Hassan
Queen’s University, Canada
E-mail: dayi.lin@cs.queensu.ca, bezemer@cs.queensu.ca, ahmed@cs.queensu.ca
2 Dayi Lin et al.
new players. In addition, our findings suggest that developers can determine their
release schedule without worrying about the length of the early access stage and the
game update frequency during the early access stage.
Keywords early access games ·computer games ·Steam
1 Introduction
Every year, 70% of the software development projects do not deliver the expected
product [19], despite the expenditure of $275 billion on software development projects
in the U.S. alone [18]. The failures include total failures, time and budget overruns,
and unmet user requirements [19].
In order to prevent the problems of overrunning budget and time, and to better
meet the user requirements, a public beta-release release strategy is commonly used
by software developers. As early as in 1984, a “pioneer edition” of the WordVision
word processor for the IBM PC was available for early customers to purchase [29].
Microsoft launched “Office Insider” program in late 2015, which allows customers
to get early access to the latest Office features and provide feedback [13]. Another
example is the game Minecraft. Available since 2009, Minecraft stayed in public beta
until 2011 [33]. As the sales of the public beta version increased, its developer was
able to quit his day job to work on Minecraft full-time [31]. During the beta stage,
Minecraft raised over $33 million from the public beta sales while accumulating over
1.8 million players [35].
Inspired by the successful application of the public beta release strategy in games
such as Minecraft, Steam, the dominant digital game delivery platform, started to
offer game developers the opportunity to release their games as public betas in March
2013. These so-called “Early Access Games” (EAGs), allow customers to purchase
the public beta version of a game while developers continue working on the game
using the raised funds. Developers of EAGs receive crucial feedback and bug reports
directly from their target community, while players have the opportunity to be the first
to play new games and get involved with games as they evolve. Hence, as declared
by Steam, early access is “the way games should be made” [46].
The early access model made a name for itself through several successful games,
such as the DayZ game1. The multiplayer survival-based game reached 400,000 sales
during its first week as an EAG, according to its developer Bohemia [11].
However, the benefits of the early access model have been questioned as well.
For instance, the Spacebase DF-9 game2abandoned the early access stage unexpect-
edly as the funds raised during that early access stage were not sufficient to continue
the development process. As a result, many promised features were left unimple-
mented, disappointing many players of the game. The game received 77% negative
reviews [47]. Shortly after abandoning the early access stage and terminating the
development, twelve employees were laid off including the programmer and project
lead [23]. The developer stated that all funds raised during the early access stage went
1http://store.steampowered.com/app/221100/
2http://store.steampowered.com/app/246090/
An Empirical Study of Early Access Games on the Steam Platform 3
into the development of the game, but that eventually the studio was spending more
than it was making [38].
Along with the aforementioned failure of the early access model, the debate of
whether early access is as good as expected has been raised. One year after the release
of the Steam Early Access Release Platform, Walker [49] calculated that only 25% of
the EAGs have left the early access stage. Recently, Allen [2] manually investigated
the first 50 released EAGs and warned people that the early access model may be “a
ticking time bomb”, as the development of 32 (64%) of the first 50 EAGs is either
abandoned or inactive. Allen states that the early access model currently has a bad
reputation and is leading games to a “Development Hell”, and calls for a systematic
in-depth study of all EAGs to examine the opportunities and risks that lie behind the
early access model.
In order to get a better understanding of the impact and limitations of the early
access model, we conduct such an in-depth empirical study on EAGs on the Steam
platform. The study aims at providing developers with the characteristics of the early
access model, the degree of interaction between developers and players of EAGs with
the Steam platform, and the tolerance of players of the quality of EAGs. Additionally,
based on these results, we provide suggestions for developers to make best use of this
novel release strategy. In particular, we address the following three research questions
(RQs):
RQ1: What are the characteristics of the early access model?
Currently, 15% of the games on Steam use the early access model. The early
access model tends to appeal mostly to individuals or small studios for releasing
their indie games. However, using the early access model is not a guarantee for
collecting enough funds to continue the development of a game.
RQ2: How do developers and players of EAGs interact with the Steam plat-
form?
Developers update a game more frequently during its early access stage. Players
post less reviews, however players have more discussion posts in the early access
stage.
RQ3: How tolerant are players of the quality of EAGs?
Players of EAGs tend to be more tolerant of the quality of a game during its early
access stage. While players tend to post less reviews within the early access stage,
89% of EAGs receive an equally or more positive review rate in their early access
stage. In addition, developers do not need to rush into releasing their games, as the
tolerance of players does not correlate with the length of the early access stage.
Paper Organization. The rest of the paper is organized as follows. Section 2
provides the background of our study and discusses related work. Section 3 explains
our study methodology. Section 4 presents the results of our study. Section 5 discusses
several additional interesting insights that we came across during our study. Section 6
discusses threats to the validity of our study. Finally, Section 7 concludes the paper.
4 Dayi Lin et al.
2 Background
This section describes the Steam gaming platform, the mechanism of Steam’s early
access release platform, the differences between crowd-funding and early access, and
the related work.
2.1 Steam Gaming Platform
Steam is a digital game distribution platform developed by Valve Software. Over
8,000 games are distributed through Steam and the platform has over 161 million
active players [15]. The Steam platform consists of two major components: the Steam
Store and the Steam Community. Players can purchase and download games through
the Steam Store and interact with other players and game developers through the
Steam Community.
After playing a game through Steam, players are able to post a review for that
game on its Steam Community page. Different from most application distribution
platforms, e.g., mobile app stores, instead of the star-rating mechanism, players are
asked to provide their overall feeling for the game: “Recommended” (i.e., a positive
review), or “Not Recommended” (i.e., a negative review). Additionally, each review
can be upvoted as “helpful”, “not helpful”, or “funny” by other players. The posi-
tive review rate ( #o f recommended reviews
#o f all reviews ), and the reviews that are upvoted most as
“helpful” are displayed on the Steam Store to advise potential customers.
In addition to the review mechanism, the Steam Community provides a discus-
sion forum for each game in which players and developers can communicate. The
forum of a game can have a variety of subforums that are created by developers. By
default, a forum contains two subforums that are created by Steam, which are Gen-
eral Discussions and Trading. The Trading subforum is specifically for players to
trade in-game properties, such as rare weapons, while General Discussions normally
contains threads regarding bug reports, suggestions, questions, etc.
The Steam Community also provides functionality for developers and journalists
to publish news updates for games on so-called channels. In general, developers post
announcements about game updates to one or more channels, e.g., to the Product
Update channel. Because it is mandatory to install the latest game updates for players
of a game on Steam, developers may opt to update their game silently. Nevertheless,
in order to keep players aware of the latest news about a game, developers tend to
post news updates whenever a new update is released.
2.2 Steam Early Access Release Platform (SEARP)
The SEARP was launched on March 20, 2013, with 12 game titles available ini-
tially [51]. The platform allows developers to release unfinished, yet playable games,
so-called Early Access Games (EAGs). By purchasing an EAG, players are allowed
to download and play that game in its current state and as it evolves, even after the
game leaves the early access stage.
An Empirical Study of Early Access Games on the Steam Platform 5
The SEARP provides developers with early access sales and distribution mecha-
nisms. The developers of EAGs have the freedom to determine when to move a game
out of the early access stage. In addition, developers have the freedom to increase
or decrease the price of their game at any time. Players are aware of the risk that a
game may be incomplete, buggy, or unfinished when purchasing an EAG. All reviews
posted during the early access stage of a game are tagged as “early access review”,
hence they can be distinguished from the reviews that are posted after leaving the
early access stage.
2.3 Crowdfunding vs. Early Access
Crowdfunding is the practice of funding a project or venture by raising small amounts
of money from a large number of supporters, typically via the Internet [10]. Many
games use a type of crowd-funding model called “Reward Crowdfunding” to support
the game development costs, by which the developers pre-sell the product to launch
the project without incurring debt [7].
There exist similarities between the crowdfunding and early access model, as
both models raise funds by selling products before their completion. However, the
differences between early access and crowdfunding are worth noting. Although many
crowd-funded games promise to offer access to alpha or beta versions of the game, no
playable version usually exists during the initial crowd-funding campaign. All Steam
EAGs offer an immediately playable version of the unfinished game to customers.
However, in both models paying customers take the risk that they may never see a
final release of the game.
It is worth noting that in order to minimize the risk, Valve (the company to which
Steam belongs) tightened the SEARP rules for developers on November, 2014, stat-
ing that SEARP is “meant to be a place for games that are in a playable alpha or beta
state, are worth the current value of the playable build, and the developer plans to
continue to develop for release” [52]. The newly added rules include “Don’t launch
in Early Access if you can’t afford to develop with very few or no sales” and “Make
sure you set expectations properly everywhere you talk about your game”, which
seem to directly target the failure of the aforementioned Spacebase DF-9 game, a
month before releasing the new rules. We further discuss the learnt lessons from the
Spacebase DF-9 game failure in Section 5.
2.4 Related Work
In this section, we discuss prior research related to our study. Most of the work that
is related to our study focuses on early releases in software or on user involvement in
software development.
2.4.1 Mining Digital Distribution Platforms
Most of the work about mining digital distribution platforms focuses on mining mo-
bile app stores. Martin et al. [30] survey the field of app store analysis within a soft-
6 Dayi Lin et al.
ware engineering context. They observe an increasing size of the studied app samples
and a diverse set of techniques and applications in app store analysis, highlighting the
health and future potential of the field.
Mining data from digital gaming platforms is an area that has been gaining atten-
tion recently. In our previous work [24], we study urgent updates of popular games
on the Steam platform. One of our major findings is that the update strategy that is
chosen by a game developer affects the number of urgent updates that are released.
Chambers et al. [5] analyze two years of game traffic on several gaming platforms,
including Steam. They demonstrate the difficulty of providing enough resources at
launch time of a game and they show that gamers are extremely difficult to please.
Several game blogs explore the potential risk of SEARP. Walker [49] points out
that only 25% of the EAGs are released as a full game (by November 2014). Allen [2]
manually goes through the first 50 games released on SEARP, and finds that 20 games
(40%) have not had an update in the last 3 months (as of November 2nd, 2016).
Our work re-examines most claims in the aforementioned game blog posts with
a newer and larger dataset, and explores the topic with more depth and greater rigor.
2.4.2 Beta Releases in Software
Several studies regarding the perpetual beta (i.e., where the product is developed
in the open, with new features added on a monthly, weekly, or even daily basis) in
software have been done. O’reilly [34] points out that one of the fundamental changes
in the software release cycle in Web 2.0 is the use of the perpetual beta in which users
are treated as co-developers. Ullrich et al. [45] states that the perpetual beta increases
the value a user gets from using the service. Developers using the perpetual beta
release model are interested in feedback and are open to suggestions.
Al-Ani et al. [1] find that traditional software development models either impose
too tight (i.e., costly and infeasible) or too loose (i.e., not efficient) restrictions on
user participation in the development process. They suggest a continuous form of
participation is the most efficient form of participation. Maalej et al. [28] propose a
continuous and context-aware approach for communicating user input to engineering
teams.
Our study is one of the first to study beta releases (i.e. the early access model) in
games.
2.4.3 Interaction between Users and Developers
Several studies on interaction between users and developers exist in literature. One
of the topics is about the participatory design in games. Jacobs et al. [17] distinguish
two forms of participatory design between players and developers that are commonly
implemented: direct participatory design (connecting with a small number of highly
active players) and silent participatory design (silently log all actions from all play-
ers). Jacobs et al. use the example of Facebook games that are developed by the Zynga
company to show that these two forms can be implemented in a perpetual beta. How-
ever, Jacobs et al. warn that once the game development is centered around player
An Empirical Study of Early Access Games on the Steam Platform 7
Table 1: Dataset description
# of games 8,025
# of EAGs 1,182
# of current EAGs 786
# of former EAGs 396
# of news updates 104,236
# of release notes 38,249
# of EAG news updates 31,916
# of EAG release notes 16,780
# of reviews 12,338,364
# of early access reviews 1,564,574
# of discussion posts of former EAGs 801,128
feedback, in the end, the game environment will become unbalanced as players only
design the game from a player perspective (wanting what is scarce in the game).
L¨
owgren et al. [27] claim that participatory design is a mutual learning process
between users and designers and it is not only users participating in design, but also
designers participating in use. Taylor [42] explores relationships between players and
developers of massively multiplayer online games (MMOG). L¨
owgren et al. state that
“at the heart of games is a complex negotiation between what the player might like
to do and what they must or should do.”
Other examples include user involvement in software development. Kujala [20]
conducted a study of the benefits and challenges of user involvement. The study
claimed that user involvement generally has a positive effect, especially on user sat-
isfaction. However, the role of users must be carefully considered, as developers and
users tend to have difficulties in communicating, and user groups may have conflicts.
Damodaran [9] provide guidelines for user involvement in the system design pro-
cess. Gallivan et al. [14] proposed a process model that delineates the four stages of
communication between users and software developers, and advised researchers and
practitioners on how to leverage the potential benefits of user participation, rather
than take the benefits for granted.
The early access model has a potential to improve user involvement in game de-
velopment. This paper makes an initial step by exploring how users and game devel-
opers interact with the Steam platform.
3 Methodology
This section introduces the methodology of our empirical study of EAGs. We detail
how we extract and process data. Table 1 presents the description of our collected
dataset. Figure 1 gives an overview of our methodology.
3.1 Collecting Basic Game Information
We develop a customized crawler to take a snapshot of all the 8,025 games that are
available in the Steam Store on March 7th, 2016. We collect the title, developer, pub-
8 Dayi Lin et al.
Collecting Release Notes,
User Reviews and Discussions
Collecting Basic Game Information
Collecting Historical Data
Steam Store
Extract basic
info of all
games
Extract news
updates
Steam
Community
Extract
release notes
Extract
historical
owner data
Basic info of
all available
games
Release
notes
Historical
owner
data
Extract
discussions Discussions Characteristics
of EAGs
(Section 4.1)
Steam DB
Interactions of
EAGs
(Section 4.2)
User tolerance
of EAGs
(Section 4.3)
Steam Spy
Extract
historical
price data
Historical
price
data
Identifying EAGs
Identify
Current/
Former EAGs
Current/
Former
EAGs
Extract user
reviews User reviews
Price change of
EAGs
(Section 5.1)
Lessons from
a failure
(Section 5.2)
BI
UR
D
RN
HO
HP
EAG
EAG
HO
RN
D
BI
UR
EAG
RN
D
HO
EAG
UR
RN
EAG
HP
RN
D
Fig. 1: Overview of our study
lisher, tags, genres, and current early access status (i.e. whether the game is in the
early access stage or not) of each game. The tags of a game are specified by its play-
ers, while the genres of a game are specified by its developer.
An Empirical Study of Early Access Games on the Steam Platform 9
Table 2: Release note for the Team Fortress 2 game
Title Team Fortress 2 Update Released
Channel Product Updates
Date 12 Oct, 2015
An update to Team Fortress 2 has been released. The update will be applied
automatically when you restart Team Fortress 2. The major changes include:
- Fixed a client crash related to the contract menu.
- Fixed an issue where some players could not use some of the crafting recipes
- Running in textmode now places the client in insecure mode
- Updated the localization files
3.2 Collecting Release Notes, User Reviews and Discussions
In order to study the update frequency of games, we use the accompanying release
notes that are posted on channels in the Steam Community. We use the process de-
scribed in our previous work [24] to extract release notes from the channels. We
briefly describe the process below.
Although the Steam Community provides developers with a special channel named
Product Updates for release notes, we observe that many release notes are posted on
other channels, e.g., Community Announcements. To avoid missing any release notes,
we extracted all news updates on all channels for all games. We observe that the
channels for online magazines that review games are irrelevant to release notes, and
official game blogs that cross-post patch notes are redundant. Hence, we only keep
the release notes that are posted on the Client Updates,Product Releases,Product
Updates and Steam Community Announcements channels for further analysis.
All news updates posted on all the Steam Community channels of a game are
aggregated in the “Related news” page of the Steam Store3. These news updates
include information such as game announcements, promotions, and release notes.
Table 2 shows an example of a release note for the Team Fortress 2 game4.
We extracted all 104,236 news updates for all available games on March 7th, 2016
using a custom-written crawler, and perform the following steps to extract release
notes from all news updates.
1. We keep all news updates that are posted on the Product Release or Product
Update channel.
2. We remove all news updates of which the title does not contain the words update,
release,patch,hotfix,change log OR a version number.
3. The news updates that remain, together with the news updates from step 1 are
considered as release notes.
We identified 38,249 release notes for all 8,025 games. In order to validate the
precision and recall of our extraction steps, we manually analyze a statistically rep-
resentative sample of 383 news updates (95% confidence level and 5% confidence
interval, taken from 104,236 news updates for the studied games). The precision is
3E.g., related news for the Dota 2 game: http://store.steampowered.com/news/?appids=570
4http://store.steampowered.com/app/440/
10 Dayi Lin et al.
calculated as #of correctly identi f ied release notes in t he sampl e
#o f ident i f ied rel ease notes in the sample , and the recall is calculated
as #o f correctl y ident i f ied release not es in the sampl e
#o f release not es in the sample . The manual analysis shows that our ex-
traction steps have a precision of 89% and a recall of 87%.
We extract all the reviews for each game from the Steam Community. There are
in total 12,338,364 reviews across all supported natural languages. We also extract
all the threads from the discussion forums on the Steam Community for all EAGs
that have left the early access stage. We extract discussion posts for EAGs that have
left the early access stage only, because doing so allows us to study the difference
in interaction between players and the Steam platform through the discussion forums
within and after leaving the early access stage. We extract the discussion posts (i.e.,
a message by a user or developer) from all the subforums except for the Trading
subforum, because the discussion posts in Trading do not contain player feedback,
but discuss trades among players. In total, we extract 801,128 discussion posts.
3.3 Identifying EAGs
Because Steam does not provide a list of EAGs, we use the following approach to
identify them.
3.3.1 Current EAGs
If a game is currently in the early access stage, its Steam Store page would explicitly
state that this game is an EAG. We use the existence of this statement to identify
games that are currently in the early access stage. These games are in the remainder
of this paper referred to as current EAGs.
3.3.2 Former EAGs
Because the Steam Store does not explicitly identify games that have already left the
early access stage, we use the existence of early access reviews (i.e. reviews with the
“early access review” tag) to get a minimal indication of whether the game used the
early access model at some point. The identified games are in the remainder of this
paper referred to as former EAGs.
3.4 Collecting Historical Data
We extract the history of the number of owners since March 20th, 2015 for all games
from Steam Spy [15], a third-party project which continuously monitors the Steam
platform. People own a game when they buy the game on Steam, in retail and then ac-
tivate on Steam, or when they receive the game through a promotion or as a gift [15].
Different from owners, the players of a game are people who play the game during a
specific time range. Hence, the number of owners is not necessarily the same as the
number of players in a day. However, as we only use the number of owners in our
study, we use players and owners interchangeably in the remainder of this paper.
An Empirical Study of Early Access Games on the Steam Platform 11
Due to the large quantity of data that is collected from the Steam platform, the
crawl cannot be done instantly. In fact, the crawling process started on March 7th
and ended on March 19th, after which the number of owners’ data was crawled from
Steam Spy on March 20th. We use the data from both sources up to March 7th to
ensure that we study the same time frame for all games.
As often happens in the game industry, all the data needed to track sales figures
on Steam are not publicly available. Nevertheless, Steam Spy estimates the number of
owners of a game [36]. The method uses information from user profile pages on the
Steam Community, which shows the games that a user owns. Theoretically, by crawl-
ing the profile pages for all users, we can calculate the accurate ownership statistics.
Practically, with about 172 million users (and growing every day) on Steam, it is hard
to have the computing power needed to churn through all profile pages in a timely
manner. Steam Spy randomly crawls a representative sample of user profile pages
to estimate the number of owners. To be more accurate, Steam Spy uses a three-
day rolling sample to generate the final reported numbers of owners, i.e., every day,
the data from three days prior are replaced by newly-crawled data. About 1,700,000
randomly-selected profiles are crawled every three days.
We also extract the price history since November 27th, 2014 for all games from the
Steam DB project [37], another third-party project that monitors the Steam platform.
We use the price of a game in U.S. Dollar in our study.
4 Early Access Games (EAGs)
This section presents the results of our empirical study on EAGs.
4.1 RQ1: What are the characteristics of the early access model?
Motivation: We study the characteristics of the early access model. As few previous
studies have focused on the early access model in the game industry [2, 49], it is
essential to have a general understanding of the current status of the model. The
results that are described in this section motivate the remainder of our paper.
Approach: We analyze the popularity of the early access model by studying the num-
ber of games on the SEARP and the number of owners of EAGs. We plot the propor-
tions of games that are released as EAGs by each developer. In addition, we calculate
the length of the early access stage, and study the drivers for short and long early
access stages.
In order to get the length of the early access stage of each game, we manually
check the release notes for each game and identify the release notes that describe the
availability of the game on Steam and the game leaving the early access stage. We
use the number of days between the publication dates of these two release notes as
the length of the early access stage. As mentioned in Section 2, it is not mandatory
for developers to publish these release notes. We were able to identify 227 out of 396
games which have both release notes for entering and leaving the early access stage.
We use Rand Python for our statistical analysis.
12 Dayi Lin et al.
Mar−13
Apr−13
May−13
Jun−13
Jul−13
Aug−13
Sep−13
Oct−13
Nov−13
Dec−13
Jan−14
Feb−14
Mar−14
Apr−14
May−14
Jun−14
Jul−14
Aug−14
Sep−14
Oct−14
Nov−14
Dec−14
Jan−15
Feb−15
Mar−15
Apr−15
May−15
Jun−15
Jul−15
Aug−15
Sep−15
Oct−15
Nov−15
Dec−15
Jan−16
Feb−16
Number of games
0 10 20 30 40 50 60
All EAGs released in a month
Current EAGs Former EAGs
Fig. 2: The number of EAGs that are released since the start of the SEARP. The darker
part represents the number of EAGs that are released in that month that are still in the
early access stage at the time of our data collection. The lighter part represents the
number of EAGs that are released in that month that have left the early access stage
at the time of our data collection.
Findings: 15% of the games on Steam make use of the early access model and its
popularity is growing. Of the 8,025 games that are available on Steam, 786 games
are current EAGs, and 396 games are former EAGs. As a result, 1,182 (15%) games
are or were making use of the early access model.
Figure 2 shows the popularity of the early access model. The figure clearly shows
that there is a growing trend of popularity in the use of the early access model. With
64 games released on the Steam early access platform in 2013, and 485 games newly
available through early access in 2015, the model shows a 660% increase in the ab-
solute number of releases.
The increasing trend in popularity is confirmed by Figure 3, which shows the ratio
of the number of EAGs that are released each month and the total number of games
that are released in that month. The ratio increases from approximately 0.05 to 0.20
in early 2016.
25% of the EAGs have more than 48 thousand owners, with almost 29 million
owners for one of the studied EAGs. Figure 4 shows the distribution of the number
of owners of EAGs. A considerable number (62%) of the EAGs has been available
for less than a year, leading to a median number of owners of 11,270. Moreover,
25% of the EAGs have more than 47,950 owners, with 43 (3%) of the EAGs having
more than 1 million owners. The most popular EAG, the Killing Floor 2 game5has
28,878,959 owners.
5http://store.steampowered.com/app/232090/
An Empirical Study of Early Access Games on the Steam Platform 13
0.00 0.05 0.10 0.15 0.20 0.25
Ratio of EAGs
Mar−13
Apr−13
May−13
Jun−13
Jul−13
Aug−13
Sep−13
Oct−13
Nov−13
Dec−13
Jan−14
Feb−14
Mar−14
Apr−14
May−14
Jun−14
Jul−14
Aug−14
Sep−14
Oct−14
Nov−14
Dec−14
Jan−15
Feb−15
Mar−15
Apr−15
May−15
Jun−15
Jul−15
Aug−15
Sep−15
Oct−15
Nov−15
Dec−15
Jan−16
Feb−16
Fig. 3: The ratio of the number of released EAGs each month and the total number of
games that are released in that month. The smooth curve is computed using a Local
Polynomial Regression Fitting [6].
103104105106107
Number of owners
Fig. 4: Distribution of the number of owners of EAGs.
34% of all EAGs have left the early access stage. This number can partly be
explained by the recency of an early access release. However, EAGs from 2013 do
not have a considerably higher percentage of leaving the early access stage. Only 162
(50%) of the 322 EAGs that were available before 2014 have left the early access
stage. Hence, customers are taking the risk that an EAG will possibly spend a long
time in development (or that the game will even fail to leave the early access stage
eventually).
14 Dayi Lin et al.
Table 3: Top 10 genres for EAGs and non-EAGs
EAGs Non-EAGs
Genre # of games %∗Genre # of games %∗
Indie 1,046 88.49 Indie 3,863 56.45
Early Access 783 66.24 Action 3,246 47.44
Action 752 63.62 Adventure 2,859 41.78
Adventure 499 42.22 Singleplayer 2,123 31.02
Strategy 403 34.09 Casual 2,003 29.27
RPG 363 30.71 Strategy 1,663 24.30
Simulation 348 29.44 RPG 1,318 19.26
Multiplayer 322 27.24 Simulation 1,173 17.14
Singleplayer 296 25.04 Multiplayer 1,084 15.84
Casual 231 19.54 Puzzle 906 13.24
* Note that these percentages do not add up to 100% as developers can
assign multiple genres to their games.
Walker [49] has conducted a similar calculation in 2014, and obtained a per-
centage of 25% instead of 34%. We contacted Walker and he kindly provided a list
of games that were studied in his article on early access games. After comparing our
dataset with his list, we found that only 266 of the 366 games in his list were available
on Steam at the time at which we collected our data. Hence, one possible explana-
tion of the 9% growth is that in the past two years some EAGs were removed from
the Steam store. Therefore, these games were no longer available at the time that we
collected our dataset.
88% of the EAGs are indie games, indicating that most EAGs are developed
by individual developers or small studios. Table 3 shows the top 10 developer-
defined genres for EAGs. For both EAGs and non-EAGs, indie games are the largest
genre. However, only half of the non-EAGs are defined as indie games, while 88% of
the EAGs are indie games.
To the best of our knowledge, there is no official definition of what an “indie”
game involves. We use the universal definition as concluded by Stern [40]: “A game
that is both (a) developed to completion without any publisher or licensor interfer-
ence, and (b) created by a single developer or a small team.” We assume that games
classified under the “indie” genre on Steam follow this definition.
To validate this definition, we extract 4,927 unique developers from the basic
information of all 8,025 games and count the number of games that are released by
each developer. Figure 5 shows the relation between the number of games and the
percentage of EAGs that are developed by the same developer. When calculating
the percentage of EAGs, we manually filter out the games that are released before
March 20, 2013 (i.e., the start date of the SEARP), the games that are re-released
back to Steam, and the games that are content packs for existing games, as logically
these games did not have the chance to be released as EAGs. The developers that do
not have any game released after March 20, 2013 are also not shown in the figure.
Figure 5 indicates that, as developers release more games, the percentage of EAGs
decreases, indicating that EAGs are mainly developed by individuals or small studios
with less games.
An Empirical Study of Early Access Games on the Steam Platform 15
5 10 15 20
0 20 40 60 80 100
Number of developed games
% of EAGs
1 developer
10 developers
20 developers
>100 developers
Fig. 5: Relation between the number of games and % of EAGs that are developed
by the same developer (darker dots represent a larger number of developers with that
relation)
Table 4: The number of EAGs per development studio
Number of EAGs Number of Studios
0 3,814
1 1,062
2 41
3 9
4 1
Total 4,927
Table 4 shows the number of EAGs per development studio. Table 4 shows that
most studios have zero or one EAGs.
Most former EAGs have spent less than a year in the early access stage. Fig-
ure 6 shows the distribution of the length of the early access stage for former EAGs.
160 of 227 (70%) former EAG spend less than 365 days, i.e. a year, in the early access
stage, with a median of 225 days. The longest early access stage record, 929 days,
is kept by the Prison Architect game6. We manually check the discussions between
developers and players on discussion forums in the Steam Community of games that
are more than 800 days in the early access stage, and identify the following reasons
as claimed by developers for the long length of the early access stage:
1. A lack of developers in the team, or a lack of funds to hire developers for the team
(Grim Dawn [32]).
6http://store.steampowered.com/app/233450/
16 Dayi Lin et al.
0 200 400 600 800
Days in the early access stage
Fig. 6: Distribution of the days in the early access stage for former EAGs.
2. A lack of experience or specific skills by the developers (e.g., UI art) (Under-
rail [25], Grim Dawn [32]).
3. Difficulties of estimating the full budget, which causes the delay of hiring more
developers (Grim Dawn [32]).
4. A refusal to launch the game until it reaches a very high standard with which the
developers themselves are satisfied (Edge of Space [21, 22]).
In addition, we manually check the release notes of the ten EAGs with the shortest
length in the early access stage. The developers of two of such games give the reason
for the short length of the early access stage, while the other eight EAGs do not
give a reason. The developers of the Parcel game7, which only stayed in the early
access stage for 26 days, stated that they had gone over budget and that the early
access model failed as a funding channel for them [41]. The developers of the RONIN
game8, which spent 34 days in the early access stage, explained that the game had
already been tested by beta testers, other developers and third-party testing studios,
and their goal was to perfect the game by using the honest feedback that is gathered
in the early access stage [48].
Currently, 15% of the games on Steam use the early access model. The early
access model tends to appeal mostly to individuals or small studios to release
their indie games. However, using the early access model is not a guarantee
for collecting enough funds to continue the development of a game.
4.2 RQ2: How do developers and players of EAGs interact with the Steam platform?
Motivation: One of the major benefits of the early access model for developers is
that it is possible to get early feedback on a game, for example, through reviews
7http://store.steampowered.com/app/316080/
8http://store.steampowered.com/app/274230/
An Empirical Study of Early Access Games on the Steam Platform 17
that players post on the Steam platform. As early access players should be deeply
involved in the development process as claimed by Steam [46], we expect to see a
stronger interaction of players with the Steam platform in the early access stage of a
game. In addition, we expect that developers post more updates for an EAG, as they
are improving the game (for example, based on the feedback that they acquire from
user reviews).
Approach: We compare the average review rate (#o f reviews
#o f owners ) within the early access
stage and during the entire lifetime of a game. In addition, we compare the discussion
participation rate ( #o f post s
#o f owners ) within and after leaving the early access stage. We use
the average review rate and the discussion participation rate to capture the interac-
tion between players and the Steam platform. Furthermore, we calculate the update
frequency (# of days between adjacent release notes) within and after leaving the
early access stage. We use the update frequency to capture the interaction between
developers and the Steam platform.
Because of promotional actions on Steam, the number of owners of a game can
decrease. For example, there is a type of promotion named “free weekends”, which
temporarily offers certain games at no charge. Players who get a game on a “free
weekend” would only own the game for a limited time. However, these temporary
owners are able to review a game as well. Hence, we use the highest number of
owners that is observed during the early access stage and the lifetime of a game for
our analysis.
We use the Wilcoxon signed-rank test to compare the metrics within and af-
ter leaving the early access stage. The Wilcoxon signed-rank test is a paired, non-
parametric statistical test of which the null hypothesis is that two input distributions
are identical. If the p-value computed by the Wilcoxon signed-rank test is smaller
than 0.05, we conclude that the two input distributions are significantly different. On
the other hand, if the p-value is larger than 0.05, the difference between the two input
distributions is not significant.
In addition, we calculate Cliff’s delta d[26] effect size to quantify the difference
in the distributions of the metrics. We use the following threshold for interpreting d,
as provided by Romano et al. [39]:
Effect size =
negligible(N),if |d| ≤ 0.147.
small(S),if 0.147 <|d| ≤ 0.33.
medium(M),if 0.33 <|d| ≤ 0.474.
large(L),if 0.474 <|d| ≤ 1.
Findings:
63% of the EAGs update more frequently in their early access stage. The
beanplot in Figure 7 shows the distribution of the update frequency during and af-
ter leaving the early access stage. A beanplot shows the density plots for two dis-
tributions side by side so that they can be easily compared. In general, developers
update their game more frequently in the early access stage, with a median of 11
days between adjacent updates in the early access stage. The number of days be-
tween releases after leaving the early access stage increases to 15 days. The Wilcoxon
signed-rank test shows that the difference between the two distributions in Figure 7 is
18 Dayi Lin et al.
110 100 1000
Median days between adjacent updates
Early access stage
After early access stage
−1.000
−0.298
−0.207
−0.112
0.000
1.000
Negligible Small Medium Large
Effect size (Cliff's Delta)
−0.298
Fig. 7: Distribution of the update frequency (measured as the median number of
days between adjacent updates) during and after leaving the early access stage for
all EAGs (the vertical line shows the median of each distribution). The figure be-
low the beanplot shows Cliff’s Delta effect size (-0.207) and its confidence interval
([−0.298,−0.112]). The colored areas represent the thresholds that we used to inter-
pret Cliff’s Delta.
significant (p-value = 5.833e-10) with a small effect size (Cliff’s Delta = -0.207). We
calculate that almost two third (63%) of the EAGs have a higher update frequency in
their early access stage, and 3% of the EAGs have the same update frequency in and
after leaving their early access stage.
We inspect the update frequency of the EAGs that have a higher update frequency
after their early access stage. 72% of these games have left the early access stage after
2015. A possible explanation for the update frequency being higher after leaving the
early access stage is that the update frequency tends to be higher for a short-while
directly after leaving the early access stage, because of a boost in new players or
funds. After a while, the game tends to become more stable, resulting in a lower
update frequency.
Figure 8 shows an example of the update timeline of the Fight The Dragon game9,
a former EAG which has left the early access stage since December 2014. There exists
9http://store.steampowered.com/app/250560/
An Empirical Study of Early Access Games on the Steam Platform 19
Jun Jul Aug Sep Oct Nov Dec Jan
2015
Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec Jan
2016
Feb
|
Fig. 8: Update timeline of the Fight The Dragon game (each line represents an update,
and the date of leaving the early access stage which is Dec 4, 2014 is marked in red)
00.2 0.4 0.6 0.8 1
Ratio of updates in month 1-3 / in month 1-12
after leaving the early access stage
Fig. 9: Distribution of the ratio of updates within 3 months and 12 months after leav-
ing the early access stage.
a clear difference between the update frequency in and shortly after leaving the early
access stage and the stage after June 2015.
To support our explanation of a stabilizing update frequency, we further study the
update timeline of former EAGs which have left the early access stage for at least a
year. Figure 9 shows the distribution of the ratio of updates within 3 months and 12
months after leaving the early access stage. We calculate that for 29% of these former
EAGs, 100% of their updates within 12 months after leaving the early access stage
were released in the first three months. 51% of these former EAGs release 60% of the
updates within the first three months.
65% of the EAGs see an equal or lower activity of owners posting reviews in
the early access stage. Figure 10 shows the distribution of the average review rate
during the early access stage and the lifetime. The Wilcoxon signed-rank test shows
that the difference between the two distributions is significant (p-value = 0.009) with
a negligible effect size (Cliff’s Delta = -0.039), suggesting that users post reviews less
often in the early access stage of a game than after leaving the early access stage. We
calculate that 62% of the EAGs see an lower average review rate in the early access
stage, and 3% of the EAGs see an equal average review rate in and after leaving the
early access stage.
A possible explanation is that owners are aware that an EAG is still under devel-
opment and not in its best shape. Hence, they prefer to give the developers more time
to improve the game, and wait until the game leaves the early access stage to give
their reviews, rather than judge the game in its unfinished shape.
20 Dayi Lin et al.
00.020.040.060.080.10
Average review rate
Early access stage
Lifetime
−1.000
−0.143
−0.039
0.000
0.066
1.000
Negligible Small Medium Large
Effect size (Cliff's Delta)
Fig. 10: Distribution of the average review rate during the early access stage and the
lifetime for all EAGs (the vertical line shows the median of each distribution). The
figure below the beanplot shows Cliff’s Delta effect size (-0.039) and its confidence
interval ([-0.143, 0.066]). The colored areas represent the thresholds that we used to
interpret Cliff’s Delta.
81% of the EAGs observe an equal or higher activity on the discussion fo-
rums in the early access stage. Figure 11 shows the distribution of the discussion
participation rate during and after leaving the early access stage. As shown in the fig-
ure, a game receives a median of 0.04 discussion posts per owner in the early access
stage, which is twice as high as the median number of discussion posts per owner
after leaving the early access stage (0.02). The Wilcoxon signed-rank test shows that
the difference between the two distributions is significant (p-value = 4.918e-16) with
a small effect size (Cliff’s Delta = 0.304). We calculate that 66% of the studied for-
mer EAGs observe a higher discussion participation rate in their early access stage.
15% of the studied former EAG have a consistent discussion participation rate in and
after leaving the early access stage.
The higher discussion participation rate in the early access stage supports the
explanation that we provide for the finding that owners post less reviews in the early
access stage. It appears that early access owners tend to provide their feedback in
An Empirical Study of Early Access Games on the Steam Platform 21
0 0.1 0.2 0.3 0.4 0.5
Discussion participation rate
Early access stage
After early access stage
−1.000
0.000
0.203
0.304
0.399
1.000
Negligible Small Medium Large
Effect size (Cliff's Delta)
Fig. 11: Distribution of the discussion participation rate during and after leaving the
early access stage for all EAGs (the vertical line shows the median for each distri-
bution). The figure below the beanplot shows Cliff’s Delta effect size (0.304) and its
confidence interval ([0.203, 0.399]). The colored areas represent the thresholds that
we used to interpret Cliff’s Delta.
discussion forums instead of in reviews, which does not affect the positive review
rate of a game.
For developers, the lower review rate and the higher discussion participation rate
in the early access stage appears to be a double-edged sword. On the one hand, the
lower review rate reduces the chances that a possibly buggy and imperfect version
of the game leads to complaints in reviews, which might mislead potential customers
after leaving the early access stage. On the other hand, it is difficult for developers to
perceive and quantify how satisfied the owners are in the early access stage. Although
the discussion forums on the Steam Community offer a place for developers and
players to communicate, the posts normally only consist of concrete issues such as
questions or suggestions, rather than specific, quantifiable sentiment as provided by
reviews.
Developers update a game more frequently in its early access stage. Players
post significantly less reviews but more discussion posts in the early access
stage (all with small effect size).
22 Dayi Lin et al.
4.3 RQ3: How tolerant are players of the quality of EAGs?
Motivation: EAGs are unfinished by definition. Although owners have access to a
playable version of a game, the content of this version can be incomplete, the client
can be buggy or the performance can be poor. Players are aware of the possible issues
when they purchase an EAG. Because the reputation accumulated in the early access
stage can impact the popularity of a game after leaving the early access stage, we
study whether owners are more tolerant of the quality during the early access stage
of a game.
Approach: We quantify the tolerance of owners of the quality of former EAGs within
and after leaving the early access stage using the positive review rate of games
(#o f positive reviews
#o f tot al reviews ). The reviews of a game can greatly affect the will to purchase
of potential customers as stated in Section 2. Hence, a higher positive review rate in
the early access stage can lead to a higher popularity after the game leaves the early
access stage. In addition, we calculate the correlation of the positive review rate, the
length of the early access stage and the update frequency in the early access stage.
We use Spearman correlation because the data is not normally distributed.
Findings:
89% of the EAGs receive an equally or higher positive review rate during
their early access stage. Figure 12 shows the distribution of the positive review rate
during and after leaving the early access stage. The Wilcoxon signed-rank test shows
that there is a significant difference (p-value <2.2e-16) with a medium effect size
(Cliff’s Delta = 0.454) between the two distributions.
We calculate that 88% of the former EAGs receive a higher positive rate in their
early access stage, with a median positive rate of 88%, which is higher than the me-
dian positive rate after leaving the early access stage (69%). 1% of the former EAGs
receive a consistent positive rate in and after leaving their early access stage.
As mentioned in Section 2, the positive review rate is used in the games’ Steam
Store page as the official indicator of the quality of a game. As a result, a higher
positive review rate can greatly benefit the popularity of a game after it leaves the
early access stage. The above findings suggest that games can receive more positive
reviews when using the early access model. However, the higher positive review rate
does not suggest that the early access model is a fix for low-quality games. More
likely is the possibility that the people who buy EAGs are more tolerant of the un-
finished status of a game. Another possibility is that the developers that use the early
access model are good at keeping their players happy.
The positive review rate is not correlated with either the length of the early
access stage or the update frequency in the early access stage The Spearman cor-
relation between the positive review rate and the length of the early access stage is
-0.06. The update frequency in the early access stage and the positive review rate have
a Spearman correlation of 0.01. These findings indicate that neither the length of the
early access stage, nor the update frequency in the early access stage are correlated
with the positive review rate.
These findings suggest that developers can take time to polish their EAGs until
they are ready to leave the early access stage, without worrying that the long length
of the early access stage might decrease their positive review rate. In addition, devel-
An Empirical Study of Early Access Games on the Steam Platform 23
00.2 0.4 0.6 0.8 1
Positive review rate
Early access stage
After early access stage
−1.000
0.000
0.381
0.454
0.522
1.000
Negligible Small Medium Large
Effect size (Cliff's Delta)
Fig. 12: Distribution of the positive review rate during and after leaving the early ac-
cess stage for all EAGs (the vertical line shows the median for each distribution). The
figure below the beanplot shows Cliff’s Delta effect size (0.454) and its confidence
interval ([0.381, 0.522]). The colored areas represent the thresholds that we used to
interpret Cliff’s Delta.
opers can choose the update schedule that best fits their development process during
the early access stage, rather than rush to add more content and features.
Players of EAGs tend to be more tolerant of the quality of a game during its
early access stage. While players tend to post less reviews within the early
access stage, 89% of EAGs receive an equally or more positive review rate
in their early access stage. In addition, developers do not need to rush into
releasing their games, as it appears that the tolerance of players is not cor-
related with the length of the early access, though other factors might be at
play, such as the budget and funding of their games.
5 Additional Interesting Insights
In this section, we discuss several observations that are worth noting and can lead to
future work.
24 Dayi Lin et al.
−20 −10 0 10 20
Price growth after early access stages (US dollars)
Fig. 13: Distribution of price changes.
5.1 The Price of a Game Within and After Leaving the Early Access Stage
As explained in Section 2, developers of EAGs have the liberty to change the price
of the game at any point in time. Steam states that, depending on the “goals and the
level of commitment and feedback” developers desire from early access players, they
can start by offering a discount, or on the contrary, charge a premium [46]. Therefore,
we consider the change of price as a reflection of the purpose of developers to use the
early access model. We assume that developers ask a lower price in the early access
stage when they aim to gather more feedback and use the low price to attract more
players. On the other hand, when developers charge a higher price in the early access
stage of their game, they tend to use the model as a funding source to support the
development process of their games.
We compare the price during and after leaving the early access stage. Figure 13
shows the distribution of the price changes, i.e., we subtract the early access price
from the price of the game after leaving the early access stage. 95 (24%) of the EAGs
are free to play throughout their lifetime, including the early access stage. We remove
them from the figure for better demonstration. 145 (48.3%) of the remaining former
EAGs have the same median price within and after leaving the early access stage,
while 91 (30.3%) increase their price and 64 (21.3%) decrease their price.
In addition, of the 64 games of which the price decreases, 6 (9%) become free
to play after leaving the early access stage. We manually check the release notes of
these 6 games to identify the reasons for making the game free to play after leaving
the early access stage. We were able to find the reasons for making the game free for
three games, while the other three silently become free. When the Pool Nation FX Lite
game10 left the early access stage, developers divided the game into the basic free-
to-play part and two optional packs which need to be purchased [8]. The developers
of the Bierzerkers game11, however, stated that it is the early access players who
suggested them to make the game free, in order to build the base of the game. To
10 http://store.steampowered.com/app/314000/
11 http://store.steampowered.com/app/348460/
An Empirical Study of Early Access Games on the Steam Platform 25
reward the early access players, they each received all of the launch characters [3]. As
for the Cards and Castles game12, although developers did not specify the reasons of
making the game free, they offered an early access bundle containing unique content
to early access players and persuaded people to buy the game in the last two weeks
of the early access stage [4].
For the free EAGs and the EAGs with a lower price in the early access stage,
which represent 47% of the EAGs, it is likely that their developers focused on gath-
ering early feedback from the community. The percentage is significantly higher than
the EAGs which charge a premium for early access (16%), indicating that their devel-
opers aim at raising development funds. Although this is only one possible explana-
tion for the change of price and developers might have several goals when using the
early access model, the phenomenon suggests that the majority of EAG developers
value the opportunity to elicit feedback more than the opportunity to raise develop-
ment funds.
5.2 Lessons Learned from an Early Access Failure: the Spacebase DF-9 Game
In this section, we discuss lessons learned from the the Spacebase DF-9 game. Prior
work (e.g., Washburn et al. [50]) discusses what went wrong and what went right
during the development of a game, but no prior work focused on the failure of EAGs
specifically.
The Spacebase DF-9 game is developed by Double Fine Productions, an indie
game development studio [12]. The game was available on the SEARP on October 15,
2013. On October 27, 2014, the game unexpectedly terminated the early access stage
and released a final product that lacked many of the planned features. On November
21, 2014, twelve employers including the project lead were laid off. On December 16,
2014, an announcement was posted on the official technical support forums, stating
that there were no further plans for patches and there was no team assigned to the
project [16].
The abandonment of the game led to the disappointment of a large number of
players. As a result, the game received 79% (2,598) negative reviews, and raised a
debate between the players and the studio on the discussion forums of the game on
the Steam Community [47].
The game is considered to be a failure of the early access model. In order to
understand the reasons for its failure, and the lessons that can be learned for future
EAGs, we manually study two threads on the Steam Community. One of the threads is
posted by the studio [44] and the other thread is posted by the players [43]. Together
the threads contain around 800 discussion posts. We identify the following lessons
that can be learned:
Lesson 1: It is risky to use the early access model as the main funding source.
The reason for terminating the development of the Spacebase DF-9 game, claimed
by the studio, is that this project was started with an open ended-production plan,
with the hope that it can progress similarly to some other early access-funded games.
12 http://store.steampowered.com/app/360730/
26 Dayi Lin et al.
However, the sales quickly became insufficient to support the development process.
Although the developer put all the raised funds back into the development of the
game, it turned out that the raised funds were not sufficient to fund a complete devel-
opment team.
However, players argue that the developer should have considered the game as
an investment, and that the profit would come after leaving the early access stage.
They consider the funding of a game’s development solely with early access sales
to be “irresponsible if not downright delusional”. Some players even question the
money management of the studio, although the studio later responded that it con-
sidered continuing development on a game that costs more than it makes to be bad
money management.
Lesson 2: Do not release a game on the SEARP too early. A potential reason
for not selling enough copies, posted by some players, is that the game was released
into the SEARP too early, lacking content and features for players (“basically nothing
meaningful to do after 45 mins of playtime”). The players suggested that, to ensure
sales remain above made costs, developers should release the game in a more content
and feature-rich state.
Lesson 3: State promises and plans clearly. The most obvious lesson that can
be learned from the failure, as stated by the studio, is that it did not clearly indicate in
the “original promise” which features were securely funded, and which portion of the
game was dependent on early access sales. This point is supported by many players,
who considered the original statement “none of these features are set in stone” to be
too ambiguous and vague.
Lesson 4: When a game is abandoned by its development studio, the repu-
tation of the studio as a whole can be damaged. Besides the anger towards the
abandonment of the game, a large quantity of players are doubting the integrity of
the studio, and claimed that they would never purchase any future game from Double
Fine Productions. The players consider the abandonment of the EAG as betraying a
long term commitment, as they purchased the EAG not for its current form, but for the
potential it had. In addition, players were concerned about whether the studio would
be capable of improving the development of future EAGs. The lack of introspection
totally “bankrupted the company by ruining the reputation”, as said by players. It is
worth noting that players stated clearly that they would not stop supporting EAGs or
indie developers, but would specifically stop supporting this “irresponsible” studio.
Lesson 5: Communicate issues and changes to the promised plan on time.
The studio claimed that they announced the situation and the decision to terminate the
development rather than “vanish quietly in the night”. However, the players argued
that if the studio could communicate with players immediately when trouble firstly
came up, the players could have helped by recommending the game to friends and
relatives, or even bought copies for them. It was the lack of communication of the
troubles that the game was facing that killed this game.
The aforementioned lessons demonstrate that players get emotionally involved in
the development of EAGs. One of the players posted that “I am sorry that powers
above you have closed your beloved project down, and I’m also frankly sorry that I
don’t get a finished product. This game could have been so amazing.”
An Empirical Study of Early Access Games on the Steam Platform 27
Although the aforementioned lessons come from one game, they give an overview
of realistic dangers that apply to EAGs. The main lesson that can be learned is that
the player involvement should work in both directions. On the one hand, developers
appreciate the feedback from the players of their game. On the other hand, developers
should show appreciation of their players by communicating and actually involving
them in the decision-making process.
6 Threats to Validity
This section presents the threats to the validity of our findings.
6.1 Internal Validity
A threat to the validity of our findings is that it is not necessary for game developers
to publish release notes for a game update to one of the Steam channels. Hence, all
numbers that we provide in this paper may be low estimates of the actual number of
updates.
The number of owners used in our study are estimated from a representative ran-
dom sample by Steam Spy. Although a three-day rolling sample is used to increase the
accuracy, there can still exist a deviation from the actual number of owners. However,
because the sales data is confidential in the game industry, this is the most accurate
method to our knowledge to estimate the number of owners of a game.
We estimated the total number of games that are released in a month using the
release date as advertised on the Steam Store page. This number is an estimation
because developers are allowed to change the release date that is shown on the Steam
Store page. We observe that for some games that exist before they are released on
Steam, developers changed the release date to the real release date. We do not have
data (reviews, discussions, price, etc.) between the real release date and the date that
the game is released on Steam. However, we expect it is sufficiently accurate to be
used to give a reasonable estimate of the number of games released in a month. Note
that we used release notes to identify the date on which a game was released as an
EAG, hence this threat does not affect the validity of our findings that are related to
EAGs.
The learned lessons that we describe about failed games come from one game.
However, at the time of writing, it is the only game for which the failure of the early
access model has been explicitly documented. These lessons can be revisited later
when documentation about the early access model for more EAGs becomes available.
As in all empirical studies, separating causation and correlation is a challenge in
our work. While we cannot show that the early access model leads to more satisfied
game owners, there exists a correlation between a higher positive review rate and the
usage of the early access model. One possibility is that the type of game owner that
buys an EAG is more happy in general than non-EAG buyers. Another possibility is
that EAGs are only bought by more tolerant owners. Either possibility supports the
findings that are presented in this paper.
28 Dayi Lin et al.
We use the frequency of game updates as a proxy of interaction between develop-
ers and the Steam platform. We believe that this interaction is a rough estimate of how
much developers care about the quality of their game. In this paper, we did not study
whether updates are a direct response to user feedback. Future studies should inves-
tigate more closely the link between game updates and user feedback, for example,
that is acquired through various avenues such as user reviews.
6.2 External Validity
In our empirical study, we studied the EAGs on Steam. The findings of our study may
not generalize to other EAGs on different distribution platforms. However, as stated
in Section 2, Steam is the largest digital distribution platform for PC gaming. Hence,
the EAGs on Steam are representative for a large number of EAGs.
6.3 Construct Validity
We manually validated our approach for identifying release notes and found that our
approach has 89% precision and 87% recall, as described in Section 3.
7 Conclusion
In this paper, we study the early access release model for games. Games that are re-
leased through this model, so-called Early Access Games (EAGs), are early versions
that allow developers to raise funds for development or to elicit early feedback from
players. In particular, we study the characteristics of 1,182 EAGs, the interaction
between players and developers of EAGs and the Steam platform during and after
leaving the early access stage, and the tolerance of players of the quality of EAGs.
Below are the most notable findings of our study:
1. 15% of the games that are currently on Steam make use of the early access model.
The most popular EAG has approximately 29 million owners.
2. EAGs tend to be “indie” games, which suggests that the early access model is
used mostly by smaller development studios.
3. The percentage of players that review a game during its early access stage is lower
than the percentage of players that review a game after leaving the early access
stage. However, the average rating of the reviews is much higher during the early
access stage.
In addition, we discuss several learned lessons from the failure of an early access
game. The main learned lesson from this failure is that the communication between
the game developer and the players of the EAG is crucial. Players enjoy getting in-
volved in the development of an early access game and they get emotionally involved
in the decision-making about the game.
Based on the findings that are presented in this paper, we suggest the following
to developers that are considering the early access model for releasing their game:
An Empirical Study of Early Access Games on the Steam Platform 29
1. If you have a small marketing budget, the early access model can help you to
build a positive reputation, as players of EAGs tend to give more positive reviews.
However, the early access model will not lead to more reviews to your game.
2. Although you can get a larger amount of concrete feedback in the discussion
forum, that feedback is not explicitly linked to negative or positive feelings (as is
the case with reviews), making it more difficult to quantify the feelings of your
players in general.
3. Be aware that using the early access model as your main funding source is a risky
strategy.
We believe that the findings of our paper provide a first step in helping developers
better understand the pros and cons of the early access model.
While our findings do not suggest that using the early access model inherently
leads to more satisfied players, there exists a correlation between EAGs and a higher
positive review rate. One possible explanation for this correlation is that players who
buy EAGs are friendlier towards developers. Another explanation is that developers
that use the early access model are good at keeping their players satisfied. Either
way, while the early access model is not a fix for low-quality games, the early access
model appears to be a valuable tool for developers that want to improve their games
by interacting with their players.
Future studies should use methods such as developer surveys, user studies, and
controlled experiments to examine in more depth the causality between using the
early access model and the satisfaction of both players and developers.
Acknowledgements We are grateful to Sergey Galyonkin, the owner of Steam Spy, who generously gave
us access to all the historical data of Steam collected by Steam Spy for this research. We also thank
professor Jennifer R. Whitson from the University of Waterloo and Mr. Patrick Walker from EEDAR for
the inspiring discussions about the gaming industry.
References
1. Al-Ani B, Trainer E, Ripley R, Sarma A, Van Der Hoek A, Redmiles D (2008)
Continuous coordination within the context of cooperative and human aspects
of software engineering. In: Proceedings of the 2008 international workshop on
Cooperative and human aspects of software engineering, ACM, pp 1–4
2. Allen P (2016) Is Steam’s Early Access a ticking time bomb?
http://www.gamasutra.com/blogs/PaulAllen/20160211/265598/
Is_Steams_Early_Access_a_ticking_time_bomb.php, (last visited: Apr
18, 2017)
3. Bierzerkers (2016) DEVBLOG 5. http://archive.is/XHOjf, (last visited:
Apr 18, 2017)
4. Cards, Castles (2015) Free To Play Coming Soon! http://store.
steampowered.com/news/?appids=360730&headlines=1, (last visited:
Apr 18, 2017)
5. Chambers C, Feng Wc, Sahu S, Saha D (2005) Measurement-based character-
ization of a collection of on-line games. In: Proceedings of the 5th ACM SIG-
30 Dayi Lin et al.
COMM Conference on Internet Measurement (IMC), USENIX Association, pp
1–14
6. Chambers JM, Hastie TJ (1991) Statistical models in S. CRC Press, Inc.
7. Clifford C (2014) Crowdfunding Generates More Than $60,000 an Hour
(Infographic). http://www.entrepreneur.com/article/234051?
newsletter=true, (last visited: Apr 18, 2017)
8. CPx (2015) Pool Nation FX - Lite NOW AVAILABLE. http:
//store.steampowered.com/news/?appids=314000&headlines=1,
(last visited: Apr 18, 2017)
9. Damodaran L (1996) User involvement in the systems design process-a practical
guide for users. Behaviour & information technology 15(6):363–377
10. Dictionary O (2016) Oxford Dictionary Definition of Crowdfunding.
http://www.oxforddictionaries.com/us/definition/american_
english/crowdfunding, (last visited: Apr 18, 2017)
11. EuroGamer (2014) DayZ week-one sales rocket past
400,000. http://www.eurogamer.net/articles/
2014-01-02-dayz-week-one-sales-rocket-past-400-000, (last
visited: Apr 18, 2017)
12. Fine D (2016) Double Fine - What is Double Fine? http://www.doublefine.
com/about/, (last visited: Apr 18, 2017)
13. Foley MJ (2015) Microsoft launches new Office In-
sider test program. http://www.zdnet.com/article/
microsoft-launches-new-office-insider-test-program/, (last
visited: Apr 18, 2017)
14. Gallivan MJ, Keil M (2003) The user–developer communication process: a crit-
ical case study. Information Systems Journal 13(1):37–68
15. Galyonkin S (2016) SteamSpy - All the data and stats about Steam games. http:
//steamspy.com/, (last visited: Apr 18, 2017)
16. Greg D (2014) Double Fine - Action Forums. https://archive.is/vFdqw,
(last visited: Apr 18, 2017)
17. Jacobs M, Sihvonen T (2011) In perpetual beta? on the participatory design of
facebook games. In: Think Design Play: The Fifth International Conference of
the Digital Research Association (DIGRA)
18. Johnson J (1999) Turning chaos into success. Software Magazine 19(3):30–34
19. Johnson J (2000) Chaos in the new millennium: The ghost of christmas future.
The Standish Group, West Yarmouth, MA
20. Kujala S (2003) User involvement: a review of the benefits and challenges. Be-
haviour & information technology 22(1):1–16
21. LadyAijou (2014) About the early access - Edge of Space General
Discussions. http://steamcommunity.com/app/238240/discussions/0/
45350790838400003/#c45350790880063101, (last visited: Apr 18, 2017)
22. LadyAijou (2015) ETA on getting out of Early Access? - Edge of Space General
Discussions. http://steamcommunity.com/app/238240/discussions/0/
41973820433803052/#c594820656453333967, (last visited: Apr 18, 2017)
23. LeBreton J (2014) JP LeBreton’s twitter. https://twitter.com/
vectorpoem/status/536933544472240128, (last visited: Apr 18, 2017)
An Empirical Study of Early Access Games on the Steam Platform 31
24. Lin D, Bezemer CP, Hassan AE (2016) Studying the urgent updates of popular
games on the steam platform. Empirical Software Engineering
25. Logfeller (2014) Release Date? - Underrail General Discussions.
http://steamcommunity.com/app/250520/discussions/0/
35222218811239879/#c626329186828487604, (last visited: Apr 18,
2017)
26. Long JD, Feng D, Cliff N (2003) Ordinal Analysis of Behavioral Data. John
Wiley & Sons, Inc.
27. L¨
owgren J, Stolterman E (2004) Thoughtful interaction design: A design per-
spective on information technology. MIT Press
28. Maalej W, Happel HJ, Rashid A (2009) When users become collaborators: to-
wards continuous and context-aware user input. In: Proceedings of the 24th
ACM SIGPLAN conference companion on Object Oriented Programming Sys-
tems Languages and Applications, ACM, pp 981–990
29. Manes S (1984) Taking a gamble with word vision. PC Mag p 211
30. Martin W, Sarro F, Jia Y, Zhang Y, Harman M (2016) A survey of app store
analysis for software engineering. RN 16:02
31. McDougall J (2010) Community heroes: Notch, for Minecraft. http://www.
pcgamer.com/community-heroes-notch-for-minecraft/, (last visited:
Apr 18, 2017)
32. Medierra (2015) When is this game coming out? - Grim Dawn General
Discussions. http://steamcommunity.com/app/219990/discussions/0/
530649887209076036/#c530649887215003016, (last visited: Apr 18, 2017)
33. Minecraft (2016) Minecraft official website. https://minecraft.net/, (last
visited: Apr 18, 2017)
34. O’reilly T (2007) What is web 2.0: Design patterns and business models for the
next generation of software. Communications & strategies (1):17
35. Orland K (2011) Minecraft Draws Over $33 Million In Revenue From 1.8M
Paying Customers. http://www.gamasutra.com/view/news/33961/
Minecraft_Draws_Over_33_Million_In_Revenue_From_18M_Paying_
Customers.php, (last visited: Apr 18, 2017)
36. Orland K (2014) Introducing Steam Gauge: Ars reveals Steams
most popular games. http://arstechnica.com/gaming/2014/04/
introducing-steam-gauge-ars-reveals-steams-most-popular-games/,
(last visited: Apr 18, 2017)
37. Pavel Djundik MB (2016) Steam DB - Steam Database. https://steamdb.
info/, (last visited: Apr 18, 2017)
38. PCGamer (2014) Tim Schafer explains Spacebase DF-
9’s unexpected v1.0 release. http://www.pcgamer.com/
tim-schafer-explains-spacebase-df-9s-v1-0-release/, (last visited:
Apr 18, 2017)
39. Romano J, Kromrey JD, Coraggio J, Skowronek J, Devine L (2006) Exploring
methods for evaluating group differences on the NSSE and other surveys: Are the
t-test and Cohen’s d indices the most appropriate choices. In: Annual meeting of
the Southern Association for Institutional Research
32 Dayi Lin et al.
40. Stern C (2012) what makes a game indie: a uni-
versal definition. http://sinisterdesign.net/
what-makes-a-game-indie-a-universal-definition/, (last visited:
Apr 18, 2017)
41. Takkunen P (2015) Moving out of Early Access. http://store.
steampowered.com/news/?appids=316080&headlines=1, (last visited:
Apr 18, 2017)
42. Taylor TL (2006) Beyond management: Considering participatory design and
governance in player culture. First Monday
43. ThunderPeel2001 (2015) The facts about Spacebase DF-9.
https://steamcommunity.com/app/246090/discussions/0/
616198900637753626/, (last visited: Apr 18, 2017)
44. TimOfLegend (2014) Tim Schafer answers common questions about
v1.0. https://steamcommunity.com/app/246090/discussions/0/
613936673464943075/, (last visited: Apr 18, 2017)
45. Ullrich C, Borau K, Luo H, Tan X, Shen L, Shen R (2008) Why web 2.0 is good
for learning and for research: principles and prototypes. In: Proceedings of the
17th international conference on World Wide Web, ACM, pp 705–714
46. Valve (2016) Introducing Early Access. http://store.steampowered.com/
earlyaccessfaq/, (last visited: Apr 18, 2017)
47. Valve (2016) Spacebase DF-9 on Steam. http://store.steampowered.com/
app/246090/, (last visited: Apr 18, 2017)
48. Waclawek T (2015) Ronin, the turn-based action-platformer, is now on Steam.
http://store.steampowered.com/news/?appids=274230&headlines=
1, (last visited: Apr 18, 2017)
49. Walker P (2014) Early Access popularity growing, but only 25% have
released as a full game. http://www.gamesindustry.biz/articles/
2014-11-13-early-access-popularity-growing-but-only-25-percent-have-released-as-a-full-game,
(last visited: Apr 18, 2017)
50. Washburn M Jr, Sathiyanarayanan P, Nagappan M, Zimmermann T, Bird C
(2016) What went right and what went wrong: An analysis of 155 postmortems
from game development. In: Proceedings of the 38th International Conference
on Software Engineering Companion, ACM, New York, NY, USA, ICSE ’16, pp
280–289
51. Welch C (2013) Steam Early Access lets gamers buy and play titles still
in development. http://www.theverge.com/2013/3/20/4128644/
steam-early-access-buy-and-play-games-still-in-development,
(last visited: Apr 18, 2017)
52. Yin-Poole W (2014) Valve tightens Steam Early Access
rules for developers. http://www.eurogamer.net/articles/
2014-11-21-valve-tightens-steam-early-access-rules-for-developers,
(last visited: Apr 18, 2017)
A preview of this full-text is provided by Springer Nature.
Content available from Empirical Software Engineering
This content is subject to copyright. Terms and conditions apply.
