Conference PaperPDF Available

Evaluating AI-Based Games through Retellings



We propose a new approach to the human-centered evaluation of AI-based games, grounded in the analysis of player retellings of their play experiences. Retellings offer unique insight into dimensions of player experience that can be hard to get at through existing evaluation methods, such as the typical narrative structures that tend to emerge in the player’s mind when they play a particular game; the variety of subjectively experienced narratives that are possible and probable within a particular game; and the ways in which a game supports, or fails to support, the player’s process of narrativization. We used a grounded theory methodology to analyze retellings of play experiences in Civilization VI, Stellaris, and two distinct versions of the research game Prom Week. We also interviewed the creators of several retellings to gain insight into the subjective experience of story construction in collaboration with these games.
Proceedings of the Fifteenth AAAI Conference on Artificial
Intelligence and Interactive Digital Entertainment (AIIDE-19)
Evaluating AI-Based Games through Retellings
Max Kreminski,1Ben Samuel,2Edward Melcer,1Noah Wardrip-Fruin1
1University of California, Santa Cruz
2University of New Orleans
{mkremins, eddie.melcer, nwardrip},
We propose a new approach to the human-centered evalua-
tion of AI-based games, grounded in the analysis of player
retellings of their play experiences. Retellings offer unique
insight into dimensions of player experience that can be hard
to get at through existing evaluation methods, such as the typ-
ical narrative structures that tend to emerge in the player’s
mind when they play a particular game; the variety of sub-
jectively experienced narratives that are possible and proba-
ble within a particular game; and the ways in which a game
supports, or fails to support, the player’s process of narra-
tivization. We used a grounded theory methodology to an-
alyze retellings of play experiences in Civilization VI,Stel-
laris, and two distinct versions of the research game Prom
Week. We also interviewed the creators of several retellings
to gain insight into the subjective experience of story con-
struction in collaboration with these games.
Introduction and Related Work
One enduring argument for the introduction of artificial in-
telligence and procedural content generation into games has
been that they will enable a wider variety of more mean-
ingfully distinct experiences from player to player. Another,
closely related argument holds that AI and PCG will enable
games to respond more deeply and meaningfully to player
input. However, existing methods of evaluation do little to
surface this dimension of player experience. As a result,
this dimension of player experience tends to be left out of
our analyses of AI-based (Treanor et al. 2015) and PCG-
based (Smith et al. 2011) games.
Ryan’s work on what he terms curationist interactive nar-
rative (Ryan 2018) has recently called for the development
of AI systems that assist players in sifting through deep sim-
ulations to find storyful content. As a step in this direc-
tion, we believe it is valuable to look at how players nar-
rativize their experiences in existing AI-based games. More
broadly, we are interested in games as sites of player cre-
ativity, and—through a narrative-focused lens—particularly
as sites of story construction by players. Therefore, we are
interested in developing an evaluation method that addresses
the extent to which games support narrativization by players
of their play experiences.
Copyright c
2019, Association for the Advancement of Artificial
Intelligence ( All rights reserved.
Meanwhile, Eladhari (Eladhari 2018) has called for a fo-
cus on retellings—the stories players tell about their play
experiences—as an instrument for the critique of interac-
tive narrative systems. Framing retellings as narrative arti-
facts co-created by players and the games that they play, she
suggests that studying these narrative artifacts may enable
a deeper understanding of player experience. From this per-
spective, players are already making use of games as sto-
rytelling partners—and, in some cases, seeking out games
with the specific intent of using them to support story con-
struction. Consider, for instance, the case of Alice and Kev,
a lengthy Sims 3 retelling that Eladhari highlights as an ex-
ample of a compelling story co-created by a human and an
(AI-based) game.
Retellings are in some ways similar to and in some ways
different from playtraces, which aim to capture, objectively,
what actually happened in the course of play. We propose
to study retellings as a kind of subjectively-grounded play-
trace. We contend that, in the course of play, players often
imagine or create stories beyond those that are represented
literally in the game. They may place unexpected weight on
events or details, including apparently cosmetic ones, that
seem inconsequential from a gameplay perspective; extrap-
olate the ramifications of events or details in ways that were
not intended by the game’s developers; and generally bring
their own creativity and subjectivity to the process of narra-
tivizing their play experiences. When players reflect on their
play experiences in order to relate or retell them to others,
the retellings they produce may then capture dimensions of
experience that are absent from even the most exhaustive
playtrace. Moreover, whether and how a game is narrativiz-
able by players may tell us things about whether and how it
makes sense to people more generally.
Simultaneously, it is very difficult for players to con-
vey rich, emotional experiences encountered during game-
play (Laaksolahti, Isbister, and H¨
ok 2009), and asking
players about these experiences outright can lead to mislead-
ing results (King 2004; Leech 2002). We hope our approach
will allow the player to more deeply describe a narrative play
experience uninhibited, and in turn allow us to glean much
more insightful information on how certain mechanics en-
able or inhibit players in the construction of stories.
In this paper, we describe several possible ways that
player-created retellings of their play experiences may be
used to evaluate AI-based and PCG-based games. First, we
explore how written retellings of Civilization VI and Stel-
laris play experiences found in online player communities
can be used to develop a codebook capturing relevant di-
mensions of variation between retellings. Next, we analyze
a small number of Civilization VI and Stellaris retellings,
in conjunction with interviews with the authors of these
retellings, to develop a better understanding of the subjec-
tive experience of storytelling in collaboration with these
games. Finally, we apply a similar methodology in conjunc-
tion with an ablation study approach to investigate the im-
pact of social reasoning on player narrativization in the re-
search game Prom Week, for which few if any “naturally oc-
curing” retellings exist.
Collecting and Analyzing “Wild” Retellings
Civilization VI and Stellaris are strategy games that make in-
tegral use of both AI (to populate simulated worlds with rival
factions for players to interact with) and PCG (to generate
world maps on which civilizational struggles may play out
in interesting and unexpected ways from one playthrough to
the next). Both games have attracted sizable player commu-
nities, including some players who engage with the game by
constructing retellings and posting them to public forums.
We hypothesized, based on our own experience in both
games, that Stellaris facilitates player storytelling better than
Civilization VI, and therefore that Stellaris retellings would
be both more common (relative to the size of the player
community) and more narratively rich than Civilization VI
retellings. Our notion of “narrative richness” at this point
was still vague and undefined, so we turned to analysis of ex-
isting retellings to determine what a narratively rich retelling
might look like.
We used a grounded theory methodology (Salisbury and
Cole 2016) to analyze retellings found “in the wild,” in on-
line player communities for both Stellaris and Civilization
VI. GTM enables a researcher to simultaneously analyze a
body of artifacts (in this case retellings) and develop a the-
ory about what elements of these artifacts are salient, typi-
cally embodied as a codebook which evolves over the course
of the analysis and is used to note down the features of spe-
cific artifacts as they are analyzed. In this particular study,
we adopted the Constructivist Grounded Theory flavor of
GTM (Charmaz 2006), which frames the researcher as “co-
creating meaning within the domain they are studying” (Sal-
isbury and Cole 2016) and focuses on providing lenses for
analysis rather than presenting a single objectively correct
model of what matters about the domain.
In applying this methodology to retellings, we first located
the largest Reddit subreddit for each of the games we wanted
to study—/r/Civ and /r/Stellaris respectively—and gathered
50 retellings from each subreddit. Retellings were located
by searching these subreddits for the keywords “stories” and
AAR” (an abbreviation of the phrase “after-action report,”
commonly used to tag retellings in strategy game commu-
nities). Retellings from /r/Civ were also manually filtered
to ensure that they were Civilization VI retellings specifi-
cally. We then began analyzing these retellings, making note
as we went of recurring themes that appeared across multi-
ple retellings, and developing a theory which we continually
updated of what elements of retellings could reflect design
features of the games that were employed to create them.
Similar methods have previously been applied to the qual-
itative coding and analysis of other player-generated texts
around games, such as game reviews (Bopp, Mekler, and
Opwis 2016).
The codes we chose to include in the codebook were
based largely on our own intuitive understanding of what
constituted narrative interest in the context of retellings,
which grew more sophisticated as we analyzed more of the
“naturally occurring” Civilization VI and Stellaris retellings
available to us. Eventually, we reached a point of saturation
at which the study of additional retellings ceased to yield
new codes for the codebook. The set of codes included in
our final codebook is the set that we had developed by this
point in our analysis.
Ultimately, we analyzed 20 Civilization VI retellings
and 20 Stellaris retellings (chosen at random from the 50
retellings we had gathered for each game) before reaching
saturation. The final codebook (described in Table 1) con-
tained 10 codes in total, plus a qualitative “story type” clas-
sification describing the focus of the retelling in terms of
subject matter. We encountered four broad story types in
Civilization VI and Stellaris retellings:
An unstructured chronicle of game events, typically com-
prising everything that had taken place in a particular
playthrough so far. (16 Civ VI stories, 9 Stellaris stories)
The story of a demarcated period in history, with a clear
start and end—such as a particular war, or the formation
of a federation. (3 Civ VI stories, 6 Stellaris stories)
The biography of a particular named character, generally
some sort of leader. (1 Civ VI story, 4 Stellaris stories)
A day in the life of an individual member of a larger fac-
tion or civilization. (0 Civ VI stories, 1 Stellaris story)
Stellaris retellings are substantially more diverse in sub-
ject matter than Civilization VI retellings. For both games,
unstructured chronicles of game events are the most com-
mon story type, but Stellaris stories are far more evenly dis-
tributed across the four story types, and also include the
only example we identified of a “day in the life of a citi-
zen” story—wherein the point-of-view character was not the
player, not an outside narrator, and not modeled directly as
an individual character within the game’s systems.
In both Civilization VI and Stellaris, mixing of perspec-
tives within a single retelling is common. Of the Civiliza-
tion VI retellings we studied, a majority are written partly
or wholly in player perspective. Instances of narrator per-
spective are uncommon, and we identified only a handful
of instances of in-world perspective. Perspective in Stellaris
retellings is more evenly mixed, with in-world perspectives
appearing much more frequently.
Cast size varies, but Stellaris retellings tend to have more
named individual characters than Civilization VI retellings
overall. 13 Civilization VI stories and 7 Stellaris stories
do not mention any named individual characters, instead
Code name Code description
Player perspective A section of text written from the perspective of the player, typically using first-person voice (e.g.,
“I started in Malaysia, about the exact center of the map.”)
Narrator perspective A section of text written from the perspective of an outside narrator, typically using third-person
voice (e.g., “The Zulu declared a formal war on the Kongolese.”)
In-world perspective A section of text written from the perspective of a character in the fictional world. Includes sec-
tions written from the perspective of an in-universe chronicler. (e.g., “I rushed to the bridge, ex-
plosions resounding around me.”)
Specific fictional char-
Includes a specific fictional character. The number of these identified within a single retelling
corresponds to the “cast size” of the retelling as a whole. Applies exclusively to individuals, not to
groups or factions. (e.g., “This is the story of Yesenia Qasim, and his truely amazing contributions
to the human empire.”)
Extrapolated flavor el-
An instance of the player extrapolating story details not directly modeled in the game from in-game
occurrences (e.g., “As the colony descended into chaos, Dalurkot turned to drink.”)
In-world dialogue An occurrence of in-world dialogue spoken by a particular character (e.g., “Kristina, Bandar
Brunei has fallen to Montezuma!”)
Mechanical reason for
Mentions a mechanical or player-centric reason behind a decision taken by the player (e.g., “I
decided a religious victory didn’t sound too feasible, and so built St Michel for the relics.”)
reason for decision
Mentions a non-mechanical or character-centric reason behind a decision taken by the player (e.g.,
“Traak elected to stay and fight. He couldn’t bear to leave the home he had known for so long.”)
Reversal An established trend is reversed: perhaps the power relationship between two characters or factions
is inverted, or a character changes their mind about one of their goals (e.g., “Doug was starting to
doubt his attitude, and he started to look at Oswald in a different light.”)
A character’s current characterization is contrasted with their earlier characterization to signal that
the character has changed over time (e.g., “His military conquests have twisted his mind and Kupe
has begun to militarize on an immense scale.”)
Table 1: The codes used in the codebook we developed to note relevant dimensions of variation between Civilization VI and
Stellaris retellings. The same codebook was later applied to the analysis of retellings created in Prom Week as well.
referring exclusively to factions. Stellaris retellings, how-
ever, have a higher ceiling on cast size than Civilization VI
retellings: of the Stellaris stories we studied, the one with the
largest cast had 14 named individual characters, while the
Civilization VI story with the largest cast had only 8. Only
one Civilization VI retelling we analyzed had any named
characters that were not national heads of state. Typically
in Civilization VI retellings, national heads of state are used
metonymously to refer to the factions they lead. Stellaris,
on the other hand, explicitly models named leader charac-
ters (including governors, scientists, admirals, and generals)
who are not national heads of state, and these characters ap-
pear frequently in Stellaris stories.
Anecdotally, the frequency of reversals within a retelling
seems to align well with an intuitive understanding of which
retellings are more narratively compelling. Retellings with
no reversals typically follow a story pattern chronicling the
steady rise of the player faction, with few significant ob-
stacles along the way. Retellings with a small number of
reversals (up to three) tend to feel more compelling, while
retellings with more reversals begin to feel incoherent.
Within Stellaris retellings, certain specific game ele-
ments make especially frequent appearances. These include
scripted narrative events, especially those involving named
leader characters, which may occur randomly when certain
event-specific preconditions are met; instances in which a
named leader character gains a trait, which provide an op-
portunity for players to infer or extrapolate character devel-
opment; periods of building tension between rival factions,
often leading up to a war; and large decisive battles within
wars. These events seem to be perceived by players as par-
ticularly storyful.
Members of the Stellaris community seem to share
retellings with other community members more frequently
than members of the Civilization VI community, especially
relative to overall community size. As of May 2019, /r/Civ
had an estimated 283,000 members, but only approximately
6,050 search results for the “stories” and “AAR” keywords
combined. /r/Stellaris, on the other hand, had only an esti-
mated 140,000 members, but approximately 10,700 search
results for both keywords combined. The discrepancy is fur-
ther compounded by the fact that /r/Civ is a combined com-
munity for the entire Civilization series, and not all retellings
posted there are Civilization VI retellings specifically. This
may indicate that players find Stellaris to be a better sto-
rytelling partner, or that the stories they experience within
it seem more unique, personally significant, or otherwise
worth sharing. There may also be a degree of “social con-
tagion” involved: members of the Stellaris community may
first see others sharing stories and only thereafter decide to
share stories of their own.
Interviewing Retelling Creators
Analysis of a sizable corpus of preexisting retellings enables
some forms of insight into the variety of subjective play ex-
periences that are both possible and likely within a particu-
lar game. However, we hypothesized that additional insight
would be enabled by analyzing retellings not just as isolated
artifacts, but in conjunction with interviews with the players
who created these retellings. This poses some initial difficul-
ties: it may not be feasible to get in touch with the creators
of retellings that were posted online, and the creator’s mem-
ory of the experience of working with the game to create the
retelling may have faded with time since the retelling was
initially created.
As such, we decided that the best approach would be to
recruit a small number of players to create retellings of their
own, and then interview them about their experience. We
could then study the retellings they created in conjunction
with their interview responses, potentially giving insight into
how the structural features of a retelling are related to dimen-
sions of the creator’s subjective experience.
We recruited two undergraduate game design students to
play both Stellaris and Civilization VI, and to create two
retellings each—one based on their experiences in each
game—for a total of four retellings overall. Neither student
had played Stellaris before; one had played Civilization VI
before, but had never previously approached the game with
the explicit intent of telling a story.
We coded the retellings created by the students accord-
ing to the codebook we developed through analysis of wild
retellings. In addition, we interviewed both of the students
about their experiences creating retellings in both games.
All four student-created Civilization VI and Stellaris
retellings fell into the “unstructured chronicle of game
events” story type, which seems to be the natural default for
retellings in these games. In Civilization VI wild retellings,
deviations from this pattern are the exception, and even in
Stellaris wild retellings this pattern is the most common. In
addition, both Civilization VI retellings and one of the two
Stellaris retellings recounted game events more or less di-
rectly, with little extrapolation or introduction of new cre-
ative elements by players.
However, one of the two student-created Stellaris
retellings did introduce fictionalized and extrapolated char-
acterization to some extent. For instance, when a character
who had been appointed as the governor of a colony gained a
new trait representing a substance abuse problem, the player
framed this in their retelling as a response to the stress of
ruling during a particularly tumultuous time for that colony.
Both the trait and the troubled period of the colony’s history
were modeled in game to some extent, but they were not di-
rectly mechanically linked. Instead, the player chose to ex-
trapolate deeper characterization from the juxtaposition of
these two mechanical elements.
The relative shallowness of the majority of student-
created Civilization VI and Stellaris retellings may be due
partly to inexperience with the games being used. Stellaris
in particular is a complicated game with lots of menus that
takes some time to get used to, and neither of the two players
had played Stellaris before.
One player found it substantially more difficult to create
stories with Stellaris than with Civilization VI, and did not
feel especially strongly about the difference between Stel-
laris and Civilization VI with regard to storytelling support
or coherence: “The game went very quickly, and there was
a lot of things happening at once of which it didn’t always
tell me that it was happening. I sometimes missed impor-
tant things while I was trying to read about other impor-
tant things, which made my story feel like it was jumping
around from point to point and not very linear.” The other
player found that Stellaris was less difficult to tell stories
with, strongly supportive of their storytelling process, and
substantially better than Civilization VI at promoting story
coherence. This, too, may be due in part to disparity in previ-
ous experience: neither had played Stellaris before, so both
were confronted by a significant learning curve, while one
had played Civilization VI before, reducing the impact of
the learning curve.
Due to the role of experience in mitigating confusion, in-
terviews with experienced creators of Civilization VI and
Stellaris retellings would likely yield different insights. For
this study, however, we wanted to establish a baseline for
comparison with Prom Week. To the best of our knowledge,
there are no experienced creators of Prom Week retellings
in the wild, so we instead elected to use players with little
experience in any of the games being studied.
In future studies taking a retellings-based approach to
evaluation, it may be desirable to give players more time
to get acclimated to the game before having them try to tell
stories with it. Creators of retellings in the wild are often
not first-time players, and sometimes mention when report-
ing play experiences that this is “the most interesting game
they’ve had so far” out of several previous playthroughs. It
may simply be the case that it takes time to learn the tools
needed to tell stories effectively with any sufficiently com-
plicated game.
Players in both Civilization VI and Stellaris reported feel-
ing consistently surprised by the stories they ended up telling
in collaboration with the game. We address this further in the
following section.
Evaluating Prom Week Through Retellings
Prom Week is an AI-based research game that makes cen-
tral use of the Comme il Faut (McCoy et al. 2014) “social
physics engine” to determine the set of possible actions and
responses of a cast of eighteen virtual high school students.
Each turn, players select pairs of characters and have them
engage in a socially-charged action (such as “Ask Out” their
crush on a date, or “Backstab” their trusting friend), the op-
tions of which are determined based on thousands of so-
cial considerations that take into account the current social
state and the characters’ past history with one another. Each
player-selected action further evolves the social state, which
further affects character considerations for subsequent ac-
tions, resulting in playtraces that quickly become entirely
unique for each player.
To evaluate the impact of Prom Week’s social reasoning
on player narrativization of their play experiences within the
game, we adopted an approach inspired by ablation stud-
ies, in which a specific system is selectively disabled and
comparisons made between scenarios in which the system is
active and scenarios in which it is not. We created an alterna-
tive version of Prom Week that had been modified to disable
the social reasoning system, replacing “intelligent” charac-
ter decisions about which action they should take next with
random selection from the entire pool of possible actions.
We refer to the publicly available version of Prom Week as
PW-A and the modified version with social reasoning dis-
abled as PW-B.
We hypothesized that Prom Weeks social reasoning and
explicit modeling of character intentionality would support
player storytelling well, leading players to feel that story-
telling with the game was easier; that the stories they told
were more coherent; and that their storytelling process was
better supported in PW-A than in PW-B. We also hypothe-
sized that players would find the stories they told with PW-A
more surprising (due to the social reasoning giving the game
a kind of coherent creative agency, which might contradict
or react in unpredictable ways to the player’s own creative
intent); that they would enjoy telling stories more with PW-
A; and that they would experience writing block more fre-
quently when telling stories with PW-B.
The same two undergraduate students we had recruited
to create Civilization VI and Stellaris retellings were again
instructed to create two retellings each: one in PW-A, and
one in PW-B. They were not informed of the nature of the
difference between the two versions of the game, and neither
had encountered Prom Week before. We exposed them to the
two versions of the game in opposite orders, with the goal of
minimizing possible ordering effects.
Interview responses showed three of our hypotheses to
be correct: PW-A was consistently experienced as being less
difficult to tell stories with, better for story coherence, and
more supportive of player storytelling than PW-B.PW-A in
particular prompted a variety of interview responses from
both players about its success at supporting storytelling:
“This game was the perfect mixture of detailed and open-
ended. This made it easier to have a good outline for a
story, and personally fill-in the specifics with my imagi-
nation. It also wasn’t completely up to you, because the
game would sometimes end before you managed to com-
plete all your goals so sometimes you didn’t get the end-
ing you were hoping for, which allows for stranger details
in your story and more realistic life-like endings.
“I started to understand certain characters’ personality and
therefore I started considering things from their perspec-
tive or why they had made those decisions”
“events occurred that I felt I could exaggerate since the
experience were very broad but the feelings that came out
could be added onto. For example, when I wrote about the
protagonist going to talk to a crying Naomi after she was
heartbroken, I tried to write a convincing talk between the
two rather than what game actually had.
“sometimes it felt out of your control which made it more
of a compelling story to write”
Analysis of the retellings created with PW-A reveals that
they both contain six or fewer reversals, while both of the
retellings created with PW-B contain eight or more. All four
of the Prom Week retellings are of similar length (between
1500 and 2000 words), so the PW-B retellings are substan-
tially more reversal-dense than the PW-A retellings. This
corresponds well to both our anecdotal analysis of coherence
in relation to reversal count in Civilization VI and Stellaris
retellings, and to player interview responses regarding the
coherence of retellings constructed with PW-A and PW-B.
Contradicting our hypothesis regarding surprise, inter-
view responses showed players to be approximately equally
surprised by all the stories they told in both PW-A and PW-
B, with no meaningful difference between the two versions.
Similarly high levels of surprise were reported across the
board when interviewing the same players about the stories
they told using Civilization VI and Stellaris. This may be an
indicator that players will usually be surprised by the stories
they end up telling in collaboration with games as long as the
game is given meaningful creative input into the storytelling
process. Alternatively, it may simply mean that we framed
the question poorly, or that asking about the extent to which
players were surprised by the stories they ended up telling is
unlikely to yield any meaningful responses in this context.
Of the two players, one reported a consistently low-to-
moderate level of enjoyment of the storytelling process in
both PW-A and PW-B. The other enjoyed creating stories
with PW-A but reported greater frustration in PW-B due to
the greater unpredictability of character behavior: “At first I
found it fun, but it started to get more and more annoying
when the players didn’t respond how I wanted them to, or
how I had expected them to after playing the game for a bit.
This player had been introduced to PW-B first, so their ini-
tial expectations had not been shaped by the greater coher-
ence of character behavior in PW-A. This seems to suggest
that their frustration stemmed directly from the difficulty of
accurately predicting character behavior in the absence of
social reasoning.
Player interview responses on the subject of writer’s block
diverged substantially. One player frequently experienced
writer’s block in PW-A but not in PW-B, because the con-
stant failure of characters to act in predictable ways pro-
pelled the story forward despite decreasing coherence: “I
didn’t really feel blocked, I just kept writing about my failed
attempts to secure the goals.” The other frequently experi-
enced writer’s block in PW-B but not in PW-A, due to unpre-
dictable character behaviors forcing constant re-planning of
the direction the story might go. This may indicate the exis-
tence of two different attitudes toward story construction—
one more improvisational, one more rigidly planned—which
require different kinds of creativity support.
For all of the games examined here, many of the more narra-
tively resonant retellings seem to be driven by a process that
we term extrapolative narrativization. Rather than stopping
at literally reporting the events of a particular play experi-
ence as they are modeled within the game (even if in a heav-
ily filtered form), players who create retellings are prone to
seizing on particular details or combinations of details that
appear within the game—often apparent coincidences, or el-
ements of game events that are largely inconsequential from
a gameplay perspective—and extrapolating from these de-
tails to include additional story elements that are not directly
modeled within the game’s systems.
Creators of Stellaris retellings in particular have a ten-
dency to engage in extrapolative worldbuilding: inventing
details about the nature of the fictional world in which game-
play takes place, for instance by describing elements of day-
to-day life in an alien civilization at a far greater level of
granularity than is directly modeled by the game’s systems.
We argue that the high frequency of extrapolative narra-
tivization in Stellaris retellings, especially in contrast to the
relative infrequency of extrapolative narrativization in Civi-
lization VI retellings, supports our hypothesis that Stellaris
more effectively supports player storytelling than Civiliza-
tion VI, which seems less liable to prompt its players to
engage in imaginative worldbuilding processes. We suspect
that Stellaris’s use of procedural content generation to create
non-player characters and rival factions is partly responsible
for this difference. Due to their procedurally generated na-
ture, Stellaris characters and factions are underdetermined,
providing sketchy outlines of people and civilizations that
players aren’t familiar with from elsewhere. Randomly as-
signed character traits, species traits and portraits, civics,
and other aspects of government structure provide lots of
evocative details from which players are free to hang story
bits, without worrying that they may be contradicting actual
history as is sometimes the case in Civilization VI.
Similarly, Stellariss direct modeling of a cast of per-
sistent, named leader characters who participate directly
in gameplay events—such as researching technologies, ex-
ploring star systems, fighting battles, and ruling particular
planets—provides players with still more opportunities to
extrapolate story details from things that happen in-game.
Prom Week takes this a step further by directly modeling not
only individual characters but also their relationships and so-
cial motivations, and then exposing this model directly to the
player. This facilitates storytelling even more strongly than
Stellaris’s modeling of characters, albeit only in the presence
of social reasoning, which imposes a kind of predictability
on the characters that aids coherent storytelling.
This analysis suggests several possible things you can do
as a designer to facilitate player storytelling: provide evoca-
tive “hooks,” such as character traits, that can be interpreted
in a few different ways, especially in juxtaposition with
other hooks; leave some details of worldbuilding and char-
acterization strategically underdetermined, to let players fill
them in; explicitly model individual characters in game sys-
tems; enable these individual characters to participate in a
variety of game events; and do what you can to either model
character intentionality, give players hooks with which to in-
fer intentionality, or some combination of both.
Potential Drawbacks of Retelling-Based Evaluation
Compared to other approaches to the evaluation of play
experiences, retellings-based approaches can be time-
consuming. The GTM-inspired qualitative coding process
we describe here may take many hours, especially if attempt-
ing to develop one’s own codebook rather than using an ex-
isting one (which we encourage for future studies, especially
those not focusing on strategy games.)
Additionally, if studying a game for which retellings don’t
already exist in the wild, or if you want to have local players
create retellings so that you can readily interview retelling
creators, it takes more time for players to play the game and
write a story with it than it does to go through a traditional
playtest or take a simple survey. If the game is at all compli-
cated, you will likely also want to allow players some time
to get used to playing the game before they begin creating
their stories. In this study, we did not allow players enough
time to get acclimated prior to retelling construction.
Echoing Eladhari, we suggest that the existence of a sizable
corpus of “naturally occurring” retellings for a particular
game may serve as an indicator that the game is in some
sense a success. We also conclude that studying the content
of retellings can provide deeper insight into what game el-
ements lend support to player storytelling; what overall tra-
jectories and themes are possible and probable in player sto-
ries emerging from a particular game; and what dimensions
of play experiences stand out to players as especially salient,
compelling, and worth sharing with others.
Retellings in the wild capture the bits of subjective player
experience that players feel are salient enough to be worth
telling others about. Considering a number of retellings for
the same game in aggregate can also give a sense of the
overall diversity of compelling stories as subjectively ex-
perienced by players, something which is difficult to ac-
cess through asking players directly about their experience.
Moreover, many retellings we examined made direct or indi-
rect reference to particular game elements, systems, or emer-
gent play patterns. Considering which of these elements tend
to appear more frequently in retellings can give researchers
a sense of which parts of games tend to play a role in creat-
ing compelling player experiences and helping players nar-
rativize these experiences.
Simultaneously, soliciting players to create retellings as a
research method allows you to interview retelling creators
about their subjective experience. Studying retellings and
interviews in conjunction gives you more insight than you
would be able to get from studying either alone.
Future Work
Going forward, we hope to apply the approach developed
here to the evaluation of other AI-based and PCG-based
games. The ablation study-inspired approach we took to the
evaluation of Prom Week seems especially likely to be useful
for evaluating research games, which we can readily modify
to create variants with certain systems disabled.
We also intend to conduct interviews with the creators of
several high-profile or especially well-crafted retellings—
such as Alice and Kev, a lengthy episodic Sims 3 retelling
about a homeless family mentioned by Eladhari in her earlier
work on the subject of retellings (Eladhari 2018). By talking
to the creators of these especially elaborate retellings, we
hope to learn more about why players seek out certain games
as storytelling partners and what features can assist players
in constructing their own narratives more effectively.
Acknowledgements We would like to thank Anthony
Medina and Katie Moses for their contributions of retellings
and their assistance with collection and coding of “wild”
Bopp, J. A.; Mekler, E. D.; and Opwis, K. 2016. Neg-
ative emotion, positive experience?: Emotionally moving
moments in digital games. In Proceedings of the 2016
CHI Conference on Human Factors in Computing Systems,
2996–3006. ACM.
Charmaz, K. 2006. Constructing grounded theory: A prac-
tical guide through qualitative analysis. Sage.
Eladhari, M. P. 2018. Re-tellings: the fourth layer of narra-
tive as an instrument for critique. In International Confer-
ence on Interactive Digital Storytelling, 65–78. Springer.
King, N. 2004. Using interviews in qualitative research.
Essential guide to qualitative methods in organizational re-
search 2:11–22.
Laaksolahti, J.; Isbister, K.; and H¨
ok, K. 2009. Using the
sensual evaluation instrument. Digital creativity 20(3):165–
Leech, B. L. 2002. Asking questions: Techniques for
semistructured interviews. PS: Political Science & Politics
McCoy, J.; Treanor, M.; Samuel, B.; Reed, A. A.; Mateas,
M.; and Wardrip-Fruin, N. 2014. Social story worlds with
Comme il Faut. IEEE Transactions on Computational intel-
ligence and AI in Games 6(2):97–112.
Ryan, J. 2018. Curating Simulated Storyworlds. Ph.D. Dis-
sertation, University of California, Santa Cruz.
Salisbury, J. H., and Cole, T. 2016. Grounded theory in
games research: Making the case and exploring the options.
In Proceedings of the 1st International Joint Conference of
DiGRA and FDG.
Smith, G.; Gan, E.; Othenin-Girard, A.; and Whitehead, J.
2011. PCG-based game design: enabling new play experi-
ences through procedural content generation. In Proceed-
ings of the 2nd International Workshop on Procedural Con-
tent Generation in Games, 7–10. ACM.
Treanor, M.; Zook, A.; Eladhari, M. P.; Togelius, J.; Smith,
G.; Cook, M.; Thompson, T.; Magerko, B.; Levine, J.; and
Smith, A. 2015. AI-based game design patterns. In Pro-
ceedings of the 10th International Conference on the Foun-
dations of Digital Games (FDG 2015). ACM.
... Eladhari (2018) proposes another approach, where examining the stories players retell offers valuable insights into their experiences, since they took the time to write and share them. Kreminski et al. (2019) evaluate AI-based games by gathering players' stories from Reddit 2 , and interviewing their authors. ...
... In 2021, industry leading companies such as Facebook, Epic Games and Microsoft announced their active efforts towards building the so called Metaverse (Duffy, 2021), a digital universe extending the physical world we live in, with fully immersive 3D capability, supporting social interaction and vast cultures to emerge (Dionisio et al., 2013). Koster (2021) discusses the context of the metaverse 'hype' and suggests that much of the appeal of the metaverse have already happened in existing online worlds, that online world technologies is the backbone of the metaverse and how we still have to answer big questions around technology for standards, digital ownership etc. While the exact motivations for how or why these companies announce their strong interests in developing such technology remains speculative, it indicates strong future demand for better understanding persistence not only for game systems, but also for larger interconnected worlds with meaningful exchange between the physical and digital. ...
... While the exact motivations for how or why these companies announce their strong interests in developing such technology remains speculative, it indicates strong future demand for better understanding persistence not only for game systems, but also for larger interconnected worlds with meaningful exchange between the physical and digital. I propose that future researchers build upon this previous work and establish an updated cohesive taxonomy surrounding concepts such as meta persistence (Gustafsson et al., 2020), metaverse (Dionisio et al., 2013;Koster, 2021), meta gaming (Carter et al., 2012) and magic circle (Huizinga, 1955). This will help founding robust structures for new future design strategies that can leverage meta persistence and directly support peoples' activities across different digital environments. ...
Players in digital game worlds, such as Massively Multiplayer Online Games (MMO), generate long standing histories with their characters, but cannot express or see persisted traces of their adventures within the environment. I argue that understanding the role of persistence and the emergence of player narratives in digital game worlds can lead to novel systems and tools for increasing players'engagement and help game designers produce content. I combine multiple methods for understanding player narratives, and present We Ride, an MMO that embodies and tests the Narrative Substrates framework for building game architectures and mechanics that leverage persistent player narratives. Multiple studies show how Narrative Substrates successfully encourages players to codesign new content from their own persisted narratives, letting them shape the world and contribute novel content. Finally, I discuss the challenges of this work and suggest several promising directions for future work.
... We believe that several recent developments in the study of interactive digital narrative justify a reexamination of the player-authorship perspective on IEN. The study of retellings [6], or the stories that players tell about their play experiences (often in IEN games like The Sims and Dwarf Fortress), has called attention to the cultural significance of these stories-and to the extensive work that players do in constructing them, for instance by embellishing or extrapolating beyond the bare events of play to craft a better story [22,13] and constructing stories that ironically comment on or critique the IEN systems with which they were produced [33]. James Ryan [26] has drawn a clear distinction between the raw material of simulation and a particular telling or narrativization of this material, proposing a new curationist perspective on emergent narrative that highlights the work done by the human interactor in crafting a coherent story from the disorderly and overwhelming outputs of a simulation engine. ...
... This conflation-the same one that Ryan [26] argues against, with his admonition that the simulated storyworld is not itself an emergent story-leads Aarseth to treat the unedited transcripts of MUD play sessions as the artifacts of narrative interest. Without the distinction between narrative material and retelling advanced in more recent work [26,6,22,14,13,16,33], Aarseth's view on the role of the user in IEN ends up suspended ambiguously between the user-as-participant and user-as-author perspectives. ...
Full-text available
We trace the history of emergent narrative as both a term and a concept, with a particular focus on interactive emergent narrative (IEN): the use of emergent narrative as an approach or solution to the problems presented by the combination of interactivity and narrativity. We argue that discussion of IEN—both historical and modern—often fails to distinguish between two contrasting uses of IEN: to enable player participation as a character in an authored storyworld, and to enable player authorship of new stories. We additionally advocate for a clearer distinction between these perspectives, so that IEN systems which aspire to enable player authorship can be developed, studied and evaluated on their own terms.
... An incorrectly installed version of Tableau Machine that generated random artworks without taking real household data into account, however, failed to prompt user reflection -simultaneously demonstrating the power of "ambiguity as a resource" [19] and the need for systematic guidance of generative systems that produce ambiguous artifacts. Similarly, a study of player storytelling practices around simulation-driven digital games [28] found that player stories are often driven by a process of extrapolative narrativization, in which players elaborate on concrete elements of the scenario modeled in the game's systems by adding their own interpretations. This study [28] found that the generative game Prom Week [41] successfully supports this kind of storytelling by players, but not when the underlying social simulation is altered to produce random outcomes. ...
... Similarly, a study of player storytelling practices around simulation-driven digital games [28] found that player stories are often driven by a process of extrapolative narrativization, in which players elaborate on concrete elements of the scenario modeled in the game's systems by adding their own interpretations. This study [28] found that the generative game Prom Week [41] successfully supports this kind of storytelling by players, but not when the underlying social simulation is altered to produce random outcomes. Meanwhile, Kate Compton describes one genre of successful Twitter bots as generating "imagination playgrounds": strategically ambiguous scenarios with room deliberately left for human interpretation of the ambiguous elements. ...
... WAWLT represents an example of AI-based game design [6] inspired by the study of existing player storytelling practices [5,12] in simulation-driven games. In designing WAWLT, we set out to provide creativity support features that would help players overcome four major barriers to creativity documented in [13]: fear of the blank canvas; fear of judgment; writer's block; and perfectionism. ...
... The transcript holds a running record of "the story so far": an artifact of play [1] containing short system-generated summaries of all player-accepted action suggestions since the start of the play session, interleaved with more detailed player-generated prose descriptions of these actions. By allowing players to annotate events with their own descriptions, we aim to provide support for extrapolative narrativization [12]: a player storytelling behavior in which players seize on and elaborate minor details in the stories they tell about their play experiences, regardless of whether these details are explicitly modeled in the simulation. Editable free-text descriptions of events give players a place to decide for themselves what aspects of an event are most important, and to establish and reference recurring story elements not modeled in the computational system. ...
... Further, one could expect the human to use their understanding of the system implicitly or explicitly to improve the utility or informativeness of the samples as they observe the progression of the system [108]. In some cases, the samples or information can come as human retellings of past experience [55]. This can also be reversed in the sense that the system can be designed to improve the types of responses in order to guide the human and improve the information received or queries of the user [119]. ...
There is a clear desire to model and comprehend human behavior. Trends in research covering this topic show a clear assumption that many view human reasoning as the presupposed standard in artificial reasoning. As such, topics such as game theory, theory of mind, machine learning, etc. all integrate concepts which are assumed components of human reasoning. These serve as techniques to attempt to both replicate and understand the behaviors of humans. In addition, next generation autonomous and adaptive systems will largely include AI agents and humans working together as teams. To make this possible, autonomous agents will require the ability to embed practical models of human behavior, which allow them not only to replicate human models as a technique to "learn", but to to understand the actions of users and anticipate their behavior, so as to truly operate in symbiosis with them. The main objective of this paper it to provide a succinct yet systematic review of the most important approaches in two areas dealing with quantitative models of human behaviors. Specifically, we focus on (i) techniques which learn a model or policy of behavior through exploration and feedback, such as Reinforcement Learning, and (ii) directly model mechanisms of human reasoning, such as beliefs and bias, without going necessarily learning via trial-and-error.
... The prototype based on the top of the Unreal Tournament game engine relies on a 'causal engine', which essentially bypasses the native Physics engine to generate alternative consequences to player interventions. The evaluation method [78] can be applied to the human-centered evaluation of AI-based games, grounded in the analysis of player retellings of their play experiences in Civilization VI, Stellaris, and two distinct versions of the research game Prom Week. The reason is that it is difficult to understand through existing evaluation methods, such as the typical narrative structure that players tend to have in their minds when playing a specific game. ...
Full-text available
(Please download the latest version of this survey article from arXiv: ) As the latest buzzword, Metaverse has attracted great attention from both industry and academia. Metaverse seamlessly integrates the real world with the virtual world and allows avatars to carry out rich activities including creation, display, entertainment, social networking, and trading. Thus, it is promising to build an exciting digital world and transform a better physical world through the exploration of the metaverse. In this survey, we dive into the metaverse by discussing how Blockchain and Artificial Intelligence (AI) fuse with it by investigating the most related studies across the metaverse components, digital currencies, AI technologies, and applications in the virtual world, and blockchain-empowered technologies. Further exploitation and interdisciplinary research on the fusion of AI and Blockchain towards the metaverse will definitely require collaboration from both academia and industries. We wish that our survey can help researchers, engineers, and educators build an open, fair, and rational future metaverse.
... Recently, several parallel developments in the study of interactive emergent narrative (IEN) games have suggested that some players view these games primarily as expressive tools for the creation of new stories [15], rather than as devices for experiencing or participating in a partly pre-authored story or "multiform plot" [22, p. 347]. The study of retellings [7], or player-created stories about gameplay experiences, has highlighted play practices in which players make use of IEN games as tools for expressive story-making, while extensively embellishing the resulting stories by adding detail not modeled by the game itself [14,21] or even ironically commenting on the flaws or limitations of the game as a storytelling tool within the resulting retellings [33]. Ryan's work on simulationdriven IEN [26] has drawn a distinction between emergent narrative (a particular telling or representation of a series of game events) and the raw narrative material produced by simulation directly, which is only transformed into narrative by a process of curation-often undertaken by the player. ...
Full-text available
Interactive narrative systems are often embedded in games: works of playable media that enable players to participate in or experience a story through game mechanics. But play practices directed toward the expressive creation of story seem to challenge a games-centric understanding of narrative play. Consequently, we propose that some interactive narrative systems can be better understood as a different form of playable media: narrative instruments, analogous to musical instruments in their provision of support for authorship-oriented forms of play.
... These narrativizations are influenced by a player's "own resources", the stories they're most interested in telling through this sort of imaginative play. The process by which this happens is one of extrapolative narrativization [7]: in making narrative sense of Sims play, players do not simply transcribe the series of game events as it unfolds. Instead, they confer additional layers of interpretation on these events, adding extra details to the narrative-as-perceived-which then influences what actions the player is inclined to take next. ...
Full-text available
Eve Sedgwick’s theory of reparative reading offers a mode for interpreting text that is “additive and accretive” and “wants to assemble and confer plenitude on an object”. It was developed in response to what Sedgwick calls “paranoid reading”, which embodies the desire to locate a stable, canonical meaning and is therefore hostile to the notions of multiplicity and surprise. We argue that interactive digital narrative can be productively understood through the paranoid/reparative framing, and that in particular, narrative sandbox games—games that lean heavily on emergence to produce a narrative effect—invite a kind of reparative play. Narrative sandbox systems function by producing deliberately incomplete artifacts that facilitate a diversity of reparative meaning-making processes by the player; they invite repair by arriving in disrepair.
This chapter discusses the concept of the retelling as a narrative product and as an instrument of narrative design analysis and criticism. Focusing on longform video essays as a prominent category of critical retellings, we analyze transcriptions of Noah Caldwell-Gervais’ video essay and Jose Antonio Vargas’ video review of the 2020 videogame The Last of Us Part II. Using web-based reading and analysis environment Voyant, we investigate the indicators of how they approached and discussed the game’s narrative design and compare their texts as examples of critical and popular retellings of video games.
This paper presents the early findings of a pilot study on the analysis of longform video essays as critical retellings of video game narratives. Using web-based reading and analysis environment Voyant, the study explores the indicators of how players critically approach and discuss game narrative design.
Full-text available
There is a peculiar method in the area of procedural narrative called emergent narrative: instead of automatically inventing stories or deploying authored narrative content, a system simulates a storyworld out of which narrative may emerge from the happenstance of character activity in that world. It is the approach taken by some of the most successful works in the history of computational media (The Sims, Dwarf Fortress), but curiously also some of its most famous failures (Sheldon Klein's automatic novel writer, Tale-Spin). How has this been the case? To understand the successes, we might ask this essential question: what is the pleasure of emergent narrative? I contend that the form works more like nonfiction than fiction---emergent stories actually happen---and this produces a peculiar aesthetics that undergirds the appeal of its successful works. What then is the pain of emergent narrative? There is a ubiquitous tendency to misconstrue the raw transpiring of a simulation (or a trace of that unfolding) as being a narrative artifact, but such material will almost always lack story structure. So, how can the pain of emergent narrative be alleviated while simultaneously maintaining the pleasure? This dissertation introduces a refined approach to the form, called curationist emergent narrative (or just curationism), that aims to provide an answer to this question. Instead of treating the raw material of simulation as a story, in curationism that material is curated to construct an actual narrative artifact that is then mounted in a full-fledged media experience (to enable human encounter with the artifact). This recasts story generation as an act of recounting, rather than invention. I believe that curationism can also explain how both wild successes and phenomenal failures have entered the oeuvre of emergent narrative: in successful works, humans have taken on the burden of curating an ongoing simulation to construct a storied understanding of what has happened, while in the failures humans have not been willing to do the necessary curation. Without curation, actual stories cannot obtain in emergent narrative. But what if a storyworld could curate itself? That is, can we build systems that automatically recount what has happened in simulated worlds? In the second half of this dissertation, I provide an autoethnography and a collection of case studies that recount my own personal (and collaborative) exploration of automatic curation over the course of the last six years. Here, I report the technical, intellectual, and media-centric contributions made by three simulation engines (World, Talk of the Town, Hennepin) and three second-order media experiences that are respectively driven by those engines (Diol/Diel/Dial, Bad News, Sheldon County). In total, this dissertation provides a loose history of emergent narrative, an apologetics of the form, a polemic against it, a holistic refinement (maintaining the pleasure while killing the pain), and reports on a series of artifacts that represent a gradual instantiation of that refinement. To my knowledge, this is the most extensive treatment of emergent narrative to yet appear.
Conference Paper
Full-text available
Grounded Theory Methodology (GTM) is a powerful way to develop theories in domains where there are obvious opportunities to contribute in the form of carefully developed descriptive or explanatory conceptual theories. Reasonably nascent areas of academia, such as Game Studies, stand to particularly benefit from the development of new theoretical accounts. Yet, despite its proven utility in a wide range of fields and its history of rigorous methodological debate, many researchers are wary of using GTM. Conversely, many claim use of GTM but do not present an understanding of GTM's rich tradition and how this may impact their results and conclusions. This paper seeks to provide an overview of GTM, its main variants, and how they can be effectively used in research. We examine how GTM has been used in the field of games research and argue that GTM rightly be regarded a highly relevant method here.
Conference Paper
Full-text available
This paper proposes a model for designing games around Artificial Intelligence (AI). AI-based games put AI in the foreground of the player experience rather than in a supporting role as is often the case in many commercial games. We analyze the use of AI in a number of existing games and identify design patterns for AI in games. We propose a generative ideation technique to combine a design pattern with an AI technique or capacity to make new AI-based games. Finally, we demonstrate this technique through two examples of AI-based game prototypes created using these patterns.
Full-text available
In an interview, what you already know is as important as what you want to know. What you want to know determines which questions you will ask. What you already know will determine how you ask them.
Conference Paper
Emotions are key to the player experience (PX) and interest in the potential of games to provide unique emotional, sometimes uncomfortable experiences is growing. Yet there has been little empirical investigation of what game experiences players consider emotionally moving, their causes and effects, and whether players find these experiences rewarding at all. We analyzed 121 players' accounts of emotionally moving game experiences in terms of the feelings and thoughts they evoked, different PX constructs, as well as game-related and personal factors contributing to these. We found that most players enjoyed and appreciated experiencing negatively valenced emotions, such as sadness. Emotions were evoked by a variety of interactive and non-interactive game aspects, such as in-game loss, character attachment and (lack of) agency, but also personal memories, and were often accompanied by (self-)reflection. Our findings highlight the potential of games to provide emotionally rewarding and thought-provoking experiences, as well as outline opportunities for future research and design of such experiences. They also showcase that negative affect may contribute to enjoyment, thereby extending our notion of positive player experience.
This paper presents Comme il Faut (CiF), an artificial intelligence system that matches character performances to appropriate social context, with the goal of enabling authors to write high-level rules governing expected character behavior in given social situations, rather than specific fixed choice points in a curated narrative structure. CiF models characters with a complex set of traits, feelings, and relationships, who can form intents, take actions, relate to a shared cultural space, and remember and refer to past events. A set of authored rules encoding appropriate behavior within a specific story world allow these characters to select actions to take (and respond to actions by others) in a manner consistent with their own personal and social concerns as well as a shifting interpersonal context. Through the development and release of Prom Week, a complete game using CiF as its narrative engine, we show how the system successfully creates complex narratives that are unique for each player and directed by those players' attempts to make progress towards story goals. We also show how CiF continues to be used in several in-progress interactive experiences (Mismanor and IMMERSE), speaking to the utility and flexibility of its design.
In our research we made use of an instrument previously developed to facilitate nonverbal self-report of emotion, which consists of eight sculpted objects. We describe the use of this instrument in the assessment of three interactive storytelling experiences in a small user study and draw some conclusions about the instrument's effectiveness in supporting design.
Constructing grounded theory: A practical guide through qualitative analysis
  • K Charmaz
Charmaz, K. 2006. Constructing grounded theory: A practical guide through qualitative analysis. Sage. Eladhari, M. P. 2018. Re-tellings: the fourth layer of narrative as an instrument for critique. In International Conference on Interactive Digital Storytelling, 65-78. Springer. King, N. 2004. Using interviews in qualitative research. Essential guide to qualitative methods in organizational research 2:11-22.
PCG-based game design: enabling new play experiences through procedural content generation
  • G Smith
  • E Gan
  • A Othenin-Girard
  • J Whitehead
Smith, G.; Gan, E.; Othenin-Girard, A.; and Whitehead, J. 2011. PCG-based game design: enabling new play experiences through procedural content generation. In Proceedings of the 2nd International Workshop on Procedural Content Generation in Games, 7-10. ACM.