PosterPDF Available

Synergies between research and game design: Reflections on interactive narrative experiments by student game designers

April 2019

Conference: American Educational Research Association (AERA) Annual Meeting
At: Toronto

Authors:

Camillia Matuk

New York University

Steven Sutherland

University of Houston - Clear Lake

Show all 8 authorsHide

One way to prepare research-literate citizens is to engage them in designing research studies. However, this feat is challenging and has many practical barriers. We present StudyCrafter, a new platform that makes human behavior research more accessible by using game design and gameplay as a framework for research design and participation. In a qualitative exploration of students’ StudyCrafter projects within a graduate classroom implementation, we found that those projects that demonstrated well-executed principles of game design also showed qualities of robust research study design. We discuss the potential instructional value of using game design to inform research design, and the importance of accessible tools and environments. These findings can inform supports for developing students’ research design and literacy.

StudyCrafter 's Layout tab, through which users can select, customize, and arrange assets within scenes.

…

StudyCrafter' s Scriptor view, within which users can specify interactivity.

…

Student projects organized into a quadrant, based on their ranking along dimensions of game design (narrative and agency) and research design (experimental and mundane realism).

…

Course Selection Frenzy . This screen shows the two course options that players are given.

…

Figures - uploaded by Camillia Matuk

Content may be subject to copyright.

Content uploaded by Camillia Matuk

Content may be subject to copyright.

Content uploaded by Camillia Matuk

Content may be subject to copyright.

Findings'

• Projects*ranged*in*foci*(effect*of*interpersonal*relations*on*resource*management,*influence*of*context*on*motivation,*impact*of*marketing*strategies*on*purchasing*behavior,*implicit*bias,*effect*of*NPCs*on*player*behavior,*etc.)*

• Students*showed*diverse*abilities*in*each*of*game*design*and*research*design.*

• An*extreme*focus*on*game*design*appeared*to*downplay*the*research*design,*and*vice*versa.*

METHODS'

Synergies Between Research and Game Design: Reﬂections on Interactive Narrative Experiments

by Student Game Designers Matuk, C., Sutherland, S., Althoff, W., Snodgrass, S., Partlan, N., Smith, G., Seif El-Nasr, M. & Harteveld, C.

STUDYCRAFTER'

RATIONALE'

Game'design'as'an'opportunity'to'support'research'literacy'

• Learning*to*design*research*experiments*is*one*way*to*cultivate*citizens’*general*research*literacy,*

but*presents*barriers*[1]:*

• Conceptually,*students*struggle*with*operationalizing*variables,*manipulating*independent*

variables,*and*controlling*for*threats*to*internal*validity.*

• Practically,0studying*human*behavior*in*authentic*situations*is*difficult*in*both*laboratory*and*

real-life*settings,*and*impractical*within*course*settings.*

• Principles*of*good*game*design*align*with*those*of*good*research*design:*

• Good'games'compel*players*to*participate*by*immersing*them*in*believable*narratives,*and*by*

presenting*consequential*choices.*

• Robust'research'study'designs'compel*participants*to*behave*in*ways*that*are*consistent*with*

the*context,*in*spite*of*it*being*simulated;*they*show*experimental'and*mundane'realism.*

OBJECTIVES'

• We*observe*students*using*an*online*platform*to*create*and*run*experiments*into*human*behavior.*

We*explore*the*potential*for*using*game*design*as*a*framework*to*guide*students’*research*study*

design.*

Research'questions'

• What*principles*of*research*and*game*design*are*apparent*in*students’*projects?*

• In*what*ways*are*students’*research*study*designs*strengthened*by*their*use*of*game*design*

principles?*

CONCLUSIONS'

• Game*design*offers*a*useful*framework*for*research*study*design,*but*balancing*their*execution*is*an*

ongoing*challenge*for*students.*

• Findings*contribute*to*research*on*constructionist*gaming*[2]*and*to*the*growing*interest*in*using*games*

and*interactive*fiction*to*study*human*behavior*[3,4].*

• Future*work*might*ask:*

• How*do*different*levels*of*game*design*expertise*affect*students’*research*design*abilities?*

• What*research-relevant*attitudes,*concepts,*and*skills*might*game*design*promote*in*students?*

• What*instructor*and/or*system-automated*support*might*students*require*to*excel*in*using*games*as*

a*framework*for*learning*to*design*research*studies?*

Features'of'robust'

StudyCrafter'designs'

Example'from'

students’'projects'

Non-example'from'

students’'projects'

RESEARCH'DESIGN'

Mundane'

realism*People*and*

contexts*investigated*are*

true*to*reality,*such*that*

findings*are*likely*to*

generalize*to*real-world*

situations.*

Space0Exploration*uses*

dark*character*

silhouettes*to*

represent*the*

characters*of*the*

participant*and*a*

partner*selected*by*

the*participant.*This*

avoided*stereotyping*

the*characters*(space*

explorers)*as*

participants*could*

project*their*own*

mental*image*onto*

the*silhouettes.*

Buy0Courses*asks*players*

to*look*at*various*course*

bundle*packages*and*

make*the*best*decision*for*

a*character*named*Tom*

who*wants*to*take*a*

number*of*courses*in*

history.*However,*the*user*

is*not*provided*with*any*

other*information*and*the*

resulting*decision*may*

therefore*be*arbitrary*or*

biased*by*assumptions*

that*participants*make.**

Experimental'

realism'Goals,*

tasks,*and*choices*for*

which*there*are*

experienced*

consequences,*creating*

situations*in*which*

participation*feels*

meaningful.*

Click0Money*shows*

participants*a*

leaderboard*(although*

non-functioning)*to*

track*the*number*of*

clicks*they*perform,*

which*affects*the*

outcome*of*the*game.*

In*Wormhaven*players*are*

part*of*an*apocalyptic*

multiplayer*game*and*are*

asked*to*trade*different*

items*of*value*with*

another*player*(which*is*

randomized*in*terms*of*

experience).*Neither*the*

purpose*of*the*exchange*

nor*the*value*of*the*items*

are*communicated,*

making*this*exchange*

arbitrary.**

GAME'DESIGN'

Player'

Agency'Clear*goals,*

choices*with*feedback,*and*

performance*incentives*

give*players*the*sense*that*

their*actions*have*

consequences,*and*are*of*

their*own*volition.*

Yet0Another0RPG0

shows*participants*the*

damage*that*their*

character*takes*as*an*

outcome*of*their*

decisions.*This*can*

then*be*used*to*

inform*decisions*

about*what*upgrades*

to*select.*

In*Robot0Wars*players*

have*to*defeat*elitist*

robots.*While*the*robots*

are*creatively*put*

together,*and*the*

narrative*makes*

intelligent*use*of*humor,*

essentially*all*the*player*

does*is*to*click*through*a*

story*with*only*a*few*

actual*decisions.**

Compelling'

Narrative'

Immersion*is*created*

through*plot*and*character*

archetypes,*humor*and*

playfulness,*and*inclusive*

representations*that*

reflect*players*and*their*

realities,*and*that*together*

create*a*sense*of*being*

part*of*a*world*and*an*

unfolding*story.*

An0Unusual0Situation*

places*participants*in*

the*role*of*a*character,*

whose*situation*

unfolds*through*a*

mixture*of*narration*

and*character*

dialogue.*This*project*

also*intersperses*

scenes*with*text-only*

screens*to*convey*the*

sense*of*time*passing*

between*events.*

Additionally,*An*

Unusual*Situation*

transitions*between*

1st*person*and*3rd*

person*narrative*at*

times,*which*may*

influence*how*

participants*interpret*

the*situations*in*the*

game.**

Colors0and0Shapes*

displays*a*series*of*

screens*on*which*players*

are*asked*to*choose*which*

of*two*shapes*they*think*

is*best*associated*with*

each*color.*It*provides*no*

narrative*or*characters,*

nor*does*it*offer*any*

context*outside*of*the*

description*of*the*

experiment.*In*this*sense,*

it*resembles*a*typical*

laboratory*study,*which*

attempts*to*isolate*

variables*under*study.*

REFERENCES'

1. Woolley*JS,*Deal*AM,*Green*J,*Hathenbruck*F,*Kurtz*SA,*Park*TKH,*et*al.*

Undergraduate*students*demonstrate*common*false*scientific*reasoning*strategies.*

Thinking*Skills*and*Creativity.*2018;27:*101–113.*doi:10.1016/j.tsc.2017.12.004*

2. Kafai*YB,*Burke*Q.*Constructionist*Gaming:*Understanding*the*Benefits*of*Making*

Games*for*Learning.*Educ*Psychol.*2015;50:*313–334.*doi:

10.1080/00461520.2015.1124022*

3. Bainbridge*WS.*The*scientific*research*potential*of*virtual*worlds.*Science.*

2007;317:*472–476.*

4. Calvillo-Gámez*E,*Gow*J,*Cairns*P.*Introduction*to*special*issue:*Video*games*as*

research*instruments.*Entertain*Comput.*2011;2:*1–2.*

Week' Activity'

• Instructor*demonstrates*

StudyCrafter.*

• Students*submit*project*

proposals*and*form*teams*based*

on*study*themes.*

2* • Project*work*in*and*out*of*class*

• Receive*instructor*feedback*

3* • Students*recruit*participants*

• Start*data*collection*

4* • Analyze*data,*write*up*findings*

• Conduct*peer*review*

5* • Present*projects*

• Submit*final*reports*

Participants'&'Context'

• 21*students*(13*m,*8*f)*enrolled*in*graduate-level*course*on*

user*research*methods*for*game*design,*at*an*eastern*US*

university.*

• Students*ranged*in*academic*backgrounds,*and*in*abilities*in*

programming,*language,*and*familiarity*with*basic*research.*

• Students*completed*a*5-week*module*on*research*design,*in*

which*they*used*StudyCrafter*to*design*and*conduct*their*

own*research*studies.*

Data'&'Analysis'

• Data*included*students’*playable*projects*and*scripts,*

accompanying*final*reports*and*the*instructor’s*course*

observations.*

• Project*features*were*organized*into*a*matrix*based*on,*for*

example,*research*foci,*context*of*study*(game*or*real-life),*

and*study*design.*

• Themes*around*research*design*(mundane*&*experimental*

realism)*and*game*design*(narrative*&*agency)*were*refined*

through*consensus*coding.*

• Projects*were*ranked*along*these*thematic*dimensions.*

• Studycrafter.com*

• A*free,*online*platform*with*tools*to*visually*script*games*and*interactive*stories,*called*Projects.*

• An*Asset0Library0offers*a*choice*of*pre-designed*characters,*objects,*and*backgrounds*that*users*

arrange*within*a*Layout.*

• The*Layout*is*a*visual*of*one*or*more*Scenes.*

• An*Editor0allows*users*to*script*what*will*occur*in*each*scene*layout,*and*under*what*conditions.*

• Links*to*StudyCrafter*projects*can*be*distributed*to,*and*played*by*participants.*

• Data*from*players’*interactions*with*the*projects*are*logged*for*project*creators*to*analyze.*

Schedule'

Limited'research'design,'limited'game'design'

Course'Selection'Frenzy:'Does*scarcity*make*a*

resource*more*desirable?*

Between-subjects'design:'Participants*either*select*from*among*two*courses*with*

the*same*or*different*seat*capacities.*

Hypothesis:*Participants*will*choose*the*course*with*less*capacity*more*frequently,*

and*take*less*time*to*decide*on*this.*

Critique:'

• The*limited*narrative*context*allowed*isolation*of*the*factor*of*interest*(seat*

capacity)*

• The*lack*of*contextual*factors*that*typically*affect*course*choices*(e.g.,*credit*

status,*degree*requirements,*schedule,*instructor)*lowered*mundane*and*

experimental*realism.*

Funding'for'this'research'

was'provided'by'the'

National'Science'

Foundation,'an'IUSE'under'

award'numbers'

1736065,*1736185,1736056*

Robust'research'design,'limited'game'design'

Colors'and'Shapes:'Do*people*associate*color*saturation*with*

shape?*

Within-subjects'design:'Participants*view*a*series*of*colors*of*varying*saturation,*and*choose*either*the*

geometric*or*the*irregular*shape*that*they*feel*best*corresponds*to*it.*

Hypothesis:*Participants*will*associate*saturated*colors*with*geometric*shapes,*and*less*saturated*

colors*with*irregular*shapes.*

Critique:'

• High*in*experimental*realism,*low*in*mundane*realism.*

• A*lack*of*features*of*interactive*fiction*and*game*design*placed*this*project*closest*to*the*canonical*

lab-based*study.*

Robust'research'design,'robust'game'design''

An'Unusual'Situation:'Does0the0Halo0Effect0exist0in0

a0game0setting?*

Between-subjects'design:'Participants*re-encounter*a*long-time*acquaintance,*Bill.*

In*one*condition,*Bill*reveals*that*he’s*the*one*behind*the*player’s*unemployment.*

In*another*condition,*Bill*attributes*his*success*to*the*player.*

Hypothesis:*Participants*who*encounter*the*morally*ambiguous*Bill*vs.*the*virtuous*

Bill*will*be*more*likely*to*later*punch*the*stranger*in*the*bar.*

Critique:'

• Mundane*and*experimental*realism*were*strengthened*by*the*effective*use*of*

game*features*(e.g.,*narrative*closure,*dialogue,*pacing,*player*agency).*

Robust'research'design,'limited'game'design'

Spider'Lab:'How*does*a*fixed*vs.*random*reward*system*affect*

players’*desire*to*continue*playing?*

Between-subjects'design:'Participants*complete*rounds*of*spider*killing*to*gain*enough*“knowledge”*

points*to*inform*the*creation*of*a*pesticide.*In*one*condition,*knowledge*gained*after*each*round*is*

fixed.*In*another,*it*is*random.*After*a*number*of*rounds,*participants*are*given*the*choice*to*continue*

killing*spiders*to*gain*enough*“knowledge”*to*cure*cancer.*

Hypothesis:*Participants*will*be*more*likely*to*continue*playing*within*a*fixed*reward*system.*

Critique:*Sophisticated*game*features*(player*agency,*interactivity)*but*weak*execution*(e.g.,*grammar,*

pacing)*reduced*the*sense*of*player*immersion*and*lowered*the*study’s’*mundane*realism.**

ResearchGate has not been able to resolve any citations for this publication.

Undergraduate students demonstrate common false scientific reasoning strategies

Article

Full-text available

Dec 2017
Think Skills Creativ

American education is failing to fill the growing demand for science, technology, engineering, and mathematics (STEM) graduates. The lack of critical reasoning skills may be a causal factor in student attrition from STEM majors. Our objective in this study was to discover and describe common false strategies used by undergraduate students during the scientific reasoning process. Each of these false strategies is described, with accompanying examples from student responses, to illustrate the thinking patterns. We defined targeted areas for instruction that can lead to better performance, greater academic self-confidence, and increased retention in STEM degrees. Understanding how students think through problems and where they are making mistakes facilitates the creation of specialized programs to correct these false reasoning strategies and increase the scientific reasoning ability of students.

EXPLORING DIGITAL GAMES AS A RESEARCH AND EDUCATIONAL PLATFORM FOR REPLICATING EXPERIMENTS

Conference Paper

Full-text available

Jan 2015

Having students create their own experiments is an important way to teach science inquiry and research methods. Additionally, having individuals create experiments to replicate previous research creates a better understanding of the concepts derived from previous findings. Replication is often difficult due to limited access to the necessary tools. However, with the increased prevalence of digital games and the ease with which new games can be modified for research, games offer the promise that individuals will be able to recreate experiments previously difficult to replicate. In this paper we discuss a new digital game, Mad Science, created specifically for novice users with no programming skills to create their own experiments and to recruit and collect data from participants. Our findings suggest that replication of previous findings can be accomplished using this platform and will be a valuable tool for teaching concepts and methods from the decision science literature. Games as research method, games as instructional method, replication

Design Considerations for Creating Game-Based Social Experiments

Conference Paper

Full-text available

Apr 2015

In this paper, we explore the topic of games as research methods, which has received little attention and has much promise. More specifically, our aim is to provide insight into what design considerations are made for creating game-based social experiments. In order to achieve this, we discuss our experiences of translating a traditional experiment focused on the decoy effect into a game-based experiment. Our preliminary results suggest that monetary incentives with no real-world consequences can affect behavior even when time delays with real-world consequences were a competing factor. This has implications for how research can be gamified.

Personalized Gaming for Motivating Social and Behavioral Science Participation

Conference Paper

Full-text available

Mar 2017

Game-like environments are increasingly used for conducting research due to the affordances that such environments offer. However, the problem remains that such environments treat their users equally. In order to address this, personalization is necessary. In this paper we discuss the need to personalize gamified research environments to motivate participation by illustrating a playful platform called Mad Science, which is being developed to allow users to create social and behavioral studies. This discussion is both informed by the platform's affordances and use thus far as well as existing theories on player motivation, and contributes to theory-informed approaches to (gamified) personalization technologies.

Design of Playful Authoring Tools for Social and Behavioral Science

Conference Paper

Full-text available

Mar 2017

Playful environments are increasingly being used for conducting research. This makes a game platform for authoring research studies and teaching about how to conduct research a necessary progression. In this paper, we discuss Mad Science, a playful platform that is being created to allow users to create behavioral experiments. We discuss iterations of the authoring tools, including lessons learned, and the need for AI assistance to guide and teach users.

Assessing Computational Thinking in Students' Game Designs

Conference Paper

Full-text available

Oct 2016

Designing games requires a complex sequence of planning and executing actions. This paper suggests that game design requires computational thinking, and discusses two methods for analyzing computational thinking in games designed by students in the visual programming language Scratch. We present how these two analyses produce different narratives of computational thinking for our case studies, and reflect on how we plan to move forward with our larger analysis.

Constructionist Gaming: Understanding the Benefits of Making Games for Learning

Article

Full-text available

Oct 2015

There has been considerable interest in examining the educational potential of playing video games. One crucial element, however, has traditionally been left out of these discussions—namely, children's learning through making their own games. In this article, we review and synthesize 55 studies from the last decade on making games and learning. We found that the majority of studies focused on teaching coding and academic content through game making, and that few studies explicitly examined the roles of collaboration and identity in the game making process. We argue that future discussions of serious gaming ought to be more inclusive of constructionist approaches to realize the full potential of serious gaming. Making games, we contend, not only more genuinely introduces children to a range of technical skills but also better connects them to each other, addressing the persistent issues of access and diversity present in traditional digital gaming cultures. http://www.tandfonline.com/doi/abs/10.1080/00461520.2015.1124022

Decade of Game Making for Learning: From Tools to Communities

Article

Full-text available

Mar 2014

This chapter recounts the game-making resurgence in the twenty-first century. It focuses on how various tools and communities for game making have been utilized thus far by constructionists to heighten children's learning in and around schools. The chapter examines how the value of game making for learning has shifted over the years in terms of educational outcomes. It provides an overview of the educational software currently available to youth and identifies the salient elements of what qualifies as an effective game design tool. Low floors, high ceilings, and wide walls offer some visage of a metaphorical house in terms of what game-making tools should look like. But here a fourth component to the consideration of computational construction kits is presented: New Windows. The chapter concludes by looking forward to the future of game making and how the various tools and communities are increasingly becoming intertwined.

Updating a Classic: A New Generation of Vignette Experiments Involving Iterative Decision Making

Article

Jan 2018

Although the vignette method is widely used in psychology, it is often implemented without the key feature of iterative decision making that can affect the eventual outcome of the vignettes. This Tutorial provides an explanation of how to use Twine, an interactive narrative platform, to create vignettes with iterative decision making. Twine is an especially useful tool for experiments involving branching narratives, spatial navigation, and resource allocation. We provide code for creating exemplar experiments in social and cognitive psychology, as well as behavioral economics, and explain how to integrate Twine projects with survey-management platforms, such as Qualtrics. After following this Tutorial, researchers will be able to use Twine in their experiments to update the classic vignette method by incorporating iterative decision-making tasks.

Interactive Narrative: An Intelligent Systems Approach

Article

Mar 2013
AI MAG

Interactive narrative is a form of digital interactive experience in which users create or influence a dramatic storyline through their actions. The goal of an interactive narrative system is to immerse users in a virtual world such that they believe that they are an integral part of an unfolding story and that their actions can significantly alter the direction or outcome of the story. In this article we review the ways in which artificial intelligence can be brought to bear on the creation of interactive narrative systems. We lay out the landscape of about 20 years of interactive narrative research and explore the successes as well as open research questions pertaining to the novel use of computational narrative intelligence in the pursuit of entertainment, education, and training. Copyright © 2013, Association for the Advancement of Artificial Intelligence.