Improving Undergraduate Attitudes Towards Responsible Conduct of Research Through an Interactive Storytelling Game

Abstract

Responsible conduct of research (RCR) is an essential skill for all researchers to develop, but training scientists to behave ethically is complex because it requires addressing both cognitive (i.e., conceptual knowledge and moral reasoning skills) and socio-affective (i.e., attitudes) learning outcomes. Currently, both classroom- and web-based forms of RCR training struggle to address these distinct types of learning outcomes simultaneously. In this paper, we present a study providing initial evidence that playing a single brief session of Academical, a choice-based interactive narrative game, can significantly improve players’ attitudes about RCR. We further demonstrate the relationship between engagement with the game and resulting attitudes. Combined with our previous work showing Academical’s advantages over traditional RCR training for teaching cognitive learning outcomes, this study’s results highlight that utilizing a choice-based interactive story game is a uniquely effective way to holistically address RCR learning outcomes that drive ethical research behavior.

Full text

Improving Undergraduate Aitudes Towards Responsible
Conduct of Research Through an Interactive Storytelling Game
Katelyn M. Grasse
University of California, Santa Cruz
Santa Cruz, CA
katy@ucsc.edu
Edward F. Melcer
University of California, Santa Cruz
Santa Cruz, CA
eddie.melcer@ucsc.edu
Max Kreminski
University of California, Santa Cruz
Santa Cruz, CA
mkremins@ucsc.edu
Nick Junius
University of California, Santa Cruz
Santa Cruz, CA
njunius@ucsc.edu
Noah Wardrip-Fruin
University of California, Santa Cruz
Santa Cruz, CA
nwardrip@ucsc.edu
Figure 1: Two perspectives and corresponding choice points from Academical’s rst scenario, “The Head Start.” In this story,
the player can role-play as an adviser or a graduate student struggling to navigate the human subjects research approval
process. The two highlighted text blocks from each scene represent the player’s dialogue options.
ABSTRACT
Responsible conduct of research (RCR) is an essential skill for all
researchers to develop, but training scientists to behave ethically
is complex because it requires addressing both cognitive (i.e., con-
ceptual knowledge and moral reasoning skills) and socio-aective
(i.e., attitudes) learning outcomes. Currently, both classroom- and
web-based forms of RCR training struggle to address these dis-
tinct types of learning outcomes simultaneously. In this paper, we
present a study providing initial evidence that playing a single
brief session of Academical, a choice-based interactive narrative
game, can signicantly improve players’ attitudes about RCR. We
further demonstrate the relationship between engagement with the
game and resulting attitudes. Combined with our previous work
showing Academical’s advantages over traditional RCR training for
teaching cognitive learning outcomes, this study’s results highlight
that utilizing a choice-based interactive story game is a uniquely
Permission to make digital or hard copies of part or all of this work for personal or
classroom use is granted without fee provided that copies are not made or distributed
for prot or commercial advantage and that copies bear this notice and the full citation
on the rst page. Copyrights for third-party components of this work must be honored.
For all other uses, contact the owner/author(s).
CHI ’21 Extended Abstracts, May 8–13, 2021, Yokohama, Japan
©2021 Copyright held by the owner/author(s).
ACM ISBN 978-1-4503-8095-9/21/05. . . $15.00
https://doi.org/10.1145/3411763.3451722
eective way to holistically address RCR learning outcomes that
drive ethical research behavior.
CCS CONCEPTS
•Human-centered computing →User studies.
KEYWORDS
attitudes; engagement; choice-based; role-playing; interactive story-
telling; narrative game; education; responsible conduct of research
ACM Reference Format:
Katelyn M. Grasse, Edward F. Melcer, Max Kreminski, Nick Junius, and Noah
Wardrip-Fruin. 2021. Improving Undergraduate Attitudes Towards Respon-
sible Conduct of Research Through an Interactive Storytelling Game. In
CHI Conference on Human Factors in Computing Systems Extended Abstracts
(CHI ’21 Extended Abstracts), May 8–13, 2021, Yokohama, Japan. ACM, New
York, NY, USA, 8 pages. https://doi.org/10.1145/3411763.3451722
1 INTRODUCTION
It is essential that all researchers learn how to conduct their work
ethically, but teaching responsible conduct of research (RCR) is com-
plex and therefore not always eective [
5
,
50
]. Specically, being
able to successfully navigate ethical dilemmas requires mastery of a
combination of distinct learning outcomes, including relevant con-
ceptual knowledge (i.e., sensitivity to societal expectations), moral

CHI ’21 Extended Abstracts, May 8–13, 2021, Yokohama, Japan Katelyn M. Grasse, Edward F. Melcer, Max Kreminski, Nick Junius, and Noah Wardrip-Fruin
reasoning skills (i.e., judgement of possible solutions) and positive
attitudes about RCR (i.e., motivation to behave ethically) [
5
,
8
,
28
].
While past work evaluating traditional RCR pedagogy has success-
fully explored developing knowledge and moral reasoning skills, im-
proving attitudes is relatively understudied and underemphasized
for current RCR training methods [
29
,
45
,
53
]. Conversely, narrative
(regardless of medium) has been demonstrated as an extremely eec-
tive tool for changing an individual’s attitude [
10
,
16
,
57
]. Interactive
narrative (i.e., digital “choose your own adventure” storytelling) has
been implicated to have the same benecial eects for impacting
values [
21
], but currently there are very few peer-reviewed reports
of empirical studies that directly support this expectation [
43
,
56
].
Existing work has demonstrated the potential for choice-based
storytelling games to foster attitudes involving personal or social
change (e.g., improving health habits or increasing empathy for
marginalized groups of people), but the ecacy for this particular
medium to change attitudes about the value of professional ethics
training (specically RCR) remains unknown. This paper seeks to
extend the evidence supporting interactive narrative for improving
ethically-relevant attitudes, particularly within the context of RCR
training.
RCR pedagogy research has not yet empirically demonstrated a
single form of training that can simultaneously improve all three
key RCR learning outcomes (knowledge, moral reasoning skills and
attitudes). As a result, RCR courses often employ multiple types
of passive or active learning activities (e.g., lectures, case study
discussion and role-play) in an attempt to maximize training ef-
cacy. Development of an RCR training method (especially one
that is digitally automated) that can address this issue would be a
valuable tool to supplement existing training standards. Our previ-
ous research showed that Academical, an interactive choice-based
narrative game, was overall a more eective tool than traditional
web-based training for engaging university students and teaching
them RCR knowledge and moral reasoning skills [
38
,
39
]. Impor-
tantly, that study was the rst to empirically demonstrate the value
of an interactive narrative game for training moral reasoning skills.
In this study, we explore whether playing a single brief session
of Academical can also signicantly improve players’ attitudes
about RCR. A positive result would indicate that Academical, and
more broadly interactive narrative games in general, can eectively
train all three key abilities that drive improvements in the most
important learning outcome of all: ethical behavior. We also inves-
tigate the relationship between players’ engagement with the game
and their attitudes about RCR. We conclude by discussing how this
study supports the design and use of interactive storytelling games
to improve learning outcomes of ethically complex content.
2 RELATED WORK
2.1 Typical Eects of Traditional RCR
Training on Students’ Attitudes
Training scientists to recognize and engage in ethical behaviors is
critical to improving the quality of research, encouraging healthier
workplace practices and increasing the general public’s trust in the
scientic process [
26
]. The incidence of reported cases of research
misconduct has been increasing since the formal institution of RCR
training standards [
40
]. While there are many possible reasons for
this trend, it nonetheless highlights the importance of ensuring
RCR training ecacy.
RCR training experts advocate that the best pedagogical methods
both teach relevant cognitive skills and foster positive attitudes
about research ethics [
20
]. The logic follows that teaching cognitive
skills is only useful if the student also has the aective motivation to
apply them [
27
]. Despite the recent emphasis on attitudes as a crit-
ical learning outcome for driving ethical behavior, there is limited
research directly evaluating the extent to which traditional RCR
pedagogy improves students’ attitudes [
3
]. To add to this research,
this paper focuses on evaluating whether playing Academical, a
web-based interactive narrative game, can improve students’ atti-
tudes about RCR.
Self-administered online training courses have recently become
a staple for supplementing institutional RCR education, but very
little data has been reported about their ecacy for improving RCR
attitudes [
44
]. Studies evaluating the ecacy of classroom-based
instruction for improving RCR attitudes have reported mixed re-
sults. Powell, Allison and Kalichman showed that in-person training
signicantly improved students’ RCR knowledge but not their atti-
tudes [
45
]. Plemmons, Brody and Kalichman showed that trainees
found their training was benecial, but that it was signicantly
more eective for teaching knowledge over skills or attitudes. [
44
].
To explain this result, those authors surmised that many of the
participants had extensive research experience and likely entered
the course with well-formed attitudes and skills, thus making those
learning outcomes resilient to change. Supporting this idea, another
more recent study showed that graduate students (with limited re-
search experience) were signicantly more likely to endorse ethical
research practices after completing a 10-lecture course [
59
]. While
these studies are valuable for understanding the ecacy for in-
person training to improve RCR attitudes, it is dicult to generalize
these results due to the limited breadth of research on the topic and
the lack of consistent measurement tools or test conditions. Overall,
there does appear to be some evidence indicating that traditional
RCR pedagogy can improve attitudes, but experts still advocate
that education eorts should place more emphasis on targeting this
particular learning outcome [27].
2.2 Role-Play is an Eective Method for
Training Attitudes in RCR
Reviews examining the pedagogical ecacy of RCR training meth-
ods strongly recommend that learning activities should be engaging
and promote thoughtful consideration and discussion of relevant
ethical issues [
4
,
27
]. Role-play provides an engaging opportunity
for students to embody contending perspectives on an issue, mak-
ing it one of the most promising discussion methods for improving
comprehension and execution of ethical behavior [
11
]. This style of
active learning is eective for stimulating role-players’ emotions,
helping learners evaluate how their own feelings align with the
goals of the exercise and thus identify areas for improvement [
23
].
Much research has shown that live-action role-play is capable of
training each of the three learning outcomes (knowledge, moral
reasoning skills and attitudes) that drive improvements in behav-
ior [
46
], including for (but not limited to) topics involving ethical
issues like medicine [
54
,
62
], environmental sustainability [
52
] and

Improving Undergraduate Aitudes Towards RCR Through an Interactive Storytelling Game CHI ’21 Extended Abstracts, May 8–13, 2021, Yokohama, Japan
accounting [
58
]. While these observations are encouraging, it is
important to note that role-play activities are not always eective
for improving attitudes [47, 60].
The NIH’s RCR training mandate species that “reection on
responsible conduct of research should recur throughout a scien-
tist’s career” [
37
]. Unfortunately, traditional role-play activities
are relatively resource-intensive because they require experienced
guidance from an instructor combined with substantial time spent
with a partner to practice necessary skills. Therefore, despite its
pedagogical ecacy, classroom-based paired role-play is not a par-
ticularly convenient method for training ethical behavior. However,
there are a number of works supporting the idea that a single-
player role-playing video game could improve students’ attitudes
for a wide range of ethical topics [
1
,
21
,
51
], leading us to predict
that Academical would be eective for improving attitudes about
RCR specically.
2.3 Academical is Eective for Teaching RCR
Learning Outcomes
In a prior study, we hypothesized that the choice-based, role-playing
nature of interactive storytelling games could be employed to im-
prove student engagement as well as knowledge and moral reason-
ing skills important for RCR. This prediction inspired the devel-
opment of Academical, a choice-based interactive narrative video
game (accessible from any web browser) that presents a set of nine
stories addressing the nine core RCR topics identied by the NIH—
see Figure 1 for a story sample and [
38
,
39
] for a more in-depth
description of the game design and rationale. In this single-player
game, learners role-play through both perspectives in an ethical
dilemma, challenging them to make a series of dicult choices in
order to reach an optimal solution. We initially conducted a random-
ized group comparison study showing that Academical is overall
more eective than traditional web-based RCR training tools at en-
gaging students and teaching them knowledge [
39
] as well as skills
important for RCR such as moral reasoning [
38
]. Our successful
results, combined with existing evidence that role-playing activities
are useful for improving socio-aective learning outcomes, indicate
that Academical would likely also be able to train improvements in
players’ attitudes about RCR.
2.4 Engagement and Attitudes
Training can become more eective through gamication, which
motivates the student to be more engaged with the learning ac-
tivity [
15
,
57
]. Engagement, which involves sustained attentional
eort, is key for improving long-term learning outcomes, espe-
cially aective outcomes like attitudes [
35
,
49
]. Presently, much
of the serious games research for academic achievement focuses
on math [
7
,
19
,
30
], health [
6
,
24
,
34
,
61
], and problem solving
skills [
12
,
17
,
25
,
55
]. These studies exemplify that gamifying train-
ing can increase students’ engagement with the material and their
motivation to learn and perform. Narrative has become an especially
eective method for improving engagement and deep learning [
48
].
Contrasted with the passive reading study strategy promoted by
the majority of existing web training tools, Academical utilizes nar-
rative role-play and interactive choices to challenge the player to
successfully navigate various moral dilemmas common to scientic
research.
Our previous work showed that playing Academical was signi-
cantly more engaging and generally resulted in better RCR learning
outcomes compared to traditional web-based training [
38
]. While
the exact cause for players’ motivations or capability to perform
are unknown, the game demonstrates a clear engagement advan-
tage which may contribute to learning outcome achievement. Our
prior engagement results, combined with a eld of work showing
the relationship between engagement and socio-aective learning
outcomes, led us to hypothesize that engagement while playing Aca-
demical would predict participants’ post-game attitudes. Finding
a relationship between players’ engagement and their post-game
attitudes towards research ethics would provide further evidence
supporting the game’s pedagogical ecacy and provide further
support for the connection between engagement and achievement
in game-based learning (GBL).
3 METHODS
For this study, we hypothesized that 1) a choice-based interactive
narrative game (i.e., Academical) would improve players’ attitudes
towards RCR and 2) players’ reported engagement playing the game
would predict their post-game attitudes. In order to explore these
hypotheses, we conducted a quasi-experimental within-subjects
study comparing participants’ RCR attitudes before and after play-
ing a single short session of Academical as well as comparing those
attitudes with their feelings of engagement with the game.
3.1 Procedure
This study was approved by the Institutional Review Board of
the University of California, Santa Cruz (UCSC). The study was
conducted entirely online and consisted of three major components.
One group of study participants was required to 1) complete a pre-
game survey assessing demographics and attitudes about RCR, 2)
play the Academical game, and 3) complete a post-game survey
gauging their attitudes about RCR and their feelings of engagement
with the game.
3.2 Participant Recruitment
All participants were recruited from an undergraduate course of-
fered through the engineering department at UCSC (a Tier 1 re-
search university). Two weeks before the conclusion of the course,
participants were informed of the study through email and oered
extra credit toward their class grade in exchange for completing all
parts of the study. Participants were told that the purpose of the
study was to test the ecacy of a new RCR training program.
A total of 99 undergraduate students registered for this study
through a participant recruitment website hosted by the authors’
university. Within this pool, 69 successfully participated by com-
pleting all parts of the study. Nine of these participants reported
that they had received prior RCR training and were therefore ex-
cluded from analysis. Of the 60 remaining participants, there were
41 males, 16 females and 3 non-binary. The average participant age
was 20.6
±
2.2 years (median: 20, range: 18-29), which is a typical
age for university students who are starting to engage in research
and consider applying to graduate school.

CHI ’21 Extended Abstracts, May 8–13, 2021, Yokohama, Japan Katelyn M. Grasse, Edward F. Melcer, Max Kreminski, Nick Junius, and Noah Wardrip-Fruin
Table 1: Participants’ attitudes before and after playing a single short session of Academical. Responses were measured using
a 7-point Likert scale. The table contains mean scores, standard deviations, paired t-test scores (two-sided Wilcoxon sign rank
test using 95% CI), and eect size—which is small to medium for signicant dierences. Items borrowed from [28] are indicated
with †.
Pre Post Sig ES
Aitude Survey Items 𝜇 𝜎 𝜇 𝜎 p d
(1) How important is RCR training to you? 4.2 1.5 5.0 1.5 <0.001 0.49
(2) How important do you think RCR training should be for researchers? 5.7 1.2 6.3 1.0 <0.001 0.57
(3) Research ethics is serious and deserving of the attention of all researchers. †6.1 1.0 6.5 0.9 0.01 0.35
(4) Researchers have a personal responsibility to model and promote RCR. †5.5 1.1 6.1 1.0 <0.001 0.51
(5) Researchers have a responsibility to society. †5.6 1.2 6.0 1.1 <0.001 0.37
(6) Excellence in research includes RCR. †5.6 1.2 6.2 1.0 <0.001 0.55
Overall Attitude Score 5.3 0.9 5.9 0.9 <0.001 0.65
Participants reported pursuing the following undergraduate de-
grees: Arts and Design: Games and Playable Media (n = 25), Com-
puter Science: Computer Game Design (13), Cognitive Science (9),
Computer Science (7), Technology and Information Management
(4), Economics (1), Film and Digital Media (1), Physics (1), and Soci-
ology (1). Two participants reported that they had not yet declared
a major, while four reported pursuing two majors. Design-based
(26) and engineering (24) degrees were the most common, followed
by science (11) and economics (1).
3.3 Academical Gameplay
Participants accessed the surveys and game using the same methods
as the previous Academical study [
38
,
39
]—through their preferred
web browser on their personal computers and without any su-
pervision beyond automated data collection. Similarly, the same
two (of the nine possible) scenarios were selected for students
to play through (i.e., peer review and authorship). Participants
were instructed to play through each character at least once in
each scenario—equating a minimum of 4 total playthroughs (2 per
module)—before completing the post-survey.
3.4 Assessment Tools
3.4.1 RCR Aitudes Survey. To assess Academical’s ecacy for
improving attitudes about RCR, we created a short survey using a
list of attitude goals that are highly recommended by RCR instruc-
tors [
28
]. This survey included six items (two questions and four
statements, see Table 1) with possible responses along a 7-point
Likert scale indicating level of agreement. Likert response options
either ranged from “Not at all important” to “Extremely important”
or from “Strongly disagree” to “Strongly agree”. To assess within-
subject changes in these attitudes, participants completed the same
attitude survey before and after playing the game.
3.4.2 Temple Presence Inventory, Engagement Subscale. Engage-
ment is a critical aspect of the learning process [
31
], drastically
inuencing a learner’s motivation to continue interacting with a
system and the educational content [
42
]. To assess participant en-
gagement with the Academical game, we utilized the Engagement
subscale (Cronbach’s alpha = 0.90) of the Temple Presence Inven-
tory (TPI) [
32
]. The TPI is an instrument that has been validated
for use with games [33] and measuring game engagement [36].
3.5 Statistics
All data was evaluated using Matlab and is presented in the text
as mean
±
standard deviation (SD). Within-subject comparisons
were conducted using non-parametric two-sided Wilcoxon sign
rank tests with a 95% condence interval, and Cohen’s d eect size
strengths are described according to [
13
]. Because the data was
ordinal, all inter-variable relationships were evaluated using non-
parametric Spearman correlation methods (which is robust against
outliers for large N), with correlation strengths described according
to [
14
]. Therefore, the correlation coecients reported in this paper
indicate strength of a monotonically increasing relationship rather
than linearity.
4 RESULTS
4.1 RCR Attitudes
In order to gauge whether playing Academical could improve partic-
ipants’ attitudes about RCR, we conducted within-subject compar-
isons of pre- and post-game attitude ratings. A series of Wilcoxon
sign rank tests revealed that, after playing Academical, participants
on average reported a signicant improvement in agreement with
every individual item in the attitudes survey (see Table 1; all p<0.01;
eect size range of d = 0.35-0.57, which are small to medium). The
average participant initially did not feel that RCR was personally
important—the most common pre-game attitude score for item 1
was a neutral score of 4 (n = 25, 42%), with scores ranging from
1-7. After playing the game, only 13 participants (22%) reported a
neutral score for this rst item, with the most common post-game
score being 6 (n = 20, 33%). These results for item 1 indicated that
the participants felt that RCR was much more personally important
after playing the game. Conversely, participants generally initially
agreed with the importance of RCR for researchers—for items 2-6,
most pre-game scores were positive (>4). Even though these scores
were relatively high to begin with (compared to item 1), they still
signicantly improved after gameplay. The results for items 2-6
indicated that the participants also felt that RCR was much more im-
portant for researchers after playing the game. For each participant,
we averaged the six attitude scores to nd an overall attitude score
for both test-points (pre: 5.3
±
0.9; post: 5.9
±
0.9; change: 0.55
±
0.7).
Wilcoxon sign rank analysis also showed that participants’ overall

Improving Undergraduate Aitudes Towards RCR Through an Interactive Storytelling Game CHI ’21 Extended Abstracts, May 8–13, 2021, Yokohama, Japan
1234567
Pre Attitude
1
2
3
4
5
6
7
Engagement
rs = 0.158 p = 0.229
1234567
Post Attitude
1
2
3
4
5
6
7
rs = 0.406 p = 0.001
-3 -2 -1 0 1 2 3
Change in Attitude
1
2
3
4
5
6
7
rs = 0.273 p = 0.035
Figure 2: Participants’ feelings of engagement while playing Academical were signicantly correlated with post-game attitudes
and a change in attitude, but not with pre-game attitudes. For simpler visual comparison with the overall attitude scores,
engagement scores are reported here as the average (rather than the sum) of the six survey items. Non-parametric Spearman
correlation coecients are provided above each result.
attitude score increased signicantly after playing the game (Rank
sign test: r = 0.31, p < 0.001; eect size d = 0.65, which is medium).
Overall, these results conrmed our rst hypothesis and demon-
strate that playing a short session of Academical can signicantly
improve a variety of important attitudes about RCR.
4.2 Engagement Correlations with RCR
Attitudes
After playing the game, participants completed the Engagement
subscale of the TPI Survey—a set of six questions validated for
assessing feelings of engagement with an experience (i.e., playing
a game). Participants on average reported an engagement score
of 26.9
±
6.6 out of a possible 42 points (median: 28; range: 7-42).
Over half the participants (n = 38, 63%) reported feeling at least
moderately engaged (score >24, i.e., average survey item score >4).
These results show that individual participants in this study varied
greatly in their feelings of engagement while playing the game.
We expected that participants’ reported engagement with the
game would predict their post-game attitudes about RCR. First,
Spearman correlations revealed that engagement did not predict
participants’ pre-game attitude scores (r
s
= 0.16, p = 0.23). In con-
trast, we found that engagement was signicantly correlated with
post-game attitudes (r
s
= 0.41, p = 0.001, moderate strength). En-
gagement was also correlated with participants’ change in attitude
(r
s
= 0.27, p = 0.04, weak strength). Together, these results indicate
that after playing Academical, participants changed their RCR atti-
tudes to more closely align with their feelings of engagement with
the game.
5 DISCUSSION
Our work evaluating Academical highlights the potential of choice-
based interactive storytelling games for teaching the full breadth of
distinct learning outcomes essential to RCR education. Specically,
prior research on interactive narrative games has demonstrated
their ecacy in both engaging students and teaching conceptual
knowledge [
39
] as well as moral reasoning skills [
38
]. This paper
adds an important piece to further understand the potential of uti-
lizing interactive narrative games to teach ethically complex topics
by expanding upon these existing ndings to illustrate that such
games can also impact attitudes (paired t-test: r = 0.31, p<0.001,
d = 0.65). Importantly, this collection of evidence—existing litera-
ture [
38
,
39
] combined with the ndings presented here—indicates
that the choice-based interactive storytelling design of interactive
narrative video games (such as Academical) can successfully train
both cognitive and socio-aective learning outcomes. To the best
of our knowledge, this is an achievement which has not yet been
documented for existing web-based RCR pedagogy [
45
,
53
]. It also
serves to expand upon existing literature, suggesting that interac-
tive narrative can eectively inuence attitudes similar to what has
been shown with traditional narrative [16, 57].
5.1 Implications for Design
5.1.1 The Caveats of Using Interactive Role-Play for Training At-
titudes. Role-play provides an engaging opportunity to embody
contending perspectives on an issue, making it one of the most
promising discussion methods for improving comprehension and
execution of ethical behavior [
11
]. While role-play has been shown
to eectively train learning outcomes that drive improvements
in behavior for topics involving ethical issues [
46
,
52
,
54
,
58
,
62
],
role-play activities are not always eective for improving atti-
tudes [
47
,
60
]. Specically, paired live-action role-play can leave
students feeling awkward or distracted, aording opportunity for
the exercise to be less valuable than more controlled forms of dis-
cussion learning (e.g., case studies) [
53
]. Our results highlight the

CHI ’21 Extended Abstracts, May 8–13, 2021, Yokohama, Japan Katelyn M. Grasse, Edward F. Melcer, Max Kreminski, Nick Junius, and Noah Wardrip-Fruin
potential of utilizing choice-based interactive narrative games to
deliver a self-paced, single-player activity that facilitates a compar-
atively more controlled form of ethical role-play. Presenting the
role-play exercise using interactive narrative eliminates the possibil-
ity of an uncomfortable or unproductive interpersonal interaction,
which should help reduce the chance that the player will develop
negative feelings in association with the subject matter. Further-
more, players can take as much time as needed to fully understand
the stories’ dilemmas and various possible outcomes, which should
assist learning. However, without direct engagement from a peer or
educator, the single-player nature of an interactive narrative game
also aords players the opportunity to avoid genuinely executing
the learning exercise (e.g., advancing through a story without read-
ing it), which is a known issue for existing online RCR training
modules. Therefore, it is important for the designers of interactive
narrative games to develop story content that facilitates players’
genuine engagement (and subsequent learning) with the game.
5.1.2 The Importance of Engagement within Interactive Narrative
Games. GBL research has demonstrated that engagement can in-
uence a student’s motivation to learn [
12
,
41
] and can especially
impact aective learning outcomes like attitudes [
21
,
35
]. Com-
pared to passive reading, the interactive narrative format requires
a player to engage with the content in order to make choices that
direct the story toward a particular conclusion—which is an en-
joyable exercise for many people. However, care should be taken
to generalize Academical’s success across the interactive narrative
game genre. Our results highlight the importance of ensuring that
an interactive narrative is engaging for the player, as post-game atti-
tudes (r
s
= 0.41, p = 0.001) and changes in attitudes from pre to post
(r
s
= 0.27, p = 0.04) were signicantly correlated with participants’
engagement. This indicates that after playing Academical, partici-
pants changed their RCR attitudes to more closely align with their
feelings of engagement with the game, suggesting that a more en-
gaging game experience was more likely to result in more positive
feelings about RCR (and vice versa).
Clearly, merely using the interactive narrative medium does not
guarantee that a story will feel immersive or engaging for all (or
even any) readers. For instance, the results of the present study
indicate that some participants did not feel particularly engaged
with the game. This is a problem considering that engagement is
a critical component helping to drive both cognitive and socio-
aective learning outcomes. Therefore, it is crucial for designers of
interactive narrative games to consider how aspects of their design
impact engagement and employ various techniques to improve
it. For instance, the relatability or lack thereof for their content
(either characters or story) can cause players to disengage from
the narrative [
21
], so utilizing a demographically diverse cast of
characters or enabling the player to personalize their character
images for the narrative could improve relatability and subsequently
engagement. However, it is also important to consider that some
people are simply less willing to exert mental eort, suggesting
that they may not experience similar benets from the increased
engagement encouraged by the interactive narrative medium [
22
].
5.2 Study Limitations
One key disadvantage of this study is that it did not utilize a vali-
dated metric to gauge RCR attitudes, and so the relevance of the
measured attitudes for informing/predicting ethical behavior is un-
known. However, most of these survey items were taken from [
28
]
and are therefore quite likely to be relevant to RCR education. Pre-
post data from a control group (either untrained or trained by a
traditional web-based RCR course) would help to further contextual-
ize both major results of this study. Other useful control data could
include distractor questions (i.e., pre-post assessment of attitudes
unrelated to the training material).
6 CONCLUSION AND FUTURE WORK
This paper extends the breadth of evidence supporting the ecacy
of interactive choice-based narrative games for training ethics—
and more specically scientists about how to conduct research
responsibly. The results of this study support the hypothesis that
playing a single brief session of Academical, a choice-based inter-
active narrative game, can signicantly improve players’ attitudes
about RCR, thus extending the range of evidence illustrating the
game’s pedagogical ecacy for teaching every key RCR learning
outcome (i.e., knowledge, moral reasoning skills and attitudes). The
study also demonstrates a moderately signicant relationship be-
tween engagement playing the game and post-game RCR attitudes,
indicating that Academical’s advantage for engaging players is pre-
dictive of subsequent achievement for that socio-aective learning
outcome. However, this relationship warrants further study to bet-
ter understand its impact. These results further elucidate the value
of a choice-based interactive storytelling game, such as Academical,
for teaching RCR and provide implications for the use of interac-
tive storytelling games to improve learning outcomes of ethically
complex content such as RCR.
Future work with a longitudinal study should be conducted to
examine long-term retention of learning outcomes as well as the
eect of more training (e.g., playing through all nine scenarios
or supplementing other existing RCR courses). Additionally, prior
work has indicated that experienced researchers may have well-
formed RCR attitudes and therefore be less likely to feel measurably
dierent in response to training [
44
]. In order to determine the
generalizability of the current results for all researchers, future work
should investigate whether more senior scientists also experience
similar attitude boosting eects after playing Academical. Future
studies of Academical will also continue to explore more aspects
of play that may explain the game’s ecacy, particularly character
relatability [
9
,
21
]. Finally, it will also be important to understand
whether being in a research community that fosters strong RCR
values (e.g., through mentorship) can impact the game’s ecacy [
2
,
18, 27].
REFERENCES
[1]
Saleem Elias Alhabash and Kevin Wise. 2012. PeaceMaker: Changing students’
attitudes toward Palestinians and Israelis through video game play. International
Journal of Communication 6 (2012), 25.
[2]
Melissa S Anderson, Aaron S Horn, Kelly R Risbey, Emily A Ronning, Raymond
De Vries, and Brian C Martinson. 2007. What do mentoring and training in the
responsible conduct of research have to do with scientists’ misbehavior? Findings
from a national survey of NIH-funded scientists. Academic Medicine 82, 9 (2007),
853–860.

Improving Undergraduate Aitudes Towards RCR Through an Interactive Storytelling Game CHI ’21 Extended Abstracts, May 8–13, 2021, Yokohama, Japan
[3]
Alison L Antes and James M DuBois. 2014. Aligning objectives and assessment
in responsible conduct of research instruction. Journal of Microbiology & Biology
Education 15, 2 (2014), 108.
[4]
Alison L Antes, Stephen T Murphy, Ethan P Waples, Michael D Mumford, Ryan P
Brown, Shane Connelly, and Lynn D Devenport. 2009. A meta-analysis of ethics
instruction eectiveness in the sciences. Ethics & Behavior 19, 5 (2009), 379–402.
[5]
Alison L Antes, Xiaoqian Wang, Michael D Mumford, Ryan P Brown, Shane
Connelly, and Lynn D Devenport. 2010. Evaluating the eects that existing
instruction on responsible conduct of research has on ethical decision making.
Academic medicine: journal of the Association of American Medical Colleges 85, 3
(2010), 519.
[6]
Sarah A Aynsely, Kusum Nathawat, and Ruseell M Crawford. 2018. Evaluating
student perceptions of using a game-based approach to aid learning: Braincept.
Higher Education Pedagogies 3 (2018), 478–489. Issue 1.
[7]
Haiyan Bai, Wei Pan, Astusi Hirumi, and MansurehKebritchi. 2012. Assessing the
eectiveness of a 3-D instructional game on improving mathematics achievement
and motivation of middle school students. British Journal of Educational Technol-
ogy 43, 6 (2012), 993–1003. https://doi.org/10.1111/j.1467-8535.2011.01269.x
[8]
Muriel J Bebeau. 1993. Designing an Outcome-based Ethics Curriculum for
Professional Education: strategies and evidence of eectiveness. Journal of moral
education 22, 3 (1993), 313–326.
[9]
Max V Birk, Cheralyn Atkins, Jason T Bowey, and Regan L Mandryk. 2016.
Fostering intrinsic motivation through avatar identication in digital games. In
Proceedings of the 2016 CHI conference on human factors in computing systems.
2982–2995.
[10]
Kurt Braddock and James Price Dillard. 2016. Meta-analytic evidence for the
persuasive eect of narratives on beliefs, attitudes, intentions, and behaviors.
Communication Monographs 83, 4 (2016), 446–467.
[11]
Bradley J Brummel, CK Gunsalus, Kerri L Anderson, and Michael C Loui. 2010.
Development of role-play scenarios for teaching responsible conduct of research.
Science and Engineering Ethics 16, 3 (2010), 573–589.
[12]
Douglas B Clark, Emily E Tanner-Smith, and Stephen S Killingsworth. 2016.
Digital games, design, and learning: A systematic review and meta-analysis.
Review of educational research 86, 1 (2016), 79–122.
[13]
Jacob Cohen. 2013. Statistical power analysis for the behavioral sciences. Academic
press.
[14]
Christine P Dancey and John Reidy. 2007. Statistics without maths for psychology.
Pearson education.
[15]
Sara De Freitas. 2006. Learning in immersive worlds: A review of game-based
learning. (2006).
[16]
Jennifer Edson Escalas. 2004. Imagine yourself in the product: Mental simulation,
narrative transportation, and persuasion. Journal of advertising 33, 2 (2004),
37–48.
[17]
Deniz Eseryel, Victor Law, Dirk Ifenthaler, Xun Gel, and Raymond Miller. 2014.
An Investigation of the Interrelationships between Motivation, Engagement,
and Complex Problem Solving in Game-based Learning. Journal of Educational
Technology Society 17 (2014), 42–53. Issue 1.
[18]
Celia B Fisher, Adam L Fried, and Lindsay G Feldman. 2009. Graduate socialization
in the responsible conduct of research: A national survey on the research ethics
training experiences of psychology doctoral students. Ethics & Behavior 19, 6
(2009), 496–518.
[19]
Varvara Garneli, Michail Giannakos, and Konstantinos Chorianopoulos. 2017.
Serious games as a malleable learning medium: The eects of narrative, gameplay,
and making on students’ performance and attitudes. British Journal of Educational
Technology 48, 3 (2017), 842–859. https://doi.org/10.1111/bjet.12455
[20]
Andreas Gegenfurtner, Koen Veermans, Dagmar Festner, and Hans Gruber. 2009.
Integrative literature review: Motivation to transfer training: An integrative
literature review. Human resource development review 8, 3 (2009), 403–423.
[21]
Melanie C Green and Keenan M Jenkins. 2014. Interactive narratives: Processes
and outcomes in user-directed stories. Journal of Communication 64, 3 (2014),
479–500.
[22]
Melanie C Green and Keenan M Jenkins. 2020. Need for Cognition, Transporta-
bility, and Engagement with Interactive Narratives. Games for health journal 9, 3
(2020), 182–186.
[23]
Matthew Grizzard, Ron Tamborini, Robert J Lewis, Lu Wang, and Sujay Prabhu.
2014. Being bad in a video game can make us morally sensitive. Cyberpsychology,
Behavior, and Social Networking 17, 8 (2014), 499–504.
[24]
Hussein Haruna, Xiao Hu, Samuel Kai Wah Chu, Robin R Mellecker, Goodluck
Gabriel, and Patrick Siril Ndekao. 2018. Improving Sexual Health Education
Programs for Adolescent Students through Game-Based Learning and Gamica-
tion. International Journal of Environmental Research and Public Health 15 (2018).
https://doi.org/10.3390/ijerph15092027
[25]
Gwo-Jen Hwang, Po-Han Wu, and Chi-Chang Chen. 2012. An online game
approach for improving students’learning performance in web-based problem-
solving activities. Journal of Educational Computing Research 59 (2012), 1246–
1256.
[26]
Michael Kalichman. 2013. A Brief History of RCR Education. Account Res 20, 5-6
(Jan. 2013), 380–394. https://doi.org/10.1080/08989621.2013.822260
[27]
Michael Kalichman. 2014. Rescuing responsible conduct of research (RCR) edu-
cation. Accountability in research 21, 1 (2014), 68–83.
[28]
Michael W Kalichman and Dena K Plemmons. 2007. Reported goals for responsi-
ble conduct of research courses. Academic Medicine 82, 9 (2007), 846–852.
[29]
Michael W Kalichman and Dena K Plemmons. 2015. Research agenda: The eects
of responsible-conduct-of-research training on attitudes. Journal of Empirical
Research on Human Research Ethics 10, 5 (2015), 457–459.
[30]
Fengfeng Ke. 2008. Computer games application within alternative classroom
goal structures: cognitive, metacognitive,and aective evaluation. Educational
Technology Research and Development 56 (2008), 539–556. https://doi.org/10.
1007/s11423-008- 9086-5
[31]
Greg Kearsley and Ben Shneiderman. 1998. Engagement theory: A framework
for technology-based teaching and learning. Educational technology 38, 5 (1998),
20–23.
[32]
Matthew Lombard, Theresa B Ditton, and Lisa Weinstein. 2009. Measuring
presence: the temple presence inventory. In Proceedings of the 12th annual inter-
national workshop on presence. 1–15.
[33]
Matthew Lombard, Lisa Weinstein, and Theresa Ditton. 2011. Measuring telep-
resence: The validity of the Temple Presence Inventory (TPI) in a gaming context.
In ISPR 2011: The International Society for Presence Research Annual Conference.
Edinburgh.
[34]
Rocio Lorenzo-Alvarez, Teodoro Rudolphi-Solero, Miguel J Ruiz-Gomez, and
Francisco Sendra-Portero. 2020. Game-based Learning in Virtual Worlds: A
Multiuser Online Game for Medical Undergraduate Radiology Education within
Second Life. Anatomical Sciences Education 13 (2020), 602–617.
[35]
Mia Liza A Lustria. 2007. Can interactivity make a dierence? Eects of inter-
activity on the comprehension of and attitudes toward online health content.
Journal of the American Society for Information Science and Technology 58, 6 (2007),
766–776.
[36]
Rosa Mikeal Martey, Kate Kenski, James Folkestad, Laurie Feldman, Elana Gordis,
Adrienne Shaw, Jennifer Stromer-Galley, Ben Clegg, Hui Zhang, Nissim Kauf-
man, et al
.
2014. Measuring game engagement: multiple methods and construct
complexity. Simulation & Gaming 45, 4-5 (2014), 528–547.
[37]
Jennifer McCaerty, Reid Cushman, Kenneth W Goodman, Paul Braunschweiger,
and Robin N Fiore. 2012. New NSF And NIH Responsible Conduct of Research
(RCR) Guidelines: A Three-Phase Plan. Teaching Ethics 12, 2 (2012), 23–30.
[38]
Edward F Melcer, Katelyn M Grasse, James Ryan, Nick Junius, Max Kreminski,
Dietrich Squinkifer, Brent Hill, and Noah Wardrip-Fruin. 2020. Getting Academ-
ical: A Choice-Based Interactive Storytelling Game for Teaching Responsible
Conduct of Research. In Proceedings of the 15th International Conference on the
Foundations of Digital Games-FDG.
[39]
Edward F Melcer, James Ryan, Nick Junius, Max Kreminski, Dietrich Squinkifer,
Brent Hill, and Noah Wardrip-Fruin. 2020. Teaching Responsible Conduct of
Research Through an Interactive Storytelling Game. In Extended Abstracts of the
2020 CHI Conference on Human Factors in Computing Systems. 1–10.
[40]
Engineering National Academies of Sciences, Medicine, et al
.
2017. Fostering
integrity in research. National Academies Press.
[41]
Raymond A Noe. 1986. Trainees’ attributes and attitudes: Neglected inuences
on training eectiveness. Academy of management review 11, 4 (1986), 736–749.
[42]
Heather L O’Brien and Elaine G Toms. 2008. What is user engagement? A
conceptual framework for dening user engagement with technology. Journal of
the American society for Information Science and Technology 59, 6 (2008), 938–955.
[43]
Scott Parrott, Francesca R Dillman Carpentier, and C Temple Northup. 2017.
A test of interactive narrative as a tool against prejudice. Howard Journal of
Communications 28, 4 (2017), 374–389.
[44]
Dena K Plemmons, Suzanne A Brody, and Michael W Kalichman. 2006. Stu-
dent perceptions of the eectiveness of education in the responsible conduct of
research. Science and engineering ethics 12, 3 (2006), 571–582.
[45]
Sean T Powell, Matthew A Allison, and Michael W Kalichman. 2007. Eectiveness
of a responsible conduct of research course: A preliminary study. Science and
engineering ethics 13, 2 (2007), 249–264.
[46]
Deepa Rao and Ieva Stupans. 2012. Exploring the potential of role play in higher
education: development of a typology and teacher guidelines. Innovations in
Education and Teaching International 49, 4 (2012), 427–436.
[47]
Lesley M Roberts, Connie Wiskin, and Andrea Roalfe. 2008. Eects of exposure to
mental illness in role-play on undergraduate student attitudes. Family medicine
40, 7 (2008), 477.
[48]
Jonathan P Rowe, Lucy R Shores, Bradford W Mott, and James C Lester. 2010. In-
tegrating learning and engagement in narrative-centered learning environments.
In International Conference on Intelligent Tutoring Systems. Springer, 166–177.
[49]
Jennifer L Sabourin and James C Lester. 2013. Aect and engagement in Game-
BasedLearning environments. IEEE transactions on aective computing 5, 1 (2013),
45–56.
[50]
Karen B Schmaling and Arthur W Blume. 2009. Ethics instruction increases
graduate students’ responsible conduct of research knowledge but not moral
reasoning. Accountability in Research 16, 5 (2009), 268–283.
[51]
Karen Schrier.2015. EPIC: A framework for using video games in ethics education.
Journal of Moral Education 44, 4 (2015), 393–424.

CHI ’21 Extended Abstracts, May 8–13, 2021, Yokohama, Japan Katelyn M. Grasse, Edward F. Melcer, Max Kreminski, Nick Junius, and Noah Wardrip-Fruin
[52]
Karen Schrier. 2015. Ethical thinking and sustainability in role-play participants:
A preliminary study. Simulation & Gaming 46, 6 (2015), 673–696.
[53]
Stephanie N Seiler, Bradley J Brummel, Kerri L Anderson, Kyoung Jin Kim, Serena
Wee, CK Gunsalus, and Michael C Loui. 2011. Outcomes assessment of role-play
scenarios for teaching responsible conduct of research. Accountability in Research
18, 4 (2011), 217–246.
[54]
Eric P Skye, Heather Wagenschutz, Jerey A Steiger, and Arno K Kumagai. 2014.
Use of interactive theater and role play to develop medical students’ skills in
breaking bad news. Journal of Cancer Education 29, 4 (2014), 704–708.
[55]
Hiller A Spires, Jonathan P Rowe, Bradford W Mott, and James C Lester. 2011.
Problem Solving and Game-based Learning: Eects of Middle Grade Students’
Hypothesis Testing Strategies on Learning Outcomes. Journal of Educational
Computing Research 44 (2011), 453–472. Issue 4.
[56]
Sharon T Steinemann, Glena H Iten, Klaus Opwis, Seamus F Forde, Lars Frasseck,
and Elisa D Mekler. 2017. Interactive narratives aecting social change. Journal
of Media Psychology (2017).
[57]
Sujit Subhash and Elizabeth A Cudney. 2018. Gamied learning in higher edu-
cation: A systematic review of the literature. Computers in Human Behavior 87
(2018), 192–206.
[58]
Ross Taplin, Abhijeet Singh, Rosemary Kerr,and Alina Le e. 2018. The use of short
role-plays for an ethics intervention in university auditing courses. Accounting
Education 27, 4 (2018), 383–402.
[59]
L Vuckovic-Dekic, D Gavrilovic, I Kezic, G Bogdanovic, and S Brkic. 2012. Science
ethics education part II: changes in attitude toward scientic fraud among medical
researchers after a short course in science ethics. J BUON 17, 2 (2012), 391–5.
[60]
Connie Wiskin, Lesley Roberts, and Andrea Roalfe. 2011. The impact of discussing
a sexual history in role-play simulation teaching on pre-clinical student attitudes
towards people who submit for STI testing. Medical Teacher 33, 6 (2011), e324–
e332.
[61]
Jui-Mei Yien, Chun-Ming Hung, Gwo-Jen Hwang, and Yueh-Chiao Lin. 2011. A
Game-based Learning Approach to Improving Students’ Learning Achievements
in A Nutrition Course. The Turkish Online Journal of Educational Technology 10
(2011). Issue 2.
[62]
Amitai Ziv, Paul Root Wolpe, Stephen D Small, and Shimon Glick. 2003.
Simulation-based medical education: an ethical imperative. Academic medicine
78, 8 (2003), 783–788.
A APPENDIX
A.1 TPI Engagement Subscale Survey
(1)
To what extent did you feel mentally immersed in the expe-
rience?
(2) How involving was the experience?
(3) How completely were your senses engaged?
(4) To what extent did you experience a sensation of reality?
(5) How relaxing or exciting was the experience?
(6) How engaging was the story/material?