Content uploaded by Drew Crecente
Author content
All content in this area was uploaded by Drew Crecente on Aug 13, 2017
Content may be subject to copyright.
1
Gaming Against Violence: An Exploratory Evaluation through Mechanical Turk of the
Efficacy of Persuasive Digital Games in Improving Unhealthy Relationship Attitudes
Ruud S. Jacobs, MSc.1 and Drew Crecente, JD2
Abstract
Objective: The current article explores the general efficacy of digital games that aim to combat teen
dating violence by looking at the effect they have on players‟ attitudes towards this topic.
Materials and Methods: An online experiment was performed on 86 workers of Amazon‟s
Mechanical Turk. Participants either played one of five teen dating violence (TDV) games or one of
two control games unrelated to the topic. The TDV games were all prior winners in the annual
Life.Love. Game Design Challenge hosted by Jennifer Ann‟s Group, a charity organization focusing
on reducing dating violence. Attitudes were measured through previously validated dating violence
scales with a pre- and post-test to show the degree to which players change their views.
Results: Players of the TDV games showed greater increase on attitudes towards dating violence than
players of unrelated games, with small to medium effect sizes on different comparisons.
Conclusion: The results of this exploratory study are promising, showing that overall, the TDV
games have an effect on their players. The study is limited in the sample that was drawn and that its
sample was not large enough to distinguish differential efficacy among the TDV games, although it
lays the groundwork for future studies to further validate the viability of these games as persuasive
tools.
1 Erasmus Research Centre for Media, Communication, and Culture, Erasmus University Rotterdam, The
Netherlands
2 Jennifer Ann‟s Group, Atlanta, Georgia
GAMING AGAINST VIOLENCE: AN EXPLORATORY EVALUATION 2
Introduction
The issue of teen dating violence is widespread. An estimated 40 percent of graduating college
students have been in an abusive relationship at least once in the United States.1 Despite its ubiquity,
16 million (of a population of approximately 21 million) students in United States schools are being
educated in states that lack teen dating violence (TDV) education legislation.2 It is therefore necessary
to approach this issue in a way that does not rely on institutional education practices. Digital games
are an ideal and novel method to confront the problem of TDV. Jennifer Ann‟s Group, a 501(c)(3)
nonprofit charity, produces video games through the Live.Love. Game Design Challenge, an annual
videogame competition it has run since 2008 that encourages developers to create games about TDV.3
Jennifer Ann‟s Group (JAG) hosts many of the games that were awarded prizes in this challenge on
its site,4 allowing interested parties to experience this fraught topic (for free) in a safe way.5
The current article explores the influences teen dating violence games have on players. An
exploratory online experiment was performed looking at attitude change in players of several JAG
games and compared with games unrelated to TDV. After defining the issue of teen dating violence
and positing that games about this topic can help to combat it, we discuss the study‟s methods and
results.
The Problem: Teen Dating Violence
Teen dating violence is defined as any kind of physical, sexual, or emotional violence (including
stalking behavior) by the Centers for Disease Control.6 A 2013 estimate concluded that around 21
percent of high-school aged dating females had experienced a form of TDV in the past year, with a
lower percentage (10%) of men being victimized.7 TDV is a complex, nuanced problem. Although it
shares many of the same traits as „traditional‟ domestic violence, it is further complicated by lack of
awareness, insufficient policy focus, and the relative naiveté of those most affected.
Essentially, healthy relationship behavior is learned over time. If adolescents do not witness
healthy relationship modeling within their family they resort to learning those behaviors through their
GAMING AGAINST VIOLENCE: AN EXPLORATORY EVALUATION 3
peers8 and via media. As a result, perpetrators and targets of abuse are less likely than adults to
identify and engage in healthy relationship behaviors.
Efforts to curb teen dating violence are almost invariably rooted in educational practice.
However, in a comprehensive meta-analysis of current literature on the evidence behind these
programs, De La Rue et al.9 noted that while they do affect attitudes and knowledge, their influence
on subsequent behavior (victimization and perpetration of dating violence) cannot be proven. Apart
from being due to a lack of evidence rather than a lack of effects, this finding should encourage
creators to tackle this issue in novel ways.
Videogames Designed to Prevent Dating Violence
Approaching dating violence prevention through a video game allows creators to present their
message in a wholly unique way. Although video games are still often seen as mere entertainment,
their ability to effect both generally positive changes in players10 as well as more specific changes in
attitudes towards social issues11 is being increasingly recognized. By offering simulated
environments, TDV games can promote experiential learning in players, letting them safely but
interactively experience the issue from different angles.5,12 This flexibility lets players reach their own
conclusions about the critical aspects of TDV.13 They also learn about this topic in the context of
biographical or fictional stories, rather than in an abstract way.14 All these facets could be said to give
game-based learning an advantage over traditional methods of instruction.
Using games to discuss this topic also allows creators to come closer to the target audience of
students aged between 11 and 22: Teens prefer games over other media,5,15 and see the use of
computers in instruction as a more positive experience than other methods.16 In some cases teens see
video games as appropriate tools for their age group, appreciating that their preference was considered
when selecting an approach for them.
Apart from their viability as a tool for change and their fit with the audience that would
benefit most from efforts to curb TDV, games are also quite cost-effective as teaching tools. Digital
products are much easier to scale up or tailor to specific circumstances than physical resources, which
GAMING AGAINST VIOLENCE: AN EXPLORATORY EVALUATION 4
are constrained by manufacturing and production processes. Newer technologies also allow these
games to be played from teens‟ own smartphones or tablets, presenting more intimate experiences
while not having to rely on special devices, infrastructure, or legislation.
All of this makes TDV games more than just tools for use in educational settings. The games
offered by JAG4 are available to students via the internet at home, in schools, or public libraries,
allowing them to learn about abusive relationships at their own pace without feeling „preached to‟ by
adults. Schools and parents are then secondary beneficiaries: The games provide teachers and
administration with a free educational resource, while parents can benefit by educating themselves
and using the games as catalysts for conversations.
Like many other serious games, TDV games can affect knowledge, skills, and attitudes. On
the level of knowledge, TDV games can inform players by, for instance, making them aware of
warning signs of abuse. They can proffer skills onto players, teaching them how to react to abusive
situations (also as a bystander) or how to reach out for help. Apart from these effects, the primary
impact that a TDV game could have is by changing the attitudes of its players. Convincing individuals
that certain behaviors constitute abuse and should not be condoned is a critical issue facing TDV
interventions. This also makes TDV games so-called „persuasive games‟, games that were developed
with the intention to change players‟ attitudes about certain topics. While persuasive games have been
shown to positively affect player-held beliefs on several topics,11,17,18 evidence is still needed to
support attitude-changing effects of TDV games.19 Our study was therefore guided by the following
research question: Do teen dating violence games change the attitudes of their players towards this
topic when compared to games unrelated to the topic?
Materials and Methods
Procedure
To determine if playing a video game designed to prevent teen dating violence was effective in
changing attitudes towards unacceptable dating behaviors, an exploratory online study was performed
with a 2 (TDV and Control games) by 2 (pre- and post-test) mixed experimental design. Participants
GAMING AGAINST VIOLENCE: AN EXPLORATORY EVALUATION 5
were gathered through the Amazon Mechanical Turk service (mTurk)20 between November 2015 and
January 2016. Before participating, potential respondents were informed about the study with regards
to its subject matter (relationship violence), apparatus required to participate, the identity and
affiliation of the researcher, and time investment (between 45 – 75 minutes). After this briefing,
respondents were free to give or withhold consent electronically by participating (or not participating)
in the study. Using an online survey tool, participants were given a pre-test measuring dating violence
attitudes. Next, participants were shown to one of 7 games that were either TDV or control games,
and asked to come back to the survey after finishing the game. Afterwards, participants filled in a
post-test questionnaire on the same attitudes. They also completed a manipulation check item that
asked them to briefly summarize the game they played. Respondents were given the chance to
comment (or pose questions) on the study and the stimulus materials before being compensated
through the mTurk payment system.
As the study was conducted by an independent researcher (the second author), institutional
review board (IRB) approval was not available. With only the exception of the IRB (point 23), the
study conformed to the declaration of Helsinki21 as well as the mTurk guidelines for academic
requesters22 and official terms of service.23 The latter two include, among others, the availability of
grievance procedures for participants and requirements on data security/privacy. As the game stimuli
used were designed for younger audiences, they contain no harmful imagery or sounds. The
transparent, sensitive design of this study and the comments of the participants afterwards indicate
that this study carefully guarded against adverse effects on its participants. The benefits of the study
therefore outweighed the negligible negative outcomes.
Measures
Apart from demographic items on gender, age, and game-playing experience, participants filled in
two measures related to dating violence. The first was definitional, asking respondents to tick boxes to
indicate what constitutes abuse (e.g. physical, sexual). One decoy item was included („allegorical‟) to
control for respondents simply checking all the boxes. This item was analyzed by looking at the
change in the number of (real) elements of dating abuse reported, whereby an increase would show a
GAMING AGAINST VIOLENCE: AN EXPLORATORY EVALUATION 6
broader definition of what constitutes abuse. The second measure was an 11-item scale taken from the
Attitudes Towards Dating Violence Scales.24 Respondents indicated on a 5-point Likert scale their
agreement with statements about abusive situations (e.g. „a guy should not insult his
girlfriend/boyfriend‟ and „it is O.K. for a girl to slap her boyfriend/girlfriend if they deserve it‟).
Calculation of Cronbach‟s alpha showed that the reliability of this scale was acceptable (pre-test: α =
.813, post-test α = .787), and so all items were included in an average scale variable. Lastly, the
respondents rated the enjoyability of the game on a scale from 1 to 10.
Games
To get an overview of the generalized efficacy of the TDV games, 5 different JAG games were
included. The games were: Another Chance, Grace‟s Diary, Little Things, Love in the Dumpster, and
The Guardian. These games were all award winning games in different iterations of the Life.Love.
Game Design Challenge produced by JAG, and all were accessible online at the time of writing,4 with
each game taking 30 – 60 minutes to complete. Two control games were included, both relatively
popular browser-based games: Samorost,25 and Today I Die.26 Neither of these games contained any
information or behaviors related to TDV, while the dexterity requirements and gameplay were
broadly comparable to the TDV games.
Sample
The sampling procedure selected only participants living in the United States through address
verification. To avoid repeat participants, sampling proceeded by state, drawing a small number of
participants from each state for one game before opening the study with a different game to another
state. States sampled for the TDV games were California, Florida, Georgia, New Mexico, New York,
Texas, and Vermont. The control games were assigned to New Jersey and Virginia.
This sampling strategy yielded 86 participants. The majority (68%) were men, and more than
half (56%) were aged between 25 and 34 (24% was younger than 25, and 20% was older than 35).
45% of the sample reported playing games daily, with only 13% playing less than once a week. 66
GAMING AGAINST VIOLENCE: AN EXPLORATORY EVALUATION 7
participants played one of the TDV games as part of the study, with 20 participants playing either of
the two control games.
Results
A repeated measures ANOVA was performed to determine the influence of the teen dating violence
games when compared to the control games on their responses to the Attitudes Towards Dating
Violence scale. Significant main effects were found for the pre- and post-test difference (F(1,84)=7.1,
p=.009, η2p=.08), as well as for the condition (F(1,84)=6.7, p=.012, η2p=.07). An interaction effect
was also found, indicating that persons who played the TDV games changed their attitudes towards
the topic more than people who played the control game (see figure 1). All three effect sizes were
small. The differences between the pre- and post-tests between conditions were further tested through
two-sample t-tests. These showed that the two conditions did not significantly differ at pre-test
(t(84)=1.8, two-tailed-p=0.082, Hedges‟ gs=.45), though they were different at post-test (t(84)=3.3,
two-tailed-p=0.002, Hedges‟ gs=.83), with means indicating that the TDV players (M=4.6, SD=0.5)
had greater scores than the control players (M=4.2, SD=0.6). When looking at the difference scores
between the two observations, the increase was largest for players of the TDV games (Control M=.2,
SD=0.3, TDV M=.0, SD=0.2, t(84)=2.5, two-tailed-p=0.013, Hedges‟ gs=.64). Overall, the TDV
games incited greater attitude change in their players than the control games did.
GAMING AGAINST VIOLENCE: AN EXPLORATORY EVALUATION 8
Figure 1: Attitude change in the TDV and control game conditions. Higher scores indicate lower acceptance of
dating violence. The difference in pre-test between conditions is not significant at an alpha of .05, though that of
the post-test is.
Data were also analyzed for the definition of abuse, the TDV games separately, and for the
influence of demographic variables. For the item about definition, a ceiling effect negated any
possible effects. The average number of elements included in the definition was 5.6 on the pre-test,
and lowered to 5.5 on the post-test, though there were no significant discrepancies between the groups
on the difference score (t(65)=-1.5, two-tailed-p=0.150, Hedges‟ gs=.20). Looking at the TDV games
separately, no differences were found between the five games used in this study when they were
included in an ANOVA with attitude change as the dependent variable (F(4,61)=.5, p=.714). Lastly,
gender differences were found in the pre-test on an independent samples t-test when looking only at
players of the TDV games (t(64)=2.629, two-tailed-p=.011, Hedges‟ gs=.67), indicating women
(M=4.7, SD=.4) had higher scores than men (M=4.3, SD=.6). This difference was not found for the
post-test (t(64)=1.783, two-tailed-p=.079, Hedges‟ gs=.46). The attitude difference was close to
significantly different between genders (t(64)=1.947, two-tailed-p=.056, Hedges‟ gs=.50), indicating
GAMING AGAINST VIOLENCE: AN EXPLORATORY EVALUATION 9
that men‟s attitudes (M=.2, SD=.3) underwent greater changes than women‟s (M=.1, SD=.3), though
this was not significant at an alpha of .05.
Discussion
Summarizing, the interaction effect found can be taken to mean that the change in attitudes for TDV
game players was greater than it was for control game players. This can be seen in the slope of the
graph for the TDV group in figure 1, whereas the control group showed no increase. This means that
it is highly likely that the TDV games had an effect on the attitudes of its players. Because these
results cannot be separated per game, the only conclusion that can be drawn is that, on the whole,
games intended to have an effect on TDV attitudes affect these attitudes more than games that were
not designed for this purpose.
As this was an exploratory study, it could not prove without doubt that all TDV games affect
attitudes towards teen dating violence. As a first foray into this topic, however, these results are very
promising. Further research could improve on this study in the following ways. The mTurk sampling
strategy was not equal to the target audience, and it was not possible to randomize participants
completely. Although we did not find systematic differences in our sample in the origin state of the
participants, subtle differences between respondents of different states cannot currently be ruled out.
Future studies should attempt to replicate the current results by truly randomizing participants, but
also by taking a sample that is of sufficient size to detect possible differences in effects between the
TDV games. In this way, the focus of research can begin to move from simply determining effects to
establishing what properties of games increase or dampen their influences.
To conclude, despite the fact that the TDV games used in this study were not developed with
the current study‟s sample in mind, the games clearly affected their players. This result supports the
viability of persuasive games in combating the all-too-common phenomenon of teen dating violence.
Acknowledgments
This research was funded by Jennifer Ann's Group, a 501(c)(3) non-profit organization working to
prevent teen dating violence. See www.JenniferAnn.org. This article was written within the project
GAMING AGAINST VIOLENCE: AN EXPLORATORY EVALUATION 10
“Persuasive gaming. From theory-based design to validation and back” funded by the Netherlands
Organization for Scientific Research (NWO). See www.persuasivegaming.nl.
Author Disclosure
Drew Crecente is the founder/executive director of Jennifer Ann's Group and is in that capacity the
producer and publisher of the videogames hosted on the Group‟s site. There were no other competing
financial interests.
References
1. Forke CM, Myers RK, Catallozzi M, Schwarz DF. Relationship violence among female and
male college undergraduate students. Arch Pediatr Adolesc Med. 2008;162(7):634-641.
doi:10.1001/archpedi.162.7.634.
2. U.S. Census Bureau. Source and Accuracy Statement for the October 2010 Current Population
Survey Microdata File on School Enrollment. Washington; 2011.
https://www.census.gov/hhes/school/data/cps/2010/2010school_S&A.pdf.
3. Crecente D. Gaming Against Violence: A Grassroots Approach to Teen Dating Violence.
Games Health J. 2014;3(4):198-201. doi:doi:10.1089/g4h.2014.0010.
4. Jennifer Ann‟s Group. JAG Ga.me: Emotional health and wellness games for teens, parents,
and teachers. https://jagga.me/.
5. Bowen E, Walker K, Mawer M, et al. “It‟s like you‟re actually playing as yourself”:
Development and preliminary evaluation of “Green Acres High”, a serious game-based
primary intervention to combat adolescent dating violence. Psychosoc Interv. 2014;23(1):43-
55. doi:http://dx.doi.org/10.5093/in2013a5.
6. National Center for Injury Prevention and Control: Division of Violence Prevention.
Understanding Teen Dating Violence - Fact Sheet. 2016.
http://www.cdc.gov/violenceprevention/intimatepartnerviolence/teen_dating_violence.html.
7. Vagi KJ, O‟Malley Olsen E, Basile KC, Vivolo-Kantor AM. Teen Dating Violence (Physical
GAMING AGAINST VIOLENCE: AN EXPLORATORY EVALUATION 11
and Sexual) Among US High School Students: Findings From the 2013 National Youth Risk
Behavior Survey. JAMA Pediatr. 2015;169(5):474-482.
doi:10.1001/jamapediatrics.2014.3577.
8. Ashley OS, Foshee VA. Adolescent help-seeking for dating violence: prevalence,
sociodemographic correlates, and sources of help. J Adolesc Health. 2005;36(1):25-31.
doi:10.1016/j.jadohealth.2003.12.014.
9. De La Rue L, Polanin JR, Espelage DL, Pigott TD. A Meta-Analysis of School-Based
Interventions Aimed to Prevent or Reduce Violence in Teen Dating Relationships. Rev Educ
Res. 2016:1-28. doi:10.3102/0034654316632061.
10. Granic I, Lobel A, Engels RCME. The benefits of playing video games. Am Psychol.
2013;69(1). doi:10.1037/a0034857.
11. Ruggiero D. The effect of playing a persuasive game on attitude and affective learning.
Comput Human Behav. 2015;45:213-221. doi:10.1016/j.chb.2014.11.062.
12. Mellecker RR, Witherspoon L, Watterson T. Active Learning: Educational Experiences
Enhanced Through Technology-Driven Active Game Play. J Educ Res. 2013;106(5):352-359.
doi:10.1080/00220671.2012.736429.
13. Sung H-Y, Hwang G-J. A collaborative game-based learning approach to improving students‟
learning performance in science courses. Comput Educ. 2013;63:43-51.
doi:10.1016/j.compedu.2012.11.019.
14. Charles D, McAlister M. Integrating Ideas About Invisible Playgrounds from Play Theory into
Online Educational Digital Games. In: Rauterberg M, ed. Entertainment Computing – ICEC
2004 Volume 3166 of the Series Lecture Notes in Computer Science. Vol Berlin, Heidelberg:
Springer Berlin Heidelberg; 2004:598-601. doi:10.1007/978-3-540-28643-1_79.
15. Blumberg FC, Blades M, Oates C. Youth and New Media: The Appeal and Educational
Ramifications of Digital Game Play for Children and Adolescents. Z Psychol. 2013;221(2):67-
GAMING AGAINST VIOLENCE: AN EXPLORATORY EVALUATION 12
71. doi:10.1027/2151-2604/a000133.
16. Kovačević I, Minović M, Milovanović M, de Pablos PO, Starčević D. Motivational aspects of
different learning contexts: “My mom won‟t let me play this game….” Comput Human Behav.
2013;29(2):354-363. doi:10.1016/j.chb.2012.01.023.
17. Peng W, Lee M, Heeter C. The Effects of a Serious Game on Role-Taking and Willingness to
Help. J Commun. 2010;60:723-742. doi:10.1111/j.1460-2466.2010.01511.x.
18. Gerling KM, Mandryk RL, Birk MV, Miller M, Orji R. The effects of embodied persuasive
games on player attitudes toward people using wheelchairs. Proc 32nd Annu ACM Conf Hum
factors Comput Syst - CHI ’14. 2014:3413-3422. doi:10.1145/2556288.2556962.
19. Jozkowski KN, Ekbia HR. “Campus Craft”: A game for sexual assault prevention in
universities. Games Heal. 2015;4(2):95-106. doi:10.1089/g4h.2014.0056.
20. Amazon.com Inc. Amazon Mechanical Turk - Service Highlights.
https://requester.mturk.com/tour.
21. World Medical Association Inc. WMA Declaration of Helsinki - Ethical Principles for
Medical Research Involving Human Subjects. 2013.
http://www.wma.net/en/30publications/10policies/b3/index.html.
22. Guidelines for Academic Requesters. Dynamo Wiki. 2014.
http://wiki.wearedynamo.org/index.php?title=Guidelines_for_Academic_Requesters.
23. Amazon.com Inc. Amazon Mechanical Turk - Policies. 2016.
https://www.mturk.com/mturk/help?helpPage=policies.
24. Price EL, Byers ES, Belliveau N, et al. The Attitudes Towards Dating Violence Scales:
Development and Initial Validation. J Fam Violence. 1999;14(4):351-375.
doi:10.1023/A:1022830114772.
25. Amanita Design. Samorost. http://amanita-design.net/samorost-1.
GAMING AGAINST VIOLENCE: AN EXPLORATORY EVALUATION 13
26. Benmergui D. Today I Die. http://ludomancy.com/games/today.php.
Address correspondence to:
Ruud S. Jacobs, MSc.
Erasmus Research Centre for Media, Communication, and Culture
Department of Media & Communication
Erasmus University Rotterdam
Burgemeester Oudlaan 50
3062PA, Rotterdam
The Netherlands
E-mail: jacobs@eshcc.eur.nl
