Content uploaded by Sean Brantley
Author content
All content in this area was uploaded by Sean Brantley on Apr 03, 2023
Content may be subject to copyright.
Beat the Bots: Exploring the Effects of Placebo Manipulation on
Performance During Video Gameplay
Sean Brantley, Michael Wilkinson, Jing Feng
North Carolina State University
This study investigates placebos and video games' usefulness as psychological research tools. One proposed
underlying mechanism of the placebo effect is participants' expectations. Such expectation effects exist in
sports psychology and healthcare domains, but inconsistent findings have emerged on whether similar effects
impact a participants' cognitive performance. Concurrently, using video games as task environments is an
emerging methodology relating to expertise and large-scale behavioral data collection. Therefore, this study
examines the expectancy effect induced by researcher instructions on in-game performance. The instructional
expectancy condition for this study is in-game successes framed using emoting (e.g., emoting under the
pretense of subsequent performance increases) versus a control group. Preliminary results showed no
evidence of different in-game performance between expectancy conditions. Potential mechanisms that could
have led to a lack of effect were discussed.
INTRODUCTION
The placebo effect is a psychoneurobiological response to
an inert treatment, which has been noted in many contexts
(Hurst et al., 2020). For example, in a sports psychology meta-
analysis, Hurst and colleagues (2020) found a significant
correlation between physical performance, such as power
output and speed, and two types of ergogenic aids, nutritional
and mechanical. All of these placebo treatments were devoid of
their active biological or mechanical components, leading the
researchers to propose that a likely mechanism modifying
performance may be participants' expectations of treatment
benefits. Similar expectancy effects have been observed in
other healthcare domains. Rossettini et al. (2020) proposed that
an extensive range of factors in a healthcare setting, one of
which being the patient-physiotherapist relationship (verbal and
non-verbal communication), is a possible modulator of patients'
treatment outcomes.
Despite the demonstration of how one's expectation could
play a role in sports performance or treatment outcomes, it is
unclear how such expectation may play a role in participants'
performance on cognitive tasks. Literature explicitly examining
objective cognitive performance alterations due to the
expectancy effects is split. On the one hand, the expectation that
auditory frequency stimulation would increase performance did
not impart increases to flanker task performance results
(Schwarz & Büchel, 2015). On the other hand, Tiraboschi, et al.
(2019) observed that an expectancy effect, the instructional
verbiage associated with single-session cognitive training (a
framed sudoku puzzle), did significantly increase posttest
useful field of view task performance, although the effect size
is small.
As the effect of expectation on cognitive task performance
is still unclear, this study explores this phenomenon while also
evaluating the usefulness of a novel remote task environment,
the video game League of Legends (LoL). LoL is a multiplayer
online battle arena (MOBA) video game developed in 2009 by
Riot Games. In 2015 this game represented twenty-two percent
of global online gameplay (Silva et al., 2017), and by 2019 the
game peaked at eight million concurrent players (Li et al.,
2020). LoL is updated using bi-weekly software patches and is
currently on patch 11.12, boasting 153 playable characters
(champions), and various featured game modes (Riot, 2021).
The complexity associated with using this task
environment is at face value a deterrent for rigorous
psychological experiment development. However, this upfront
shortcoming hides the immense underlying potential of this
methodology. Supporting this claim, a research group studying
expertise using an online game of the same genre, StarCraft 2,
states that real-time strategy games can serve cognitive science
the same way Drosophila has served biological science due to
the following list of seven qualitative or quantitative reasons
outlined in their introduction: (1) Dynamic task environment
(2) motivated participants (3) accurate measures of motor
performance and attention allocation (4) noninvasive measure
of performance (5) large datasets (6) numerous variables (7)
large range of expertise (Thompson et al., 2013).
Players also partake in this activity independent of any
study meaning that the complex in-game behaviors, which
might be psychologically manipulatable, are likely more
ecologically valid than those induced by typical psychological
tasks. Additionally, the player's specific motor outputs via
mouse/keyboard inputs translate into equally specific sub-
second in-game behaviors within the dynamic environment.
The combination of which suggests that gameplay meets the
criteria of ethological observation, possibly enabling the
crossover analysis of the two central cognitive science fields,
psychology, and ethology in future research (Gomez-Marin et
al., 2014).
Apart from this hypothetical usefulness, there are four
game-features that make LoL a feasible psychological task
environment specifically for exploring expectancy effect
manipulations. First, players control a single champion for the
entirety of a match (Li et al., 2020). Enabling detailed
performance analysis required to determine significant results
in any experiment. Second, the virtual arena summoner's rift
(Figure 1) does not change spatially or temporally between
matches (Silva et al., 2017). This provenance state creates
starting state task standardization like established experiments
(Kohwalter et al., 2018). Third, this game boasts two
mathematically derived in-game skill measures, an Elo ladder
(Figure 2), which is considered a player's peak in-game skill,
Copyright 2021 by Human Factors and Ergonomics Society. All rights reserved. 10.1177/1071181321651311
Proceedings of the 2021 HFES 65th International Annual Meeting 923
and champion mastery points (CMP), an accumulation of
champion-specific performance-based experience points,
equating to time spent playing that champion (Do et al., 2020;
Li et al., 2020). Together when these skill indicators are
combined, they allow researchers to account for two levels of
expertise, mitigating statistical noise surrounding expertise
inherent within any naturalistic task. Fourth, the in-game
emoticon wheel allows players to quickly communicate mental
states in-game, permitting noninvasive in-game communication
between the participant and researchers (Kou & Gui, 2020).
Figure 1. The entire summoner's rift from a zoomed-out perspective, like the
minimap perspective with more detail. Reproduced from "Gendered design
bias: Gender differences of in-game character choice and playing style in league
of legends," G. Gao, 2017, 29th Australian Conference on Computer-Human
Interaction, 307-317. Copyright 2017 by Association for Computing
Machinery.
Figure 2. Histogram displaying the global player distributions of the nine LOL
ranks and outlining the two ranked populations used in this study. Data were
gathered from League of Graphs (2021). Created with BioRender.com
It is expected that participants who received different
instructions, and thus have different expectations, will display
significantly different performances between expectancy
conditions. More specifically, participants emoting under the
illusion of presumed performance benefits (framing) will show
increased in-game performance over the control (no framing)
(Hypothesis 1).
METHOD
Participants
Twenty-three season-10 gold and platinum players (20
males and 3 females) with over thirty thousand MP on the
champion Ashe were recruited through social media sites,
amateur leagues, discord servers, and the featured custom game
lobby. This subpopulation is considered above-average both in
terms of the overall game and champion skill, but they would
not be considered exceptional. Work done by Thompson et al.
(2013) on the game StarCraft 2 corroborates this decision to use
the combination of gold and platinum level players as a subject
group. During their examination of variables importance versus
expertise, this group found that their data was more informative
when researchers grouped three neighboring ranks as a unit, for
instance, silver-platinum and gold-diamond. However, as silver
players are below average in terms of game skill and diamond
players are pushing toward being exceptional, the intermediate
ranks of gold-platinum were selected for this study to create a
large above-average group. All participants were over eighteen
and spoke fluent English. Before the experiment, all
participants signed an informed consent form and filled out a
demographic survey disturbed via Qualtircs.
Materials
Participants and researchers used their own computers and
LoL's free-to-play downloadable software as a proxy for the
typical laboratory settings (https://na.leagueoflegends.com/en-
us/; Riot, 2021). The study's task was designed using the custom
game mode feature available within the game client to create
two repeatable battles against a team of intermediate difficultly
AI's (Artificial Intelligence's), referred to as bots by the LoL
community (Figure 3).
Figure 3. A screenshot of the custom game lobby with the accurate bot
champion lineup.
In LoL, champions fit into one of six classes that correlate
to an optimal play strategy (Poeller et al., 2020). This study
utilized the anecdotally weak marksmen class, whose defining
characteristics are a ranged auto-attack and a general lack of
repositioning abilities within their four distinct abilities (kit).
This kit composition makes this champion type particularly
vulnerable when paired with an opposing team composition
possessing a large amount of crowd control (CC), which are
movement inhibiting status effects. Together this combination
of participant champion and AI composition combine to create
a losable task for even experienced players.
LoL matches are subdivided into three ambiguous phases,
and while participants will play through all three, much of the
collected data for this study will come from the early-game,
more commonly referred to as the laning phase (Ferrari, 2013).
Performance at this stage is preferred for evaluating expectancy
effects for three reasons: (1) Player and AI champions begin at
approximately the same strength level, (2) champions are at
Copyright 2021 by Human Factors and Ergonomics Society. All rights reserved. 10.1177/1071181321651311
Proceedings of the 2021 HFES 65th International Annual Meeting 924
their weakest making subtasks like last hitting minions or
killing a bot, the two main performance metrics of this study,
the most difficult during this period, and (3) LoL inherently
incorporates a snowballing mechanic wherein success increases
a player's strength, and then in turn that strength increases the
likelihood of future successes within the same match.
Collectively, these traits should make the early-game data the
most likely to be resistant to confounding effects and a likely
period to observe performance alterations due to expectancy
effects.
Finally, as the combat units called minions or creeps (non-
controllable AI's) are not a self-explanatory concept and are a
critical piece of this study, these units warrant a detailed
description. Every thirty seconds, starting at one minute and
five seconds, six or seven of these minions, collectivity called a
minion wave marches from both bases towards the middle of
the map in all three lanes. These minions are essential for many
reasons, but primarily, these units, upon death, provide the two
main currencies of LoL, gold and experience (Maymin, 2021).
Although any champion near a dying minion gains experience,
the primary focus of laning and this study is landing the final
damaging blow on these minions granting the player gold and
adding a tally to their overall creep score (CS). Ferrari (2013)
notes that one significant mark of an exceptional player is
consistently obtaining high CS per minute totals, as
professionals can average ten CS per minute while the rest of
the Elo ladder obtains anywhere from four to eight per minute.
Because this subtasks' success is closely tied to player skill and
highly variable, evaluation of this data point will likely indicate
the effectiveness of the placebo manipulations.
Design
Emoting expectancy is a between-subject factor that each
participant was randomly allocated to either group A
(experimental group) or group B (control) (Figure 4). The two
groups received different instructions via two separate rule
sheets regarding in-game emoting. This experimental set-up
mirrors the experimental framework of Tiraboschi et al. (2019)
that suggests the most effective way to explore the significance
of expectancy effects is for both experimental and control
groups to go through the same procedure but be given separate
instructions regarding the procedure's purpose.
Figure 4. Task design structure outlining experimental flow from top to bottom,
how conditions will be introduced, and Group/Subgroup Expectation
Manipulations. Created with BioRender.com
Tasks
The overall objective for all participants, regardless of
group, is to win both matches while following the match rules
provided via the rule sheet. The rule sheets contain instructions
on what champion to play, the summoner spells/rune page/item
build to take, when they can make rotations to other lanes, and
the overall objective of winning. However, there is a single
difference between the rule sheets given to groups A and B,
which are marked as important and are bolded. Group A's rule
sheet instructs participants to emote after kills under the false
pretense of prior research indicating the act of positively
framing in-game events has been shown to significantly
increase in-game performance during the second match, while
group B's rule sheet instructs the participate emote after kills
merely to maintain communication with the researcher during
this remote task. Following the baseline match, all participants
will rejoin the second custom game lobby and play out the
second match.
Survey
This study's survey included demographic questions and
an op.gg (a data analytics website with accurate statistics
associated with each LoL account) review performed by the
researcher as manual verification of the player's rank. The
demographic questionnaire portion will collect age, sex, Ashe
CMP, education level, typical game latency, and game
experience.
Procedure
Prior to the experiment, a researcher logged onto either
NCSU1 or NCSUA, two league accounts created to make the
researcher's rank inaccessible to participants. The researcher
then selected the custom game mode and populated the three
bots Nasus, Cassiopeia, and Karthus, creating the composition
for the baseline match. Next, the researcher swapped the AIs to
the intermediate difficult in this exact order described in the
previous sentence, as any other order changes the AI’s lane
assignments. After set-up, the participant was invited to join the
lobby, and the researcher made sure the participant has read and
understood the consent form and match rules. If so, the baseline
match began. During the matches, the researcher selected a
random champion and used the camera function of LoL to hover
their perspective above the participant's champion. When the
participant emoted, the researcher responded with a response
emote, and if the participant forgot to emote, they were
reminded of the condition either by a researcher emote or a text
prompt.
After the first match, the participant rejoined the custom
lobby, where the researcher then set up the experimental five-
bot lineup (Figure 3). The five populated in the experimental
match are Nasus, Blitzcrank, Zyra, Leona, and Karthus. Again,
the set-up is finished by switching the bot difficulty from
beginner to intermediate in the exact order as stated previously.
Before the second match began, the researcher reminded the
participant of the rules. After the second match, the participant
was asked to read and sign the debriefing form outlining the
Copyright 2021 by Human Factors and Ergonomics Society. All rights reserved. 10.1177/1071181321651311
Proceedings of the 2021 HFES 65th International Annual Meeting 925
study's deception, giving the participant a chance to delete their
data.
RESULTS
Participant performance variables collected and analyzed
include a binary match win/lose value, win/lose time, the creep
scores after the first eight waves (CS8), the creep score
differential after eight waves (CSD8), the time it takes to reach
level six (L6T), and the first kill time (FKT). All six of these
metrics are outlined as useful performance indicators by
Maymin (2018). The first two variables represent a participant's
overall performance, while the other four metrics represent
laning performance. This analysis focuses on the CS8 and the
FKT as these two metrics are not only a good representation of
the participant’s in-game performance, but also showed the
most variance of the six variables, making them the most
informative.
An independent sample t-test was conducted to compare
each of the two dependent variables between the two
expectancy conditions. There was no significant difference on
the creep score between no-framing (M = 32.82, SD = 5.08) and
framing conditions (M = 30.50, SD = 4.68), t(21) = 1.14, p =
0.27 (Figure 5). There was no significant group difference on
framing group on first kill time either between the no-framing
(M = 241.00, SD = 83.27),and framing (M = 194.25, SD =
82.79) conditions, t(21) = 1.35, p = 0.19 (Figure 6).
Figure 5. CS8 by framing group; no framing (n = 11) and framing (n = 12).
Figure 6. FKT in seconds by framing group; no framing (n = 11) and framing
(n = 12).
DISCUSSION
This study focused on the impact of an expectancy effect
manipulation on participant performance in a cognitively
loading video game task, specifically the performance metrics
differences across two custom bot matches held within LoL.
The preliminary results were not concomitant with the work
done by Tiraboschi et al. (2019), as different instructional
expectations did not produce significantly different task
performances between groups (2019). This lack of significance
might be attributed to a list of possibilities: (1) expectancy
effects do not alter cognitive task performance, (2) LoL is too
complex to be used as a psychological task, (3) the effect size
of the previously identified expectancy effect was small thus
difficult to observe reliably especially given we have
incomplete collected data from a small sample size so far, (4)
the instructions in this study were not strong enough to induce
framing, or (5) this subpopulation as experts were resilient to
expectancy effects. Our group believes that the lack of
significance is likely due to a combination of possibilities three
and four or solely a function of the fifth possibility.
Addressing the combination of the small effect size for
instructional expectancy effects seen in prior cognitive
performance work and the strength of the instructional placebo
implement in this study, it is likely that our instructional
wording needs alteration or that more potent expectancy effects
should be implemented over emote framing to elicit a
significant effect. Two potential performance modulators worth
considering in future research as stronger expectancy effect
conditions over emote framing are background music and the
perceived skill of the researcher. Prior research indicates that
background music can increase the speed of spatial processing
performance, meaning that background music paired with the
expectation of performance increases versus no expectation
pairing is, would likely make for a stronger expectancy
condition (Angel et al., 2010). Second, the perceived skill of the
researcher (running the study using a silver/gold-ranked
account versus using a grandmaster/challenger-ranked account)
acting as a proxy for competency may modulate the
participants' belief in the expectancy condition, in line with
"physiotherapist features" seen to be effective in clinical
settings (Rossettini et al., 2020).
Further related to participant characteristics, our
preliminary result may also be explained in accordance with
Atkinson's expectancy-value theory, that the effectiveness of
expectancy effects in clinical settings relies on patient self-
efficacy (Rief & Petrie, 2016). According to this theory, higher
self-efficacy relates to better treatment outcomes. A similar but
opposite effect may occur during this cognitive task, as the
expert players are possibly so confident in their skills that they
are resilient to performance alterations due to expectation. A
follow-up study swapping the gold/platinum players for
bronze/silver players (a below-average subpopulation) may
show significant effects, which would suggest that there is a
window of expertise susceptible to instructional placebos in
other cognitive tasks.
Although with limitations, future studies could still
consider using RTS games as task environments. These virtual
environments may provide a viable way to look at naturalistic
moment-to-moment human behavior through the lens of
ethology while not forgoing researchers' ability to implement
experimental psychology manipulations. Relating to this
proposed use is ethological research on animal migration
patterns using high-resolution tracking (Kays et al., 2015).
Tracked animals' high-resolution lines have led to a better
understanding of animal-animal and animal-environment
interactions, and similar paths could be created to represent the
Copyright 2021 by Human Factors and Ergonomics Society. All rights reserved. 10.1177/1071181321651311
Proceedings of the 2021 HFES 65th International Annual Meeting 926
human mind during virtual combat (Figure 7). We believe that
creating software similar to the open-source optical tracking,
which enabled Maymin (2018) to record millions of matches
using the spectate function, could achieve this type of
ethological observation for psychological research as long as
the players are informed that they are participating in a study.
It is believed that these in-game virtual reality paths could
be extended along an axis of time, creating three-dimensional
blocks representing exact behavior output, essentially
providing virtual spacetime lines of behavior. As mathematical
representations of motor output, these virtual reality paths could
find uses in game theory, neurobiology, sports psychology,
game development, and AI research. One future direction using
this analytical approach could be exploring guided versus non-
guided learning (using an instructional video) of an LoL jungle
path. By collecting both the difference in clear speed time and
evaluating the path refinement process that correlates to these
improvements, these virtual reality lines could provide insights
into the process of motor learning over an extended period of
twenty or thirty trials, with the added benefit of looking at path
refinement. Second, as LoL is a team game, these lines could
also be looked at as parts of collective dynamic team shape
(shape created by the team over time) informing corporation
research.
Figure 7. Traditional Animal Tracking (Yellow Dots) vs. High-Resolution Big-
Data Animal Tracking (Grey and Red Line). Reproduced from "Terrestrial
animal tracking as an eye on life and planet," by R. Kay, 2015, Science,
348(6240). DOI: 10.1126/science.aaa2478. Copyright 2015 by American
Association for the Advancement of Science
REFERENCES
Angel, L. A., Polzella, D. J., & Elvers, G. C. (2010). Background
Music and Cognitive Performance 1,2. Perceptual and Motor
Skills, 110(3C), 1059–1064.
https://doi.org/10.2466/04.11.22.pms.110.c.1059-1064
Do, T. D., Yu, D. S., Anwer, S., & Wang, S. I. (2020). Using
Collaborative Filtering to Recommend Champions in League of
Legends. IEEE Conference on Computatonal Intelligence and
Games, CIG, 2020-August, 650–653.
https://doi.org/10.1109/CoG47356.2020.9231735
Ferrari, S. (2013). From generative to conventional play: MOBA and
League of Legends. DiGRA 2013 - Proceedings of the 2013
DiGRA International Conference: DeFragging GameStudies.
Gao, G., Min, A., & Shih, P. C. (2017, November). Gendered
design bias: Gender differences of in-game character
choice and playing style in league of legends. In
Proceedings of the 29th Australian Conference on
Computer-Human Interaction, 307-317.
Gomez-Marin, A., Paton, J. J., Kampff, A. R., Costa, R. M., & Mainen,
Z. F. (2014). Big behavioral data: Psychology, ethology and the
foundations of neuroscience. Nature Neuroscience, 17(11),
1455–1462. https://doi.org/10.1038/nn.3812
Kays, R., Crofoot, M. C., Jetz, W., & Wikelski, M. (2015). Terrestrial
animal tracking as an eye on life and planet. Science, 348(6240),
aaa2478. https://doi.org/10.1126/science.aaa2478
Kohwalter, T. C., Murta, L. G. P., & Clua, E. W. G. (2018). Capturing
game telemetry with provenance. Brazilian Symposium on
Games and Digital Entertainment, SBGAMES, 2017-
November, 66–75.
https://doi.org/10.1109/SBGames.2017.00016
Kou, Y., & Gui, X. (2020). Emotion Regulation in eSports Gaming: A
Qualitative Study of League of Legends. Proceedings of the
ACM on Human-Computer Interaction, 4(CSCW2).
https://doi.org/10.1145/3415229
League of Graphs. (2021, March 16). Rank Distribution.
https://www.leagueofgraphs.com/rankings/rank-distribution
Maymin, P. Z. (2018). An Open-Sourced Optical Tracking and
Advanced eSports Analytics Platform for League of Legends.
Sports Analytics Conference, 1–16.
Li, X., Huang, L., Li, B., Wang, H., & Han, C. (2020). Time for a true
display of skill: Top players in League of Legends have better
executive control. Acta Psychologica, 204(August 2019),
103007. https://doi.org/10.1016/j.actpsy.2020.103007
Maymin, P. Z. (2021). Smart kills and worthless deaths: ESports
analytics for League of Legends. Journal of Quantitative
Analysis in Sports, 17(1), 11–27. https://doi.org/10.1515/jqas-
2019-0096
Poeller, S., Baumann, N., & Mandryk, R. L. (2020). Power Play: How
the Need to Empower or Overpower Other Players Predicts
Preferences in League of Legends. Conference on Human
Factors in Computing Systems - Proceedings, April.
https://doi.org/10.1145/3313831.3376193
Rief, W., & Petrie, K. J. (2016). Can psychological expectation models
be adapted for placebo research? Frontiers in Psychology,
7(NOV), 1–6. https://doi.org/10.3389/fpsyg.2016.01876
Riot Games (2020). League of Legends. League of Legends Home
Page. Retrieved from: https://na.leagueoflegends.com/en-
us/?utm_source=riotbar&utm_medium=card1&utm_campaign
=lol&utm_content=lol_key01
Rossettini, G., Camerone, E. M., Carlino, E., Benedetti, F., & Testa,
M. (2020). Context matters: the psychoneurobiological
determinants of placebo, nocebo and context-related effects in
physiotherapy. Archives of Physiotherapy, 10(1), 1–12.
https://doi.org/10.1186/s40945-020-00082-y
Schwarz, K. A., & Büchel, C. (2015). Cognition and the placebo effect
- Dissociating subjective perception and actual performance.
PLoS ONE, 10(7), 1–12.
https://doi.org/10.1371/journal.pone.0130492
Silva, M. P., Silva, V. do N., & Chaimowicz, L. (2017). Dynamic
difficulty adjustment on MOBA games. Entertainment
Computing, 18, 103–123.
https://doi.org/10.1016/j.entcom.2016.10.002
Thompson, J. J., Blair, M. R., Chen, L., & Henrey, A. J. (2013). Video
Game Telemetry as a Critical Tool in the Study of Complex
Skill Learning. PLoS ONE, 8(9).
https://doi.org/10.1371/journal.pone.0075129
Tiraboschi, G. A., Fukusima, S. S., & West, G. L. (2019). An
Expectancy Effect Causes Improved Visual Attention
Performance After Video Game Playing. Journal of Cognitive
Enhancement, 3(4), 436–444. https://doi.org/10.1007/s41465-
019-00130-x
Copyright 2021 by Human Factors and Ergonomics Society. All rights reserved. 10.1177/1071181321651311
Proceedings of the 2021 HFES 65th International Annual Meeting 927
