Motivational and Cognitive Impact of Badges in Games for Learning

Abstract

How does the design of digital badges in a middle school geometry game impact motivational and cognitive learning outcomes? Based on achievement goal theory, learners were first primed for either mastery or achievement, and then received either mastery badges, achievement badges, or no badges while playing the game. Dependent measures included situational interest and learning outcomes. Result show that learning outcomes in the performance badges condition was significantly better than in the Mastery Badges condition. However, learners with higher situational interest performed better with Mastery badges. Findings suggest the importance of learner characteristics such as situational interest for the effectiveness of motivational features such as badges for games for learning.

Full text

AERA$2014$$ $ $$$$$$$$$$$$BADGES$AND$MOTIVATION$
Motivational and Cognitive Impact of Badges in Games for Learning
Jan L. Plass*, Paul A. O’Keefe°, Melissa L. Biles*, Jonathan Frye*, & Bruce D. Homer**
*New York University, **CUNY Graduate Center, °Stanford University
How does the design of digital badges in a middle school geometry game
impact motivational and cognitive learning outcomes? Digital games have
turned from a fringe genre to an accepted type of educational environment
(Honey & Hilton, 2011), and both male and female students are spending many
hours per week playing video games (Homer, et al., 2012). For researchers
studying the impact of game design features on their effectiveness (Plass et al.,
2013), incentive systems have received a great deal of interest, especially since
games’ ability to motivate learners has been linked to the incentives they
provide. One such incentive feature are digital badges, i.e., visual indicators of
accomplishments or skills within games that are earned during game play and
can be collected and displayed to other players. We are interested in exploring
how the decision what accomplishments or skills to recognize impacts learning
outcomes.
Literature Review
Research on Badges
Educational badges are often seen as a way to evaluate learning outside of a formal school
environment by providing a symbolic reward for particular knowledge, skills, or achievements
(Abramovich et al., 2013). While educational badge use is gaining in popularity in online
learning contexts such as Edmodo.com, the Khan Academy, and Mozilla’s P2PU, there is
relatively little research on how badges affect motivation. Initial findings by Li, Huang, &
Cavusoglu (2012) on badge use in StackOverflow found that participants who earned more
badges reported increased motivation to contribute in a range of online activities. Preliminary
research with cloud-based learning systems found that the increase in number of badges
earned was linked to a decrease in performance avoidance goals (Higashi et al., 2012).
For videogames, with the introduction of Microsoft’s Xbox Live online community in 2002,
badges gained widespread use representing in-game achievements, providing focused goals,
challenges, clear standards, and performance affirmation (Dickey, 2005). Altering particular
game elements (i.e. goals, choice, and personalization) has been found to affect the intrinsic
motivation of the player (Abramovich et al., 2011). However, the use of other motivational
frameworks such as achievement goal theory to experimentally examine in-game motivation
has not yet been explored.
Individual and Situational Interest
Interest and motivation are often described as interdependent (Schiefele, 1991; Ainley, 2006).
Interest can directly affect students’ motivation, learning strategies, and ability to process

AERA$2014$$ $ $$$$$$$$$$$$BADGES$AND$MOTIVATION$
information (Schiefele, 1991). Interest mediates the relationship between students’ overarching
goal orientation, their task-related goals, and their motivation to learn (Ainley, 2006). Interest is
typically described as content-specific for an individual, e.g., a learner may be interested in
history but not in math. However, we find it useful to also consider situational interest, which is
brought about by different circumstances or events. A person may not be interested in math
(low individual interest), but might find a math game with badges situationally interesting. Such
situational interest can lead to individual interest in the topic that persists over time (Hidi &
Renninger, 2006; Schiefele, 1991).
Achievement Goal Theory
According to achievement goal theory, there are two primary goals or reasons why students
engage in achievement behavior (Dweck & Leggett, 1988; Elliot, 2005). A mastery goal
orientation reflects a focus on learning and the development of abilities, while a performance
goal orientation reflects a focus on demonstrating or validating abilities relative to others. These
two main goal orientations differentially predict students’ behaviors, thoughts, and affect (Dweck
& Leggett, 1988), as well as their incentive preferences (see Butler, 2000b for a review). The
mastery-performance distinction also differentiates between approach and avoidance
dimensions (Elliot, 1999; Elliot & McGregor, 2001; Pintrich, 2000). In the current research, we
primarily focus on mastery-approach and performance-approach goal orientations.
Achievement Goal-Related Feedback
Achievement goal orientations can be invoked or reinforced with particular types of feedback
(see Butler, 2000b; O'Keefe, 2013). Because performance-approach goals are generally
focused on demonstrating ability relative to others, rather than developing it (Dweck & Leggett,
1988; Elliot, 2005), normative feedback, such as a percentile score, can support a performance-
approach goal orientation (Butler, 1993, 1995). In contrast, mastery-approach goals are
concerned with developing abilities and improving upon them (Dweck & Leggett, 1988; Elliot,
2005). Therefore, feedback about how one has performed compared to their previous
performance, such as their objective score on a game across several trials, can support a
mastery-approach goal orientation (Butler, 1993, 1995). Badges can be designed to provide
either type of feedback, and we are interested in how such designs impact learning outcomes.
Mastery-approach goals are highly adaptive. They provide a framework that supports an
authentic desire for challenges and learning, and resilience in the face of failure (O’Keefe, 2013).
Performance-approach goals have also shown to be beneficial, although they support a
motivation for learning that is less authentic. Most notably, they tend to be associated with
relatively higher task performance and achievement (Harackiewicz et al., 2002). Because failure
can have serious social consequences, performance-approach goals have the potential to
augment the motivation to perform well.
We therefore expect that students receiving badges with performance goal-related feedback will
outperform those receiving badges with mastery goal-related feedback. It is unclear, however,
how performance goal-related badges will compare with a condition in which there are no
badges. It could be predicted that the performance goal condition would outperform the

AERA$2014$$ $ $$$$$$$$$$$$BADGES$AND$MOTIVATION$
condition without badges given its addition of a powerful motivational framework. That said,
students from typical schools, such as the ones we recruited in the current study, tend to be
indoctrinated with performance goal-related concerns. Therefore, we might expect that students
in the condition without badges will perform equally as well as those in the performance goal
badge condition. Mastery goal badges may provide less motivation to perform at high levels in
the context of our geometry game.
Research Questions
● How do different goal orientations (Mastery vs. Performance) and badge designs (No
badges, Mastery badges, and Performance badges) affect learning outcomes?
Method
Participants and Design
Participants (N = 117) were sixth through eighth grade students from urban schools in a major
Northeastern city. Each participant was randomly assigned to one of six conditions within a 3 x
2 design; three levels of the badge condition: No badges, Mastery badges, and Performance
badges and two levels of priming: Mastery priming and Performance priming. Mastery badges
were designed to encourage the learner based on their own ability, utilizing phrases like “You
have mastered the triangle rule!” (Figure 1). Performance badges were designed to reward the
player in comparison to their peers, using phrases like “You figured out the straight angle rule
faster than most other players!” (Figure 2). Twenty-three students had to be excluded due to
game glitches or insufficient game play time. The remaining 85 participants were included in the
analyses.
Figure 1. Noobs v. Leets game with Mastery Badges

AERA$2014$$ $ $$$$$$$$$$$$BADGES$AND$MOTIVATION$
Figure 2. Noobs v. Leets game with Achievement Badges
Procedure
Participants were randomly assigned to conditions. They first completed survey measures with
Likert-scale items that assessed their individual interest in games and individual interest in
geometry using items such as “I like geometry” and “Games are exciting to me.” Afterwards,
they were administered a timed 10 minute pre-test of geometry-specific arithmetic and
conceptual knowledge questions. Participants were given a general introduction to the overall
game narrative, and participants in the badge condition were given an overview of the badges
available within the game.
Depending on their induction condition (mastery or performance), participants were asked to
read a short paragraph instructing them on the purpose of the game and how they should
approach gameplay and measure achievement within the game.
Participants next played the game, Noobs vs. Leets, for 25 minutes on desktop computers using
a mouse. Participants were given the instruction: “When playing the game, do the best you can.”
Game-play instructions were provided through cut-scenes and in-game tutorials. An
experimenter was also available during this time for questions about game play. At the end of
the game-play session, participants were given a timed 10 minute post-test of geometry-specific
questions and a situational interest survey.
Materials
Noobs vs. Leets is a PC-based puzzle game designed to teach the geometry concepts of
angles. The object of the game is to unlock angles, allowing the ‘Noob’ character to free a fellow
‘Noob’ from a cage. By unlocking angles, players open lines that the character can traverse. For

AERA$2014$$ $ $$$$$$$$$$$$BADGES$AND$MOTIVATION$
example, a player could click on a known angle and an unknown angle that together make a
straight line and then select the ‘supplementary rule’ button to unlock an angle (Figure 3).
Figure 3. Noobs v. Leets game
The game was modified from a previously studied version (Plass et al., 2012) to systematically
implement different badge designs (mastery and performance) in response to the player’s in-
game actions. In the mastery badges condition, badge messages align with mastery goal
orientation, helping the player assess their own ability and progress. In the performance badges
condition, messages align with performance goal orientation, giving feedback on progress
relative to other players’ performances.
The game is divided into six chapters that each focus on one particular angle concept from the
common core mathematics standards: simple, complementary, supplementary, vertical, triangle,
and quadrilateral. Each chapter consists of 8-10 levels that progressively increase in difficulty.
The chapter begins with a cut-scene introducing and explaining the new concept being
introduced. Levels are considered complete when a player has unlocked a path for the player’s
Noob to reach the caged Noob.
Measures
Participants completed pretests and posttests of geometry-specific arithmetic and conceptual
knowledge designed by an experienced middle school math teacher, using items from New
York state standards based exam questions. The pre-test of geometry-specific conceptual
knowledge consisted of 15 items that required participants to demonstrate their understanding
of acute, obtuse, right, straight, supplementary, and congruent angles; the post-test of
geometry-specific conceptual knowledge skills was 15 comparable items. The pre-test of
geometry-specific arithmetic skills consisted of 6 items that required participants to solve for

AERA$2014$$ $ $$$$$$$$$$$$BADGES$AND$MOTIVATION$
angles using simple arithmetic operations; the posttest of geometry-specific arithmetic skills was
a comparable set of 6 items.
Participants were also given the Situational Interest Survey (Linnenbrink-Garcia et al., in press).
The language in the situational interest measure was simplified to ensure comprehension in the
middle school sample.
Analysis & Results
The data were analyzed by conducting a 2x3 ANCOVA, with Induction (Mastery v.
Performance) and Condition (Mastery, Performance, or No Badges) as between subject factors
with pretest score, age and situational interest as covariates.
The main effect of pretest was significant and accounted for considerable variance in the
posttest score, F (1, 84) = 18.02, p < .001, partial eta squared = .196. There was also a main
effect of Condition, F (2, 84) = 3.60, p = .032, partial eta squared = .089. Pairwise comparisons
indicated that the Performance badges condition was not significantly different from the No-
Badges condition (p = .27), but was significantly better than the Mastery Badges condition (p
= .007). The No-Badges and Mastery badges did not significantly differ (p = .13) (Figure 4). The
other main effects were not significant.
Figure 4. Mean Knowledge Posttest Score by Condition
There was a significant interaction between Condition and Situational Interest, F (1, 84) = 4.50,
p = .01, partial eta squared = .11. Further analysis revealed that learners with higher situational

AERA$2014$$ $ $$$$$$$$$$$$BADGES$AND$MOTIVATION$
interest performed better with Mastery badges, but there was no relation between situational
interest and learning outcomes in the other conditions (see Figure 5). The three-way interaction
between Pretest x Condition x Situational Interest was also significant. Further analyses
revealed that learners with high Situational Interest did significantly worse than would be
expected by their pretest score in the Performance badges condition.
Figure 5. Interaction of Mean Knowledge Posttest Score, Situational Interest, and Condition
Discussion & Conclusion
Based around achievement goal theory, the current study investigated the impact of badges
with different designs on learning and situational interest. Individuals in the Performance badges
condition and No Badges condition outperformed individuals in the Mastery badges condition on
the posttest measures. Conversely, results also suggested that learners with higher situational
interest performed better with Mastery badges. However, the level of situational interest did not
significantly affect the Performance badge and No badge conditions. These findings suggest
that participants with high situational interest benefitted more from the badges with Mastery-
oriented messages.
References
Abramovich, S., Schunn, C., & Higashi, R. (2013). Are badges useful in education?: It depends upon the
type of badge and expertise of learner. Education Tech Research Development. DOI:
10.1007/s11423-013-9289-2.

AERA$2014$$ $ $$$$$$$$$$$$BADGES$AND$MOTIVATION$
Abramovich, S., Higashi, R., Hunkele, T. Schunn, C.D., Shoop, R. (2011). Achievement Systems to Boost
Achievement Motivation. GLS Conference Proceedings.
Ainley, M. (2006). Connecting with learning: Motivation, affect and cognition in interest processes.
Educational Psychology Review, 18(4), 391-405.
Butler, R. (1993). Effects of task- and ego-achievement goals on information seeking during task
engagement. Journal of Personality and Social Psychology, 65(1), 18-31.
Butler, R. (1995). Motivational and informational functions and consequences of children's attention to
peers' work. Journal of Educational Psychology, 87(3), 347-360.
Butler, R. (2000a). Making judgments about ability: The role of implicit theories of ability in moderating
inferences from temporal and social comparison information. Journal of Personality and Social
Psychology, 78(5), 965-978.
Butler, R. (2000b). What learners want to know: The role of achievement goals in shaping information
seeking, learning, and interest. In C. Sansone & J. M. Harackiewicz (Eds.), Intrinsic and extrinsic
motivation: The search for optimal motivation and performance (pp. 161-194). San Diego, CA:
Academic Press Academic Press.
Dweck, C. S., & Leggett, E. L. (1988). A social-cognitive approach to motivation and personality. [Journal;
Peer Reviewed Journal]. Psychological Review, 95(2), 256-273.
Elliot, A. J. (1999). Approach and avoidance motivation and achievement goals. [Print]. Educational
Psychologist, 34(3), 169-189.
Elliot, A. J. (2005). A Conceptual History of the Achievement Goal Construct. In A. J. Elliot & C. S. Dweck
(Eds.), Handbook of competence and motivation (pp. 52-72). New York, NY: Guilford Publications
Guilford Publications.
Elliot, A. J., & McGregor, H. A. (2001). A 2x2 achievement goal framework. [Journal; Peer Reviewed
Journal]. Journal of Personality and Social Psychology, 80(3), 501-519.
Halavais, A.M. C. (2012). A Genealogy of badges: Inherited meaning and monstrous moral hybrids.
Information, Communication and Society, 15(3), 354-373.
Harackiewicz, J. M., Barron, K. E., Pintrich, P. R., Elliot, A. J., & Thrash, T. M. (2002). Revision of
achievement goal theory: Necessary and illuminating. Journal of Educational Psychology, 94,
638-645.
Hidi, S., & Renninger, K. A. (2006). The four-phase model of interest development. Educational
Psychologist, 41(2), 111-127.
Hintz, R. (2009). Science Education in the Boy Scouts of America. Dissertation. Retrieved from Proquest:
http://www.eric.ed.gov/ERICWebPortal/search/recordDetails.jsp?ERICExtSearch_SearchValue_0
=ED513200&searchtype=keyword&ERICExtSearch_SearchType_0=no&_pageLabel=RecordDet
ails&accno=ED513200&_nfls=false&source=ae
Homer, B.D., Hayward, E.O, Frye, J. & Plass, J.L. (2012). Gender and Player Characteristics in Video
Game Play of Preadolescents. Computers in Human Behavior, 25(5), 1782-1789.

AERA$2014$$ $ $$$$$$$$$$$$BADGES$AND$MOTIVATION$
Honey, M. A., & Hilton, M. (2011). Learning Science Through Computer Games and Simulations. National
Academies Press.
Knight, E., & Casilli, C. (2012). Case study 6: Mozilla open badges. In D. Oblinger (Ed.), Game Changers:
Education and Information Technology. Retrieved from
http://www.educause.edu/library/resources/case-study-6-mozilla-open-badges
Kriplean, T., Beschastnikh, I., and McDonald, D.W. (2008). Articulations of wikiwork: uncovering valued
work in wikipedia through barnstars. Proceedings of the 2008 ACM conference on Computer
supported cooperative work, ACM, 47-56.
Linnenbrink-Garcia, L., Durik, A. M., Conley, A. M., Barron, K. E., Tauer, J. M., Karabenick, S. A., et al. (in
press). Situational interest survey (SIS): An instrument to assess the role of situational factors in
interest development.
Nussbaum, A. D., & Dweck, C. S. (2008). Defensiveness versus remediation: Self-theories and modes of
self-esteem maintenance. Personality and Social Psychology Bulletin, 34(5), 599-612.
O’Keefe, P. A. (2013). Mindsets and self-evaluation: How beliefs about intelligence can create a
preference for growth over defensiveness. In S. B. Kaufman (Ed.), The Complexity of Greatness:
Beyond Talent or Practice. Oxford: Oxford University Press.
Plass, J.L., Homer, B.D., Chang, Y. K., Frye, J., Kaczetow, W., Isbister, K. & Perlin, K. (2013). Metrics to
Assess Learning and Measure Learner Variables in Simulations and Games. In Eds. El-Nasr et
al., Game Telemetry and Metrics. Morgan Kaufman.
Plass, J.L., Homer, B.D., Hayward, E.O., Frye, J., Huang, T.T., Biles, M., Stein, M., & Perlin, K. (2012).
The Effect of Learning Mechanics Design on Learning Outcomes in a Computer-Based Geometry
Game. E-Learning and Games for Training, Education, Health and Sports. Lecture Notes in
Computer Science, 2012, Volume 7516/2012, 65-71. DOI: 10.1007/978-3-642-33466-5_7
Scheifele, U. (1991). Interest, learning, and motivation. Educational Psychologist, 26, 299–323.
Zhuolun, L., Huang, K. & Cavusoglu, H. (2012). Can We Gamify Voluntary Contributions to Online Q&A
Communities? Quantifying the Impact of Badges on User Engagement. ICIS Conference
Proceedings. Retrieved from
http://www.krannert.purdue.edu/faculty/kkarthik/wise12/papers%5Cwise12_submission_131.pdf