Engaging Engineering Students with Gamification

Abstract

Well-designed games are good motivators by nature, as they imbue players with clear goals and a sense of reward and fulfillment, thus encouraging them to persist and endure in their quests. Recently, this motivational power has started to be applied to non-game contexts, a practice known as Gamification. This adds gaming elements to non-game processes, motivating users to adopt new behaviors, such as improving their physical condition, working more, or learning something new. This paper describes an experiment in which game-like elements were used to improve the delivery of a Master’s level College course, including scoring, levels, leaderboards, challenges and badges. To assess how gamification impacted the learning experience, we compare the gamified course to its non-gamified version from the previous year, using different performance measures. We also assessed student satisfaction as compared to other regular courses in the same academic context. Results were very encouraging, showing significant increases ranging from lecture attendance to online participation, proactive behaviors and perusing the course reference materials. Moreover, students considered the gamified instance to be more motivating, interesting and easier to learn as compared to other courses. We finalize by discussing the implications of these results on the design of future gamified learning experiences.

Full text

Engaging Engeneering Students with Gamification
An empirical study
Gabriel Barata, Sandra Gama, Joaquim Jorge, Daniel Gonçalves
Dept. of Computer Science and Engineering
INESC-ID/IST/Technical University of Lisbon
Lisbon, Portugal
gabriel.barata@ist.utl.pt, sandra.gama@ist.utl.pt, joaquim.jorge@inesc-id.pt, daniel.goncalves@inesc-id.pt
Abstract—Well-designed games are good motivators by
nature, as they imbue players with clear goals and a sense of
reward and fulfillment, thus encouraging them to persist and
endure in their quests. Recently, this motivational power has
started to be applied to non-game contexts, a practice known as
Gamification. This adds gaming elements to non-game processes,
motivating users to adopt new behaviors, such as improving their
physical condition, working more, or learning something new.
This paper describes an experiment in which game-like elements
were used to improve the delivery of a Master’s level College
course, including scoring, levels, leaderboards, challenges and
badges. To assess how gamification impacted the learning
experience, we compare the gamified course to its non-gamified
version from the previous year, using different performance
measures. We also assessed student satisfaction as compared to
other regular courses in the same academic context. Results were
very encouraging, showing significant increases ranging from
lecture attendance to online participation, proactive behaviors
and perusing the course reference materials. Moreover, students
considered the gamified instance to be more motivating,
interesting and easier to learn as compared to other courses. We
finalize by discussing the implications of these results on the
design of future gamified learning experiences.
Keywords—Gamification; Education; Student participation;
Classroom learning; Evaluation
I. INTRODUCTION
Education is a great concern of modern society and many
efforts have been applied to make it both more effective and
available to everyone. Traditional teaching techniques rely on
blackboards, oral lectures, books and written exercises as the
main vehicles to transmit knowledge. However, technological
process brought new possibilities to teach and educate, such as
using video games [1, 2], which spiced up curiosity and fired
discussions amongst many educators and researchers [3, 4, 5].
Research shows that video games have a great potential to
improve learning experience and learning outcomes. In
independent experiments, middle-school, high-school and
college students were subject to learning with video games,
reporting significant improvements in subject understanding,
diligence and motivation [6, 7, 8, 9, 10, 11]. Good games are
natural learning machines, as found by Gee [12]. Unlike
traditional educational materials, games can deliver
information on demand and within context. Successful games
are designed to be challenging enough to prevent players from
becoming either bored of frustrated, thus allowing them to
experience flow [13, 14].
The motivational power of games is being explored in
many non-game contexts, through a novel technique called
gamification. It adds game elements to non-game processes,
rather than using full-fledged games [15, 16]. Gamification has
been used mostly to keep users engaged with products and
motivated to perform certain behaviors [17, 18], which has
made it particularly useful for marketing [19, 20]. However, it
has also found applications in many other domains, such as
helping people become healthier (e.g. Nike+1, ZombiesRun2),
more productive [21] or more eco-friendly [22].
Many gamified systems focus on keeping users engaged
while learning new techniques and tools. Microsoft Ribbon
Hero3, for example, encourages users to explore Microsoft
Office tools, and Adobe LevelUp4 does the same for
Photoshop. Jigsaw [23] helps users learn Photoshop, through
an embedded jigsaw puzzle that challenges players to match a
target image. Although an empirical evaluation of this
technique has yet to be performed, users reported being able to
explore the tool and discover new techniques. GamiCAD [24]
is a gamified tutorial system on how to perform line and
trimming operations in AutoCAD. By completing tasks, users
help NASA build a spacecraft to participate in an Apollo
mission. Tasks are designed to be challenging and users are
encouraged to repeat those until they achieve the required
score. Results show that users completed tasks faster in
GamiCAD and found the experience to be both more engaging
and enjoyable than using a non-gamified version.
In his book, Lee Sheldon [25] describes how a conventional
learning experience can be designed as a game without
resorting to technology, to engage students and make classes
more fun and exciting. Students start with an F and go all the
way up to an A+, by completing quests and challenges, and
gaining experience points. However, little statistical data are
provided to support the claimed potential benefits of this
approach. Khan Academy5 is a free online service that allows
1 http://nikeplus.nike.com/
2 https://www.zombiesrungame.com/
3 http://www.ribbonhero.com/
4 http://success.adobe.com/microsites/levelup/index.html
5 http://www.khanacademy.org

users to learn about several subjects, like algebra, economics or
history, by watching videos and then performing exercises.
Their progress is rewarded with energy points and badges. In a
similar fashion, Codeacademy6 teaches online students to code
in several programming languages, also using points and
badges to track their progress. Even though systems like these
have been used in flipped classrooms [26], their applicability to
conventional classroom setups is limited. We still lack
empirical data to support any correlation between gamified
setups and student engagement. With a wide range of course
types, student backgrounds, learning preferences and socio-
economic environments, more systematic studies of the
influence of different gamification techniques are required to
assess how effective they can be.
In this article we evaluate how gamification affects students
of a college course named Multimedia Content Production
(MCP). Firstly, we describe a long-term study carried out over
two consecutive academic years. In the first year, a non-
gamified version of the course was used, similarly to previous
years. In the second year, we deployed a gamified edition,
containing game elements such as experience points, levels,
leaderboards, challenges and badges. Secondly, we compare
different aspects of the students’ learning experience, including
attendance, participation and usage of reference materials.
Then we carefully analyze the resulting empirical data, which
show significant gains in engagement, dedication and
satisfaction. We finalize by discussing the lessons learned from
this experiment and by deriving relevant design implications to
future gamified learning experiences.
II. THE MCP COURSE
MCP is an annual semester-long MSc course in
Information Systems and Computer Engineering at Instituto
Superior Técnico. This engineering school has two campi,
Alameda and Taguspark, where the course runs
simultaneously. In the first year, course evaluation consisted of
five theoretical quizzes (25% of total grade, the best out of six),
a multimedia presentation (20%), lab classes (15%), a final
exam (35%, the best out of two attempts), online participation
(5%, for online forum discussions) and class attendance (5%
extra grade). The final grade was represented with a value
between 0 and 20. In the second year, instead of grade points,
students participated in a game-like experience and were
awarded experience points by meeting the traditional
evaluation criteria. It consisted of quizzes (20%), a multimedia
presentation (20%), lab classes (15%), a final exam (35%) and
a set of collectible achievements (10% plus a 5% extra).
Compared to the previous edition of the course, the evaluation
method was thus similar, with achievements replacing the
online participation and attendance bonuses, as well as 5% of
the grade originally awarded by the quizzes.
In both years, the faculty staff was composed by the same
four teachers, two for each campus. The courses were
completely synchronized, using a single shared online Moodle
platform7. In the first year, 42 students attended the course (12
in Alameda and 30 in Taguspark), a number that decreased to
6 http://www.codecademy.com/
7 http://www.moodle.org
35 in the second year (12 in Alameda and 23 in Taguspark).
Students had similar backgrounds in both years. A large
majority had finished their undergraduate computer science
degree on the previous year, and three foreign exchange
students (Erasmus). In the first year, the syllabus included 19
regular lectures, 13 lab classes and three invited lectures, whilst
in the second one, the numbers were similar: 18 lectures, 12 lab
classes and four invited lectures. The theoretical lectures span
multimedia concepts ranging from capture, to editing and
production techniques, file formats and multimedia standards,
as well as Copyright and Digital Rights Management. In
laboratory classes, diverse concepts and tools were introduced
to students on image, audio and video manipulation. The PCM
Media Mixer prototype, a multimedia editor built on
DirectShow, was also introduced, for which students had to
develop manipulation and navigation plugins.
III. GAMIFYING THE COURSE
By analyzing student performance data from previous
years, we found room to improve student engagement in the
course, as shown by their low online participation on Moodle,
low attendance rates and lack of interest in the reference
material (low number of downloads). In order to make the
course more engaging, fun and interesting, we gamified it by
adding experience points (XP), levels, leaderboards, challenges
and badges, which seem to be some of the most consensual
game elements used in gamification [27, 28, 20, 29].
Similarly to Sheldon’s approach [25], instead of beginning
with the maximum grade and having to strive to maintain it,
our students begin with the minimum. Thus, they are
encouraged to learn from failure instead of feeling penalized.
In place of the traditional grading system, students now have
XP, which are awarded to every course activity they can
undertake. XP provide direct feedback on how successful
students are being and also serve as instant gratification, which
was previously shown to be successful in motivating college
students [30].
Students climb through levels as they progress at a rate of
900 XP per level or 900 XP to complete level 1, 1800 XP to
finish level 2, 9000 XP to get to level 10 and 18000 XP to
reach 20, the top level. Each level thus directly translates to an
assigned grade on a 0-20 scale. To prevent rounding problems,
we gave students a head start of 450 XP, awarded as a bonus
for enrolling in the course. Furthermore, to introduce a
distinctive flavor to levels, each corresponded to a unique title.
For instance, level 1 was labeled “Starting to see the light”,
level 4 “Taking your first Steps”, level 12 “Knowledge
Pilgrim” and level 20 “Science God”. Levels, together with
points, transmit progress and positive feedback to students.
The leaderboard webpage (see Fig. 1) provides an entry
point to the gamified experience. It is publically accessible
through the Moodle forum and displays enrolled students by
row, sorted in descending order by level and XP. Each row
portrays the player’s rank, photo and name, campus, XP, level
and achievements awarded for completing course activities. As
examples of such activities we have attending lectures, finding
resources related to class subjects, finding bugs in class
materials or completing challenges. Just by looking at the

leaderboard, students can monitor their own progress but also
see how others are doing. Furthermore, by clicking a student’s
row, the achievement history for that player is displayed (see
Fig. 2). This makes progression transparent and allows students
to learn by watching others. The leaderboard transmits progress
but also provides players with means to compare themselves to
others, which spurs competitiveness and may render the
experience more engaging.
Challenges are the major behavior drivers in the
experience. They consist of actions students must perform to be
awarded with XP and collectible achievements, or badges.
There were two main Challenge categories. Theoretical
Challenges are activities presented to students throughout the
semester, at the end of some lectures. These were organized as
small creative tasks designed to explore multimedia types and
materials taught in those lectures. Students could earn a “Rise
to the Challenge” achievement by accomplishing these tasks.
The other category consists of Lab challenges, which were
assigned during the first month of classes. These were meant to
be fun and expressive, by allowing students to produce creative
content using multimedia tools introduced in lab classes to earn
the “Proficient Tool User” badges. Throughout the rest of the
semester, students could earn achievements by finding bugs in
the PCM Media Mixer prototype (“Bug Squasher” badges). All
challenges were normally assigned a one-week deadline.
Challenges were formally issued via posts to course fora by
faculty. Achievements and graded activities were recorded
based mostly on students’ posts, except for attendance, lab
grading, quizzes and exams, which were manually scored.
Challenges and achievements serve the main purpose of
structuring course activities into meaningful endeavors while
allowing students to choose what tasks and achievements to
pursue. Achievements also provide important feedback to
transmit how proficient students become on specific aspects of
the course. They also grant students with bragging rights and
explore their need to collect. Whilst some achievements are
single-level, i.e., only require players to perform the task once,
others are multi-level, which require multiple iterations with
increasing difficulty. Multi-level achievements aim to make
things more interesting for students, by portraying progress and
allowing them to choose whether to level up or not.
To make the onboarding process easier, we conceived a
few less difficult achievements to be awarded during first steps.
This provided students with early feedback and motivation to
keep going. Moreover, we assigned greater scores to the first
levels of multi-level achievements, to encourage students to try
different challenges. To further motivate students to embrace
optional activities, we created achievements that awarded them
extra XP (although the final grade tops at 18000 XP). We had a
total of 900 bonus XP distributed by 39 optional achievements
and 1800 mandatory XP shared throughout 22 achievements.
The remaining mandatory 16200 XP were allocated to quizzes
(3600 XP), lab evaluation (2700 XP), a thematic multimedia
presentation (3600 XP) and a final exam (6300 XP, best out of
two). We had five quizzes (scored best out of six) on every
other week, each worth 600 XP (120 XP per question). There
were four laboratorial evaluations worth 675 XP each, in which
students had to develop algorithms to manipulate video data.
The whole scoring process was done manually. Data from
lectures and lab classes were collected by faculty on excel
sheets with macros, which then exported the information to text
logs. Also, data logs from Moodle activity were daily
downloaded by hand. Then, faculty ran a python script
manually to process all log files and generate the leaderboard
webpage. The script was run two to three times a day to track
major updates with low perceived response time.
It is important to remark that some of the game elements
including points and badges, work as extrinsic rewards, and
that extrinsic motivation may crowd out the intrinsic [31, 32].
However, these elements try to align the goals of the course
with those of the students (to learn and pass the course), which
should motivate them through identification and integrated
regulation. According to the self-determination theory [33],
these are the most autonomous forms of extrinsic motivation
and share some qualities with the intrinsic forms. Thus, we try
to use gamification as a means to amplify the intrinsic value
[34]. To attain this, we try to improve the feelings of
competence, autonomy and relatedness in students, which
satisfy three basic needs to intrinsic motivation [33]. The sense
of competence is promoted through positive feedback and
displaying progress via points, levels and badges. These are
also important characteristics of playing and learning [35]. We
provide learners’ with a sense of autonomy by allowing them
to choose which challenges to pursue and which achievements
to level up. Finally, we boost their sense of relatedness by
allowing them not only to cooperate and share opinions on
Moodle but also to compete via the leaderboard.
Fig. 1. MCP course leaderboard
Fig. 2. Badges and Achievements

IV. RESULTS
We collected data regarding many aspects of student
performance and satisfaction, such as the number of course
lecture slides downloads, the number of posts in forums, class
attendance and grades. These elements have previously been
used as informal measures of student engagement [36]. In this
section we will compare these values between the two years
and address the main changes between them. We will also
present and discuss student feedback from the satisfaction
surveys, issued at the end of the course. Since data do not
appear to follow a normal distribution, all statistical differences
between groups were checked using a non-parametric Mann-
Whitney’s U test.
A. Number of Downloads
Despite having fewer students, we had 789 (2765 vs. 1976)
more course lecture slides downloaded in total. A Mann-
Whitney’s U test revealed significant differences between the
two years (U = 75, Z = -2,917, p < 0,003), showing an increase
in average downloads per lecture of 49.61 (153.61 vs. 104),
which translates to a gain of 47.7% (see Fig. 3). Lecture
material downloads per student also had a similar growth (see
Fig. 4), with a total increase of 31.95 (79 vs. 47.05) downloads
and a significant growth per lecture of 1.91 (4.39 vs. 2.48)
(Mann-Whitney’s U test, U = 51, Z = -3.647, p < 0.001).
When analyzing how course materials were downloaded
throughout the semester, one thing caught our attention. By the
middle of the semester, downloads dropped dramatically (see
Fig. 5). This was already expected to happen during Easter
break (April 21th to 27th), but it started a couple of weeks
earlier. It turns out that other courses had mid-term exams
during that period, which diverted the students’ attention.
Interestingly enough, the number of downloads never went
back to previous rates, which might be related to schedule
constrains on students. We could also see that there was a
slight rate increase during quiz weeks.
B. Number of Posts
Posts made by students increased drastically with the
gamified experience (see Fig. 6), thus suggesting both higher
participation and proactivity. There was a total growth of 845%
(104 vs. 11) in initiated threads (first posts) and an average
growth per week of 751% (5.2 vs. 0.61), differences which are
statistically significant (Mann-Whitney’s U test, U = 36.5, Z =
-4.314, p < 0.00001). Reply posts also grew, with total gains of
511% (801 vs. 131) and gains per week of 450% (40.05 vs.
7.28) (Mann-Whitney’s U test, U = 66, Z = -3.337, p < 0.001).
Replies and first posts combined together increased by 537%
(905 vs. 142) totally and 474% (45.25 vs. 7.89) in a weekly
basis (Mann-Whitney’s U test, U = 52.5, Z = -3.731, p <
0.0001).
Results also lead us to believe that students were not the
only ones getting excited by the MCP game. It turns out that
Fig. 3. Number of downloads of lecture slides
Fig. 4. Average lecture slides downloaded per student
Fig. 5. Slide downloads over time during the second year
Fig. 6. Growth in number of posts from students
Fig. 7. Growth in posts from faculty

faculty also made more posts, increasing 373% (534 vs. 113)
with a weekly growth of 325% (26.7 vs. 6.28), and these
differences were statistically significant (Mann-Whitney’s U
test, U = 69, Z = -3.251, p < 0.001) (see Fig. 7). However, this
significance only extends to the number of replies, where total
gains reached 504% (465 vs. 77) and with weekly gains of
444% (23.25 vs. 4.28). As for first posts, there were slight
increases whose significance could not be demonstrated. These
changes in the faculty activity were expected, due to increased
student demands for feedback.
Student posts were expected to increase, given that
evaluation changed to include achievements. Despite being
worth 10% of the grade (+5% bonus), this new component was
responsible for 363 posts, which is more than twice the total
posts in the previous year, thus suggesting it can indeed engage
students into participating in forums. The evaluation
components unchanged between the two years were
responsible for 542 student posts, which grew by 281%
compared to the first year’s 142. This suggests that the
gamified course engaged students to participate significantly
more, even without considering the new evaluation component.
Post activity to forums throughout the semester, both by
students and faculty, increased by 464% to 1439 (either initial
or reply posts), as compared to the previous year’s 255. Of
these, 402 were related to Theoretical Challenges and 279
discussed Lab Challenges (see Fig. 8). This suggests that the
tandem challenge/reward is a powerful driver of student
behavior, accounting for 47% of all posts.
Analyzing post activity over time, we saw that during the
three-week period leading to Easter break, posts follow a
similar pattern to downloads (see Fig. 9). There was a
noticeable decrease that might be related to mid-term exams
from other courses, a phenomenon also evident in the previous
year (see Fig. 10). Furthermore, from the Easter break on,
posts activity never quite recovered pre-break rates. This might
have been caused by lack of time due to other courses, which
usually hand out their project assignments at this time. Further
study is necessary to understand whether this is the sole reason.
A close comparison of post evolution between the two
years shows that the second year indeed saw more posts, but
the majority occurred in the first two months of classes (see
Fig. 11). This might be related to the challenges posted during
this period. The three Lab Challenges were posted during the
first month of classes and added up to 279 posts, 19% of the
total amount for the semester. Moreover, these challenges had
the three longest discussion threads, with 94, 93 and 92
messages respectively. Theoretical Challenges also seem to
have a very significant impact, with six out of nine posted
during the first two months, accounting for almost 84% (336)
of the posts regarding Theoretical Challenges. For five of the
challenges, 43% of the students (15) gave multiple answer
posts (average of 1.6 additional posts), even though only one
would be graded, suggesting that they enjoyed the freedom of
creativity and were having fun. All of this goes to show that
challenges indeed led to more student participation and
engagement but also that the fact that they were not equally
distributed over time may have rendered the second half of the
course less appealing.
C. Attendance and Grades
The gamified approach also seems to have had a positive
impact on lecture attendance. Average attendance by lecture for
both the first and second academic years were 81% and 92%,
which represents a significant difference of 11% (Mann-
Whitney's U test, U = 51, Z = -3.654, p < 0,001). However, we
were not able to verify the same for invited lectures, whose
average attendance decreased by 14% (67% vs. 53%), even
though this result was not statistically significant. This lower
value might be related to students deeming invited lectures to
be less worthy of their time, either because they are not
relevant to exams or because they would occur on a single
campus, which posed mobility issues. As for student grades,
there were no statistically significant changes in the grade
Fig. 8. Post distribution in the second year
Fig. 9. Total posts over time in the second year
Fig. 10. Posts over time in the first year
Fig. 11. Comparison of the number of posts made by students between the
two years

average. Additional studies are needed to identify possible
correlations between gamification and student grades.
D. Student Feedback
By the end of the semester, we carried out a questionnaire
to gather quantitative and qualitative feedback about the
gamified experience. Students first had to rate a set of phrases
using a five-point Likert scale. Taking the mode of the answer
into account, students considered that the gamification
experiment applied to the MCP performed very well (4) [1 –
terrible; 5 – excellent]. When comparing to other courses, they
considered the MCP course to be much more motivating (5)
and interesting (4) [1-much less; 5 - much more]. They also
considered that the course required more work (4) but was
neither more difficult (3) nor harder to learn from (3) [1-much
less; 5 - much more]. They considered the study to have the
same quality (3) of other courses, but with a greater continuity
(4) [1 - far less; 5 - far more]. Students mildly felt that they
were playing a game instead of just attending a regular course
(3) [1 - not at all; 5 - a lot] and they had not a clear opinion on
whether achievements should account for more of the course
grade or not (3) [1-definitely not; 5 - definitely yes].
Furthermore, students considered that they performed non-
mandatory tasks more for the sake of the game than for their
grade (4) [1-grade only; 5-game only], which suggests a deeper
engagement. Also, students deemed achievements that required
extra work, such as “Class Annotator” and Challenges, to have
contributed to their learning (4) [1-not at all; 5 - definitely] and
agreed that gamification should be extended to other courses
(5) [1-definitely not; 5 - definitely yes]. They also suggested
that the game illusion could be improved by adding usable
items and avatars, by establishing an achievement tree, and by
promoting direct competition among learners. Last, group
achievements could also be added, so that the whole class
could work together, and oral participation in class should also
be rewarded.
We also asked students to rate achievements according to
their effectiveness. Highest rated were “Proficient Tool User”,
which allowed them to understand topics taught in the labs by
producing creative content with multimedia tools; “Rise to the
Challenge”, where they had to complete theoretical challenges,
which helped them understand subjects from lectures; and
“Amphitheatre Lover”, that encouraged them to attend lectures.
They rated as least effective “Post Master”, that rewarded then
for the number of posts, because “quantity does not equal
quality”, and students noticed that a string of weak posts would
get more XP than fewer posts of more thought out messages;
“Bug Squasher”, since fixing bugs in PCM Media Mixer
required considerable coding, not commensurate with the
reward; and the “Attentive Student”, because finding errors on
slides was a tedious task and they did not feel to be learning,
even if they had to carefully read class materials in order to
find errors. There were, however, a few achievements about
which students had mixed feelings, such as “Bookworm”, that
encouraged them to do something that they already had to do,
like reading the slides (the number of downloads increased
39.9%, nonetheless); “Class Annotator”, that rewarded them
for finding resources related to lectures, because some of the
posted links were deemed irrelevant and many students did not
like homework, although some said that it helped them to think
out-of-the-box; and “Challenger of the Unknown”, where they
had to submit contributions to the online quests forum, as while
some said it provided them with a game-like feeling, others
found it uninteresting and too similar to “Class Annotator”.
The original idea of both was very different, so this shows they
may have been improperly described to the students.
V. DISCUSSION
The results from the experiment are very encouraging.
They show notable gains in terms of attendance, participation
and material downloads, which suggests improved engagement
and diligence. In this section we discuss how many aspects of
the students’ experience were affected by the gamified
installment of the MCP course, as well as the design
implications for future gamified learning experiences.
A. Engagement, Satisfaction and Performance
The average attendance for regular lectures significantly
increased 11% from the first to the second year, which suggests
that students had greater interest in attending classes. The
number of posts made by students presented the largest growth
with 511% more replies in the second year and 845% more
initiated threads. This suggests not only more participation but
also a massive increase in proactivity, which reflects a greater
willingness to engage in discussions. It is important to note that
the amount of posts in the second year is still 281% larger than
the first year if not taking into account the new evaluation
component. Furthermore, 47% of the total posts in the second
year related to challenges students had to complete, which
would award them with XP and badges. This suggests that
challenges are indeed both good engagers and behavior drivers,
as supported by recent studies (Chen, 2012), but also that the
desire to collect may also be a powerful motivator.
The significant increases in the number of downloads, posts
and attendance indicate that students were more engaged with
this new gamified learning experience than they were with the
previous non-gamified one. Students were generally satisfied
and found the MCP course to be more motivating and
interesting than other courses, even though it required more
work. This is an interesting finding. Indeed, students often
complain about heavy workloads. However, using
gamification, they did not mind spending more time working
for the course and were more satisfied. While this suggests that
students were motivated to work, it is not clear whether this
motivation was extrinsic or intrinsic. This is a topic for future
research. Even though we have no evidence that their marks
were significantly affected by the game-like experience, we
know that they spent more time working and have enjoyed
MCP more than other courses. Notwithstanding, it would be of
interest to identify a correlation between student grades and
gamified learning.
B. Design Implications
We learned that challenges have to be carefully crafted in
order to add meaning and interest to a gamified learning
experience. Students preferred challenges like “Proficient Tool
User”, “Rise to the Challenge” and “Amphitheatre Lover”,

whose main goal was to encourage them to attend lectures,
explore topics taught on lectures and, above all, to create and
be expressive, which is something that is not usually promoted
in traditional engineering courses. These goals were not only in
line with their personal objective of passing the course, but
they were also creative and fun, which is why students made
multiple replies to some challenges when only one would count
for grading. This might have rendered challenges more
meaningful, which as seen in the literature, is of major
importance [33]. It also explains why other challenges, such as
the “attentive student” (find typos in class material) were less
popular, as they were perceived as meaningless.
While points, levels, multi-level achievements seem to have
worked well to transmit competence and mastery to students, it
was much harder to convey autonomy due to the course
evaluation constraints. However, students had the freedom to
be creative in challenges and to choose tasks and achievements
to pursue, and which were worth leveling up or not. As
suggested by students, this could be further improved by
adding achievement trees, in which some achievements would
only be unlocked once the preceding ones were accomplished.
We also found that the balance between how hard and how
gratifying a challenge is, is also important to keep students
engaged, which can be related to the flow theory [13, 14]. This
explains why the “Bug Squasher” achievement was the least
popular. Fixing source code bugs and recompiling the course’s
prototype for only 40 extra XP was not worth the effort.
Furthermore, we found that challenges should be spread
throughout the term to avoid periods where appealing goals
may lack, or else students will become bored and demotivated.
Students suggested that the gaming experience could be
improved by adding items with interesting effects, achievement
trees and by representing each player with a customizable
Avatar, which would allow students to develop online identity,
reputation [28] and become more committed. It was also
suggested that there should be more direct competition among
students, such as rewarding those that accomplish an
achievement before anyone else. However, students also saw
room for cooperation, and suggested adding group
achievements, like rewarding everybody if every student
reached at least 80% of the maximum grade. This could yield
interesting effects in the whole class dynamics.
We acknowledge that there are additional improvements
that could be made to this gamified setup to further engage
students, such as rewarding oral participation and giving more
experience to higher achievement levels, as the current setup
might make top levels less appealing. The interface could also
issue notifications to transmit progress, and the leaderboard
could allow students to perform direct comparisons with
others. Student posts’ quality should also be accounted for
grading, to avoid pointless posts and promote fairness.
C. Study Limitations
Our study has three major limitations. Firstly, there is the
controversial problem of over-justification with extrinsic
rewards. Even though students are rewarded with points and
badges, these serve the main purpose of transmit progress and
positive feedback, and we avoid giving them too much
emphasis. We try to counter this issue by providing meaningful
challenges. Other game element setups could be experimented
to assess this effect.
The second limitation concerns the competitive nature of
our gamified experience. This was early decided due to the
nature of our college and student population, which tends to be
competitive. Actually, students suggested that additional
competitive features should be added to the experience.
Thirdly, student engagement was assessed using informal
measures, but a more formal evaluation, such as that proposed
by Handelsman et al. [37], should be performed in order to
have a richer understanding on this subject.
VI. CONCLUSIONS
Gamification is a novel technique that applies game
elements to non-game contexts, to engage users and solve
problems. In our experiment, we applied this to a traditional
MSc course by adding game elements including points, levels,
leaderboards, challenges and badges. While the first two serve
the main purpose of displaying progress and providing
feedback, challenges and badges give students the autonomy to
pursue different goals. On the other hand, the leaderboard
together with the course forums promoted relatedness and
competition. We compared many aspects of the learning
experience between a non-gamified and the gamified version of
the course, on two consecutive years, and assessed their impact
on student performance and satisfaction. Results were very
positive, showing significant improvements in lecture
attendance, which is optional, number of downloads of lecture
slides, and number of posts on the course’s forums. This
suggests that students paid more attention to support materials,
that they had more interest in lectures and that they were more
engaged in the course. This engagement is particularly
noticeable by the increase in post activity. Not only did
students gave 511% more replies to other posts, but they also
initiated 845% more threads, which denotes remarkable
improvements both in participation and proactivity.
Challenges proved particularly effective in encouraging
users to participate in forums and perform tasks they would
typically avoid, like reading slides and suggesting reference
materials. However, these have to be carefully tailored in order
to be meaningful or they will be ignored. It is also important to
assure that challenges are evenly spread throughout the term
and are not concentrated in a particular period. This causes
students to expect many instant gratifications and become
bored when they become scarce. Creative tasks and multi-level
achievements can boost student autonomy and render the
whole experience more fun and compelling.
When compared to other courses, students found MCP to
be more motivating and interesting, although they have
admitted that it required more work. This suggests that students
will feel more engaged with courses using gamification, even if
that translates to a higher workload. Here, gamification, and
our approach in particular, stand out by their potential to
encourage students to become more diligent and dedicated to
the course. However, we are yet to find a correlation between
gamification and student grades, which is an interesting topic
for future research.

ACKNOWLEDGMENT
This work was supported by FCT (INESC-ID multiannual
funding) under project PEst-OE/EEI/LA0021/2013 and the
project PAELife, reference AAL/0014/2009. Gabriel Barata
was supported by FCT, grant SFRH/BD/72735/2010.
REFERENCES
[1] K. D. Squire, “Video games in education,” International Journal of
Intelligent Games & Simulation, vol. 2, no. 1, pp. 49–62, 2003.
[2] M. de Aguilera and A. Mendiz, “Video games and education: (education
in the face of a "parallel school"),” Computers in Entertainment, vol. 1,
no. 1, pp. 1:1–1:10, October 2003.
[3] M. Prensky, “Digital natives, digital immigrants part 1,” On the horizon,
vol. 9, no. 5, pp. 1–6, 2001.
[4] H. F. O’Neil, R. Wainess, and E. L. Baker, “Classification of learning
outcomes: evidence from the computer games literature,” Curriculum
Journal, vol. 16, no. 4, pp. 455–474, 2005.
[5] S. Bennett, K. Maton, and L. Kervin, “The ‘digital natives’ debate: A
critical review of the evidence,” British Journal of Educational
Technology, vol. 39, no. 5, pp. 775–786, 2008.
[6] P. Mcclean, B. Saini-eidukat, D. Schwert, B. Slator, and A. White,
“Virtual worlds in large enrollment science classes significantly improve
authentic learning,” in Proceedings of the 12th International Conference
on College Teaching and Learning, Center for the Advancement of
Teaching and Learning, 2001, pp. 111–118.
[7] K. Squire, M. Barnett, J. M. Grant, and T. Higginbotham,
“Electromagnetism supercharged!: learning physics with digital
simulation games,” in Proceedings of the 6th international conference
on Learning sciences, ser. ICLS ’04. International Society of the
Learning Sciences, 2004, pp. 513–520.
[8] J. Lee, K. Luchini, B. Michael, C. Norris, and E. Soloway, “More than
just fun and games: assessing the value of educational video games in
the classroom,” in CHI ’04 Extended Abstracts on Human Factors in
Computing Systems, ser. CHI EA ’04. New York, NY, USA: AC M,
2004, pp. 1375–1378.
[9] M. Kebritchi, A. Hirumi, and H. Bai, “The effects of modern math
computer games on learners’ math achievement and math course
motivation in a public high school setting,” British Journal of
Educational Technology, vol. 38, no. 2, pp. 49–259, 2008.
[10] B. Coller and D. Shernoff, “Video game-based education in mechanical
engineering: A look at student engagement,” International Journal of
Engineering Education, vol. 25, no. 2, pp. 308–317, 2009.
[11] J. Moreno, “Digital competition game to improve programming skills,”
Educational Technology & Society, vol. 15, no. 3, pp. 288–297, 2012.
[12] J. P. Gee, “What video games have to teach us about learning and
literacy,” Comput. Entertain., vol. 1, no. 1, pp. 20–20, Oct. 2003.
[13] J. Chen, “Flow in games (and everything else),” Commun. ACM, vol. 50,
pp. 31–34, 2007.
[14] M. Csikszentmihalyi, Flow: The psychology of optimal experience.
Harper Perennial, 1991.
[15] S. Deterding, D. Dixon, R. Khaled, and L. Nacke, “From game design
elements to gamefulness: defining “gamification”,” in Proceedings of
the 15th International Academic MindTrek Conference Envisioning
Future Media Environments, vol. Tampere, F. ACM, 2011, pp. 9–15.
[16] S. Deterding, M. Sicart, L. Nacke, K. O’Hara, and D. Dixon,
“Gamification. using game-design elements in non-gaming contexts,” in
Proceedings of the 2011 annual conference extended abstracts on
Human factors in computing systems, ser. CHI EA ’11. New York, NY,
USA: ACM, 2011, pp. 2425–2428.
[17] B. Shneiderman, “Designing for fun: how can we design user interfaces
to be more fun?” interactions, vol. 11, no. 5, pp. 48–50, 2004.
[18] B. Reeves and J. Read, Total Engagement: How Games and Virtual
Worlds Are Changing the Way People Work and Businesses Compete.
Harvard Business Press, 2009.
[19] G. Zichermann and J. Linder, Game-based marketing: inspire customer
loyalty through rewards, challenges, and contests. Wiley, 2010.
[20] G. Zichermann and C. Cunningham, Gamification by Design:
Implementing Game Mechanics in Web and Mobile Apps. O’Reilly
Media, Inc., 2011.
[21] S. Sheth, J. Bell, and G. Kaiser, “Halo (highly addictive, socially
optimized) software engineering,” in Proceeding of the 1st international
workshop on Games and software engineering, ser. GAS, vol. 11, 2011,
pp. 29–32.
[22] O. Inbar, N. Tractinsky, O. Tsimhoni, and T. Seder, “Driving the
scoreboard: Motivating eco-driving through in-car gaming,” in
Proceedings of the CHI 2011 Workshop Gamification: Using Game
Design Elements in Non-Game Contexts. ACM, 2011.
[23] T. Dong, M. Dontcheva, D. Joseph, K. Karahalios, M. Newman, and
M. Ackerman, “Discovery-based games for learning software,” in
Proceedings of the 2012 ACM annual conference on Human Factors in
Computing Systems, ser. CHI ’12. New York, NY, USA: ACM, 2012,
pp. 2083–2086.
[24] W. Li, T. Grossman, and G. Fitzmaurice, “Gamicad: a gamified tutorial
system for first time autocad users,” in Proceedings of the 25th annual
ACM symposium on User interface software and technology, ser. UIST
’12. New York, NY, USA: ACM, 2012, pp. 103–112.
[25] L. Sheldon, The Multiplayer Classroom: Designing Coursework as a
Game. Course Technology PTR, 2011.
[26] C. Thompson, “How khan academy is changing the rules of education,”
Wired Magazine, pp. 1–5, 2011.
[27] A. J. Kim. (2008, March) Putting the fun in functional.
http://www.slideshare.net/amyjokim/putting-the-fun-in-functiona.
[28] C. Crumlish and E. Malone, Designing social interfaces. O’Reilly, 2009.
[29] K. Werbach and D. Hunter, For the Win: How Game Thinking Can
Revolutionize Your Business. Wharton Digital Press, 2012.
[30] L. Natvig, S. Line, and A. Djupdal, “"age of computers"; an innovative
combination of history and computer game elements for teaching
computer fundamentals,” in In proceedings of the 34th Annual Frontiers
in Education conference, ser. FIE 2004, vol. 3, 2004, pp. S2F – 1–6.
[31] A. Kohn, “Studies find reward often no motivator,” Boston Globe,
vol. 19, pp. 52–59, 1987.
[32] S. Rigb y and R. Ryan , Glued to games: How video games draw us in
and hold us spellbound. Praeger, 2011.
[33] E. Deci and R. Ryan, Handbook of self-determination research.
University of Rochester Press, 2004.
[34] S. Deterding, “Gamification: designing for motivation,” interactions,
vol. 19, no. 4, pp. 14–17, Jul. 2012.
[35] C. Linehan, B. Kirman, S. Lawson, and G. Chan, “Practical, appropriate,
empirically-validated guidelines for designing educational games,” in
Proceedings of the SIGCHI Conference on Human Factors in
Computing Systems, ser. CHI ’11. New York, NY, USA: ACM, 2011,
pp. 1979–1988.
[36] B. J. Mandernach, E. Donnelli-Sallee, and A. Dailey-Hebert, “Assessing
course student engagement,” Promoting Student Engagement, vol. 1,
2011.
[37] M. M. Handelsman, W. L. Briggs, N. Sullivan, and A. Towler, “A
measure of college student course engagement,” The Journal of
Educational Research, vol. 98, no. 3, pp. 184–192, 2005