ArticlePDF Available

Abstract and Figures

Gamification is increasingly employed in learning environments as a way to increase student motivation and consequent learning outcomes. However, while the research on the effectiveness of gamification in the context of education has been growing, there are blind spots regarding which types of gamification may be suitable for different educational contexts. This study investigates the effects of the challenge-based gamification on learning in the area of statistics education. We developed a gamification approach, called Horses for Courses, which is composed of main game design patterns related to the challenge-based gamification; points, levels, challenges and a leaderboard. Having conducted a 2 (read: yes vs. no) x 2 (gamification: yes vs. no) between-subject experiment, we present a quantitative analysis of the performance of 365 students from two different academic majors: Electrical and Computer Engineering (n=279), and Business Administration (n=86). The results of our experiments show that the challenge-based gamification had a positive impact on student learning compared to traditional teaching methods (compared to having no treatment and treatment involving reading exercises). The effect was larger for females and for students at the School of Electrical and Computer Engineering.
Content may be subject to copyright.
Contents lists available at ScienceDirect
International Journal of Human-Computer Studies
journal homepage:
The eect of challenge-based gamication on learning: An experiment in the
context of statistics education
Nikoletta-Zampeta Legaki
, Nannan Xi
, Juho Hamari
, Kostas Karpouzis
Vassilios Assimakopoulos
School of Electrical and Computer Engineering, National Technical University of Athens, Zografou 15780, Greece
Gamication Group, Faculty of Information Technology and Communication Sciences, Tampere University, Tampere 33100, Finland
Applications in education
Statistics education
Teaching forecasting
Human-Computer interface
Gamication is increasingly employed in learning environments as a way to increase student motivation and
consequent learning outcomes. However, while the research on the eectiveness of gamication in the context of
education has been growing, there are blind spots regarding which types of gamication may be suitable for
dierent educational contexts. This study investigates the eects of the challenge-based gamication on learning
in the area of statistics education. We developed a gamication approach, called Horses for Courses, which is
composed of main game design patterns related to the challenge-based gamication; points, levels, challenges
and a leaderboard. Having conducted a 2 (read: yes vs. no) x 2 (gamication: yes vs. no) between-subject
experiment, we present a quantitative analysis of the performance of 365 students from two dierent academic
majors: Electrical and Computer Engineering (n=279), and Business Administration (n=86). The results of our
experiments show that the challenge-based gamication had a positive impact on student learning compared to
traditional teaching methods (compared to having no treatment and treatment involving reading exercises). The
eect was larger for females or for students at the School of Electrical and Computer Engineering.
1. Introduction
Gamication approaches are being applied with increasing fre-
quency in an attempt to positively aect behavior and cognitive pro-
cesses by enhancing the system or service with motivational aor-
dances and eventually by bringing similar experiences as games
do (Huotari and Hamari, 2017). Motivational aordances have been
widely used in many elds such as business (Alcivar and Abad, 2016; Xi
and Hamari, 2020), crowdsourcing (Morschheuser et al., 2017),
healthcare (Johnson et al., 2016) and education (Dichev and Dicheva,
2017; Hanus and Fox, 2015; Koivisto and Hamari, 2019; Majuri et al.,
2018; Osatuyi et al., 2018; Seaborn and Fels, 2015). Additionally, ga-
mication has been employed in many education related contexts,
across dierent educational levels (Caponetto et al., 2014; Dicheva
et al., 2015; Simões et al., 2013; de Sousa Borges et al., 2014) and in
various subjects (Dichev and Dicheva, 2017; Dicheva et al., 2015;
Kasurinen and Knutas, 2018; Seaborn and Fels, 2015), showing its
potential to improve learning outcomes (Koivisto and Hamari, 2019;
Seaborn and Fels, 2015).
According to reviews of gamication literature, gamication has
been employed mostly in the eld of education (Koivisto and Hamari,
2019; Majuri et al., 2018; Seaborn and Fels, 2015). Gamied educa-
tional applications have been applied in non-academic areas as well:
language teaching (Duolingo counts 300 million active users
) or soft-
ware using (Ribbon Hero by Microsoft). Other popular gamied ap-
plications are: Kahoot and Quizizz, which can be easily congured and
used in a variety of subjects, bringing game elements into classrooms
without any special eort. Although gamication has an important
position in education both inside and outside universities, there is still
little eective guidance on how to combine dierent gamication fea-
tures to enhance learning performance in dierent educational
contexts (Hanus and Fox, 2015; Koivisto and Hamari, 2019; Seaborn
and Fels, 2015).
Beyond research problems pertaining to the general interest in ga-
mication and its eect on education, statistics education is an in-
creasingly fundamental skill to understand the world around us. The
lack of data literacy has been deemed one of the main causes behind our
inability to act against climate change, to properly ratify means towards
e.g. COVID-19 or generally as a hindrance for public understanding of
science. Therefore, there is a need to make teaching methods in
Received 12 November 2019; Received in revised form 9 June 2020; Accepted 12 June 2020
Corresponding author.
E-mail addresses: zampeta.legaki@tuni.,, (N.-Z. Legaki).
International Journal of Human-Computer Studies 144 (2020) 102496
Available online 14 June 2020
1071-5819/ © 2020 The Authors. Published by Elsevier Ltd. This is an open access article under the CC BY-NC-ND license
statistics and forecasting more engaging (Love and Hildebrand, 2002).
The acceleration of the daily data production, and ever-greater capacity
to store and process this information, has boosted the necessity for
students with a strong background in statistics and predictive analytics
skills, in business environments or even in everyday life. Consequently,
both statistics and forecasting techniques are of vital importance in the
economics curriculum (Loomis and Cox Jr, 2003) and in other elds
such as business (Makridakis et al., 2008) or social problems, where
data may help to make better decisions. However, often forecasting
courses are not even oered as an independent course in business
schools (Hanke, 1989) and when they are, students are discouraged to
participate in the courses because they nd the topic too
complicated (Albritton and McMullen, 2006; Gardner, 2008; Snider and
Eliasson, 2013; Torres et al., 2018) and demanding (Craighead, 2004).
Therefore, regarding the education and especially the education in the
eld of statistics and forecasting, student motivation is crucial for their
participation and understanding in order to reach their learning po-
tential, meet business needs and get insights of the data to support the
decision-making process.
Despite the long tradition in educational business games, thus far
there have only been a few studies on gamication or simple gamied
exercises, combined with traditional teaching methods in the area of
statistical forecasting. These studies have mostly used: score
(Craighead, 2004), spreadsheets (Gardner, 2008),
competition (Snider and Eliasson, 2013) and real-world forecasting
problems (Buckley and Doyle, 2016a; Gavirneni, 2008) in order to
encourage students' participation, without examining gamication ef-
fects. Other more quantitative studies have used forecasting in the
context of a prediction market as a tool to motivate students rather than
to teach forecasting aspects (Buckley and Doyle, 2016a; 2016b; 2017;
Buckley et al., 2011). While there are several types of games and ga-
mication designs, the challenge-based gamication (e.g points, levels,
leaderboard, clear goals/ tasks), as opposed to the immersion- and the
social-based gamication, has been suggested (Dicheva et al., 2015;
de Sousa Borges et al., 2014; Zichermann and Cunningham, 2011) and
applied to a high degree in practice as gamication design in
education (Koivisto and Hamari, 2019; Seaborn and Fels, 2015;
de Sousa Borges et al., 2014). One or more of these gamication ele-
ments have been used with promising results even in educational topics
relative to forecasting (Craighead, 2004; Gavirneni, 2008; Gel et al.,
2014; Snider and Eliasson, 2013). However, there is still a lack of ef-
fective design guides and empirical data on the combination or in-
tegration of these features in the context of educational information
systems (Koivisto and Hamari, 2019). Challenge-based gamication
introduces a design approach of integrating achievement gamication
features, positively related with intrinsic need satisfaction (Xi and
Hamari, 2019), in an educational service or application, in order to
explore its potential, motivate users and eventually improve learning.
The present study examines the impact of three treatments on stu-
dentsperformance i) reading, ii) use of a challenge-based gamied
application, iii) the combination of the two. In order to do that, we
consider a variety of student characteristics such as gender, level of
studies, academic major, expertise in the English language, and use of
personal computers and games. We designed and implemented a web-
based gamied application, called Horses for Courses and we conducted
a series of experiments over the last 4 years. The total sample is com-
posed of 365 students, with 279 undergraduate and MBA students at
the School of Electrical and Computer Engineering of the National
Technical University of Athens, Greece (hereafter ECE, NTUA) and
86 undergraduate students at the Business Administration Department
in the School of Business and Economics of the University of Thessaly,
Greece (hereafter Business Administration). Our ndings show that
challenge-based gamication improves studentslearning outcomes on
a statistics course, contributing to the knowledge of challenge-based
gamicationseect on statistics/stem education and eventually on
gamied pedagogy.
2. Background
2.1. Gamication in education
Gamication refers to a method of designing systems, services, or-
ganizations and activities in order to create similar experiences and
motivations to those experienced when playing games, with the added
educational goal of aecting user behavior (Huotari and
Hamari, 2017). Games are known to motivate and engage
players (Dichev and Dicheva, 2017) because of the enjoyment and the
excitement that this activity oers (Koivisto and Hamari, 2019). In this
regard, gamication aspires to create this experience in dierent con-
texts. This is usually attempted by using game mechanics or other
game-like designs in the target environment (Deterding et al., 2011).
Over the last decade, gamication research has aected a variety of
domains that deal with education (Koivisto and Hamari, 2019). The
educational domain is continuously evolving, incorporating the latest
developments in information technology even in elementary
schools (Karpouzis et al., 2007). Nonetheless still demands students'
commitment and persistence in order for them to gain in-depth
knowledge. Consequently, gamication has been of great interest to
educators who have been exploring its potential in improving student
learning (Dichev and Dicheva, 2017; Dicheva et al., 2015; Hamari,
2013; Koivisto and Hamari, 2019; Majuri et al., 2018; Seaborn and Fels,
2015). This potential has led to a growing literature on the eectiveness
of gamication, mainly in universities but also in other academic
contexts (Caponetto et al., 2014; Koivisto and Hamari, 2019; Seaborn
and Fels, 2015; de Sousa Borges et al., 2014) and in a variety of
subjects (Dichev and Dicheva, 2017; Kasurinen and Knutas, 2018). To
name a few: information technology (Osatuyi et al., 2018), math/
science (Attali and Arieli-Attali, 2015; Christy and Fox, 2014) and
taxation (Buckley and Doyle, 2016a; 2017).
Gamication types and types of game design have commonly been
horizontally categorized into main three overarching categories of
achievement/challenge-, immersion-, and social-based (Hamari and
Tuunanen, 2014; Koivisto and Hamari, 2019; Snodgrass et al., 2013; Xi
and Hamari, 2019; Yee, 2006; Yee et al., 2012) beyond the common
vertical categorization of e.g. the MDA model that separated game
design into mechanics, dynamics and aesthetics (Hunicke et al., 2004).
The immersion-based game design attempts primarily to engulf the
player or user into a story, roleplay and audiovisual richness. The so-
cial-based game design is commonly focused on dierent forms of
competition and collaboration. Finally, the achievement/challenge-
based game design is focused on overcoming challenges, progressing
and earning rewards and feeling competent. Within the achievement/
challenge-based gamication, the most commonly embodied mechanics
have been points, challenges, leaderboards, levels and badges (Koivisto
and Hamari, 2019; Majuri et al., 2018; Pedreira et al., 2015). According
to the self-determination theory, the use of these elements, which are
considered as achievement related features and immediate performance
indicators, is associated with intrinsic motivation for students (Xi and
Hamari, 2019). In this regard, these elements form challenge-based
gamication underpinnings in order to motivate students to maximize
their knowledge acquisition.
Review studies about the eectiveness of gamication are generally
optimistic, mainly listing either positive or mixed results of applied
gamied strategies (Buckley and Doyle, 2017; Caponetto et al., 2014;
Dicheva et al., 2015; Koivisto and Hamari, 2019; Lambruschini and
Pizarro, 2015; Majuri et al., 2018; Nah et al., 2014; Osatuyi et al., 2018;
Reiners et al., 2012; Seaborn and Fels, 2015; de Sousa Borges et al.,
2014). Nevertheless, they mention the need for more controlled ex-
perimental research on the impact of gamication, independently of the
application domain or used gamied strategy (Buckley and Doyle,
2017; Caponetto et al., 2014; Dichev and Dicheva, 2017; Dicheva et al.,
2015; Hanus and Fox, 2015; Koivisto and Hamari, 2019; Lambruschini
and Pizarro, 2015; Landers et al., 2018; Majuri et al., 2018; Nah et al.,
N.-Z. Legaki, et al. International Journal of Human-Computer Studies 144 (2020) 102496
2014; Osatuyi et al., 2018; Reiners et al., 2012; Seaborn and Fels, 2015;
de Sousa Borges et al., 2014).
The eects of gamication are bound together with the target au-
dience and the context (Buckley and Doyle, 2017; Dichev and Dicheva,
2017; Hanus and Fox, 2015; Koivisto and Hamari, 2019; Seaborn and
Fels, 2015). Hence, the results of gamication vary regarding the sub-
ject and the eld of application (Hanus and Fox, 2015; Sánchez-Martín
et al., 2017). Therefore, researchers generally agree on the need for
stronger empirical results (Buckley and Doyle, 2017; Hanus and Fox,
2015; Koivisto and Hamari, 2019; Landers et al., 2018; Maican et al.,
2016; Morschheuser et al., 2017). This study contributes to this body of
research with empirical data drawn from a series of experiments on the
impact of gamication with control and treatment groups in the context
of a forecasting course.
2.2. Teaching forecasting in higher education
As described in Gareld and Ben-Zvi (2007), statistics and statistical
literacy are of paramount importance, especially in the rapidly chan-
ging business environments. As interest in the available technology and
statistics is growing, forecasting skills are becoming more
sophisticated (Kros and Rowe, 2016) and the process of teaching fore-
casting is becoming more dicult and demanding. Statistics courses
focus on data analysis (Cobb, 1992), as competitive business environ-
ments require graduate students to interpret data and be able to use
statistical and judgmental forecasting methods and
applications (Giullian et al., 2000; Kros and Rowe, 2016). The im-
portance of forecasting skills is not a new discovery (Albritton and
McMullen, 2006; Craighead, 2004; Giullian et al., 2000; Kros and
Rowe, 2016; Loomis and Cox Jr, 2003; Makridakis et al., 2008; Snider
and Eliasson, 2013). However, recently these skills have have become
even more important since business decision-making must be supported
by data-based evidence and projections (Giullian et al., 2000). Another
aspect of forecasting that highlights its importance is its multi-
disciplinary nature, since the forecasting techniques are an essential
component in a number of elds such as business
statistics (Tabatabai and Gamble, 1997), supply chain
management (Gavirneni, 2008) and management
science (Makridakis et al., 2008).
However, the eagerness of the business sector to equip students with
a strong background in forecasting techniques is only partially reected
in the education that universities and business schools provide. Thirty-
ve years ago, 58% of the surveyed universities oered an independent
forecasting course (Hanke, 1984). The percentage is reduced to 34.48%
of the surveyed business schools based on a more recent study
by Kros and Rowe (2016) and it is almost the same (50%) regarding the
top 50 US Business Programs, which requires a forecasting time-series
course. Moreover, there is a variety of e-learning-in-statistics in-
itiatives, but even these modules do not focus on time-series and
forecasting methodologies (Gel et al., 2014).
Despite the growing popularity of and need for forecasting skills,
business schools slowly address this demand, and they generally dis-
regard the need for increasing student motivation (Debnath et al.,
2007). Business forecasting or statistical forecasting methods are
usually considered complicated (Albritton and McMullen, 2006;
Craighead, 2004; Gardner, 2008; Snider and Eliasson, 2013; Torres
et al., 2018), making it dicult for students to remain
motivated (Craighead, 2004). Taking this into account, Chu (2007);
Donihue (1995);Loomis and Cox (2000);Loomis and Cox Jr (2003);
McEwen (1994) suggest alternative teaching guidelines such as the use
of a software or new technology in combination with real data and
forecasting problems. Active learning has also been proposed (Love and
Hildebrand, 2002) in order to address the need to update the fore-
casting educational process. So the digitalization, which we experience
has boosted the statistical skillsimportance. However, universities and
business schools have not responded immediately to this challenge and
they have been criticized for not placing enough focus on the specic
skills that will improve the studentsfuture job
performance (McEwen, 1994) and career success (Pfeer and
Fong, 2002).
2.3. Gamication and teaching statistical forecasting
This study puts emphasis on gamication only in terms of simple
educational activities or systems, usually including game
mechanics (Bunchball, 2010; Deterding et al., 2011). In this direction,
we reviewed journal articles that discuss simple active learning events,
gamied exercises or games in the context of a forecasting course. Our
results indicate that the use of score (Craighead, 2004),
spreadsheets (Gardner, 2008) and competition (Snider and
Eliasson, 2013) during lectures has positive eects regarding students
attitude, but strong empirical data is not presented. The use of a cus-
tomized software (Spithourakis et al., 2015) and studentsparticipation
in prediction of a basketball score appeared benecial in the context of
an undergraduate forecasting techniques course.
Other active learning exercises have used competition based on
studentsforecasting accuracy in order to increase studentsparticipa-
tion and improve learning outcomes in a management
course (Buckley et al., 2011) and in a taxation course (Buckley and
Doyle, 2016a). Another simple game, named: FREDCAST has been de-
signed and recently used in order to teach forecasting in a macro-
economic course (Mendez-Carbajo, 2018). All of these examples, show
that forecasting due to its nature can be considered as a kind of an
artistic eld (Gavirneni, 2008), where gamication could be eciently
integrated in order to make it attractive to its audience.
Our preliminary review mainly positions gamication as a bene-
cial tool in education of forecasting and related elds such as man-
agement (Buckley et al., 2011; Makridakis et al., 2008), decision-
making (Makridakis et al., 2008), taxation (Buckley and Doyle, 2016b),
supply chain management (Gavirneni, 2008) but highlights the need for
more empirical results. Additionally, an overview of teaching fore-
casting shows the importance of forecasting courses in economics
syllabus (Loomis and Cox Jr, 2003) or business school
curriculum (Buckley et al., 2011; Gavirneni, 2008). The need for fore-
casting skills is increasing as well, due to technological changes, as
management seeks data-based approaches in dealing with decision-
making on market opportunities, environmental factors and technolo-
gical resources. However, forecasting courses are not adequately sup-
ported by students' participation or university and business school
programming (Albritton and McMullen, 2006; Snider and Eliasson,
2013). A possible approach to address this issue could be gamication,
which under proper design guidelines has produced promising results
regarding student motivation and learning outcomes in management
courses (Buckley and Doyle, 2016a; Craighead, 2004; Gardner, 2008;
Snider and Eliasson, 2013) and in a forecasting
module (Gavirneni, 2008). Nevertheless, thus far, there have not been a
lot of studies on the eects of gamication on learning outcomes, in the
specic area of statistical forecasting. Therefore, this study experi-
mentally examines the potential of the challenge-based gamication, by
designing from scratch and using a gamied application in order to
improve student learning in a forecasting course.
3. Material and methods
3.1. Participants
A series of experiments were conducted at the ECE, NTUA and at the
Business Administration. More precisely, we performed the experiments
in dierent classes and academic majors, as follows:
49 undergraduate students (class of 2015) at the ECE, NTUA.
37 MBA students (class of 2015) at the ECE, NTUA.
N.-Z. Legaki, et al. International Journal of Human-Computer Studies 144 (2020) 102496
60 undergraduate students (class of 2016) at the ECE, NTUA.
52 undergraduate students (class of 2018) at the ECE, NTUA.
21 MBA students (class of 2018) at the ECE, NTUA.
86 undergraduate students (class of 2018) at the Business
60 undergraduate students (class of 2019) at the ECE, NTUA.
The total sample is composed of 365 students; 270 students are
males and 95 are females. The experiments were performed in the
context of a forecasting course, with fourth-year undergraduate stu-
dents and second-year MBA students, respectively, at the ECE, NTUA.
At the Business Administration, the experiment was conducted in the
context of an information technology course, with rst-year students.
However, the Business Administrations curriculum contains an opera-
tional research course, which includes forecasting techniques. The ex-
perimental design of our experiments was followed strictly, in-
dependently of the academic level of the students, as described in the
next section.
3.2. Experimental design
We conducted a 2 (read: yes vs. no) x 2 (gamication: yes vs no)
factorial experiment. The dependent variable was student performance
in the learning task. Participants were randomly assigned to one of the
conditions of the experiment: i) Group Control: no treatment, ii) Group
Read: treatment of reading a research paper, named thenceforth as task
Read (see 3.3.3), iii) Group Play: treatment of using challenge-based
gamication, named thenceforth as task Play (see 3.3.4) and iv) Group
Read&Play: both tasks: Readand Play. Time was controlled and
was equal to 15 minutes for each task. Table 1, depicts the design of the
evaluation of our experiment and all the treatments are explained in the
following section.
3.3. Materials
3.3.1. Lecture
The learning objectives of this lecture were for students to under-
stand and be able to apply the Method Selection Protocolsfor reg-
ular/fast-moving and intermittent demand time-series based on specic
academic work of Petropoulos et al. (2014). During the lecture, the aim
of the Petropoulos et al. (2014) research was mentioned, along with the
data, and the research methodology used. Special attention was paid on
the results, the practical implications and the conclusions of this study
about the Method Selection Protocols. More precisely, we further
explained the relation between the time-series features and one stra-
tegic decision and the forecasting accuracy of the proposed methods for
both regular/fast-moving and intermittent demand time-series. The
visual material of the lecture was composed of 17 slides and lasted 15
minutes. The lecture content was focused on specialized knowledge of
the research of Petropoulos et al. (2014) that both undergraduate and
MBA students in any of the stages of their studies would not have
otherwise been taught.
3.3.2. Final evaluation form
An evaluation form at the end was used to measure students'
learning performance via 30 close-ended questions (i.e. questions
where the participants would select the right answer among possible
answers) of equivalent grade about the ndings
of Petropoulos et al. (2014). This was the last task for all participants,
independently of the group to which they were assigned. The answers
to all the questions were covered in the lecture material. All the ques-
tions were about topics that have been discussed in the lecture de-
scribed at subsection 3.3.1, and therefore, in principle it would be
possible to attain the highest score by only participating in the lecture.
3.3.3. Task read
The material of the reading task was: Petropoulos, F., Makridakis,
S., Assimakopoulos, V., Nikolopoulos, K., 2014. horses for coursesin
demand forecasting. European Journal of Operational Research 237,
152 -163.. The paper is 12 pages and is the foundational material of
the lecture content. The students read the article using a computer in
the computer lab.
3.3.4. Task play: Challenge-based gamication
Since there is a lack of free, computationally non-complex gamied
applications, specically created to teach statistical forecasting, we
developed a gamication approach called Horses for Courses.Itisa
simple gamied application, which is composed of main design patterns
related to challenge-based gamication, as described in Section
Horses for Courses aims to motivate students' participation, to improve
their learning outcomes regarding the choice of simple but accurate
statistical forecasting methods, and consequently enhance their fore-
casting skills. It is structured based on the above-mentioned founda-
tional material: Petropoulos, F., Makridakis, S., Assimakopoulos, V.,
Nikolopoulos, K., 2014. horses for coursesin demand forecasting.
European Journal of Operational Research 237, 152 -163.. Horses for Courses Architecture
In order to implement Horses for Courses, we considered the methods
and design principles of both gamication (Morschheuser et al., 2018;
Zichermann and Cunningham, 2011) and software
development (Barnett et al., 2005; Gallaugher and Ramanathan, 1996;
Lewandowski, 1998). As far as the architecture of the application is
concerned, a focus on exibility, accessibility, high-level programming
and the ability to be integrated in dierent platforms led us to build a
web application on the Microsoft.NET framework (Barnett et al., 2005)
and to use an MS-SQL database. Horses for Courses is a web-based ga-
mied application, structured as a three-tier system with simple ap-
plication logic layer, which is fully accessible to registered users via a
browser. Users register with an email and a password in order to save
their progress in a database scheme, which serves as a data tier. A Data
Moduleis used to retrieve data from the database and to build it into
functional objects. A class named Actions, enables the interaction
between users and the Data Modulein order to save updated data and
the users progress back into the database, composing the logic tier of
the application. The presentation tier consists of a graphical user in-
terface, including time-series, data visualization and system functions.
A compact graphical representation of the Horses for Courses application
is found in Fig. 1. Horses for Courses Design
Guidelines for the design of the challenge-based gamication and
consequently of Horses for Courses application were divided into two
main directions: (1) the eective use of motivational aordances in
Table 1
Design of the evaluation of the experiments.
Task Description Group Control Group Read Group Play Group Read&Play
Attend Lecture (see 3.3.1)
Read the Paper (see 3.3.3)
Play (see 3.3.4)
Evaluation Form (see 3.3.2)
N.-Z. Legaki, et al. International Journal of Human-Computer Studies 144 (2020) 102496
learning environments (Deterding et al., 2011; Dicheva et al., 2015;
DomíNguez et al., 2013; González and Area, 2013; Hanus and Fox,
2015; Maican et al., 2016; Nah et al., 2014; Pedreira et al., 2015;
da Rocha Seixas et al., 2016; Yildirim, 2017) and (2) the design and
development of gamied applications (Kapp, 2013; Morschheuser et al.,
2018; Zichermann and Cunningham, 2011). The most frequently used
and assessed motivational aordances in education and in general, so
far are: points, levels, badges/achievements and
leaderboards (Alhammad and Moreno, 2018; Koivisto and Hamari,
2019; Majuri et al., 2018; Pedreira et al., 2015). Thus, we incorporated
points, a level setting, challenges and a leaderboard into our application
in order to maximize the external validity of the experiment - i.e. to
mimic a possible real world implementation of this gamication style.
More precisely, Table 2 describes the motivational aordances in Horses
for Courses, along with their denitions from the literature and the
purpose they serve.
Apart from these motivational aordances, our design decisions
regarding Horses for Courses were also determined by a desire to create
a user-friendly and agile interface and work ow, with clear player
guidance and instructions (Kapp, 2013). Fig. 2 describes a full round of
the game. Initially, students had to register or sign in. Later, for each
level they had to select the most suitable forecasting method based on
the provided data and information, as it is depicted in Fig. 3. Then
participants win points according to their choices. Instructions are ea-
sily accessible as well as theMethod Selection Protocols, through
colorful buttons as Fig. 3 illustrates, as well. New challenges arise at
each level, for example to identify time-series components for the real
data time-series as it is depicted in Fig. 4, encouraging the students to
assess their knowledge and win more points. The aim is to achieve a
high ranking on the nal leaderboard, based on the collected points.
3.4. Procedure
The experiments took place as a replacement of a normal lecture of
the respective course. Therefore, participants (students) arrived at the
lecture as normal at the designated computer lab at the standard time.
After arriving at the class, the participants were informed about the
experiment and their informed consent was obtained. At ECE, NTUA,
participation was voluntary, however, the incentive for participation
was a bonus of 0.5/10 in the coursesnal grade, instead of an
equivalent exercise in the nal examination. In such manner, every
student, participating or not in the experiment, could receive the
highest grade in the nal examination. The participation of under-
graduate students at Business Administration was mandatory as part of
the course and there were no additional incentive for them to take part
in the experiment.
Participants were instructed that they should pick a computer sta-
tion at the classroom (a computer lab), attend a lecture and then
complete an evaluation form, which was based on the content of the
lecture. They were informed that their performance in the evaluation
form would not aect their course grade, however, they should try to
correctly answer the questions based on their understanding of the
topic described in the lecture. The experimenter mentioned that the
participants would be randomly divided into four groups, without
further information, but the importance of the evaluation form along
with the time constraints for all groups was highlighted.
First part of the experiment procedure was a 15-minute lecture
about the ndings of Petropoulos et al. (2014) research (see 3.3.1).
After the lecture, the participants were randomly assigned to dierent
conditions of the experiment: Group Control, Group Read, Group Play
and Group Read&Play. All students, independently of their group re-
ceived the same incentive in order to eliminate the recruitment bias.
Then, the experimenter informed the participants about their next task
and the available time. Instructions were given for each group respec-
tively. Each group had 15 minutes to complete the respective task. As
described above, Group Control did not have an extra task, Group Read
had 15 minutes to read the paper (task Read), Group Play had 15
minutes to fulll a full round in the gamied application (task Play).
Group Read&Play had 30 (15 + 15) minutes to fulll the task Read and
then the task Play. Finally, all groups had to complete the evaluation
Fig. 1. Horses for Courses architecture.
Table 2
Integrated motivational aordances in Horses for Courses application.
Aordance Denition Purpose in Horses for Courses
Points Numeric measure of playersperformances. Reward for the correct application of method selection protocol.
Levels Diculty moderated based on playersexpertise. Indicator of progression and diculty.
Challenges Predened quests and increasingly more dicult objectives. Positive impetus to keep players engaged to maximize their points.
Leaderboard Direct comparison of playersperformance. Increase of competition among students.
Source: (Buckley and Doyle, 2017; Bunchball, 2010; Kapp, 2013; Maican et al., 2016; Nah et al., 2014; Seaborn and Fels, 2015; Zichermann and Cunningham, 2011).
N.-Z. Legaki, et al. International Journal of Human-Computer Studies 144 (2020) 102496
form within 15 minutes, which measured their performance (see Ma-
terials 3.3.2). Participants completed all of the tasksstages in-
dependently and were not allowed to communicate with other parti-
cipants. The procedure was the same at both participating campuses,
including the materials and the experimenter.
4. Results
The objective of this study is to identify the impact of the challenge-
based gamication on studentsperformances in the evaluation form
and consequently on their comprehension. Thus, we examine the re-
lative performance of a control group in comparison with that of the
treatment groups. Studentsperformances on the evaluation form were
calculated as the sum of the right answers on the questionnaire, nor-
malized to a maximum of 100. Summary statistics of results are pre-
sented in Table 3 for each treatment along with the studentsgender,
their academic major: ECE, NTUA or Business Administration and their
educational level: Undergraduate or MBA. The distribution of the per-
formances is illustrated in percentiles with box-plot diagrams in Fig. 5,
which are further separated into dierent treatments, including gender,
major and educational level. Finally, for a subset of the sample
(N=146), three extra variables have been examined along with the
treatment groups, namely studentsexpertise in English, and use of
personal computers and games. Studentsresponses were ranging from
1=beginner to 5=procient. Horses for Courses is a web-based gami-
ed application which demands the use of a personal computer and the
language of its interface is English. Therefore, we examined the re-
lationship between these variables and studentsperformances by
testing if these variables would be statistically signicant in students
performances in conjunction with the treatment received.
The analysis of the results was conducted in three steps. First, we
investigated the mean values of the studentsperformances and their
statistically signicant dierences with respect to specic treatments.
Table 3 presents the number of students per treatment, the mean value
of the studentsperformances, and the standard deviation in each
treatment group, regarding their gender, academic major and educa-
tional level. Overall, the groups that experienced the challenge-based
gamication achieved greater mean values of performances than the
other groups. More precisely, Group Read&Play, which read the re-
spective paper and used the gamied application, reached the highest
mean performance of 58.05 out of 100, and had the second lower level
of standard deviation in results (SD=17.00). Group Play, which only
experienced the gamied application, had the second highest mean
performance of 52.55 out of 100 and the highest level of standard de-
viation in results (SD=19.74). Group Read had a lower mean perfor-
mance of 46.13 out of 100 (SD=18.68). Finally, Group Control had the
Fig. 2. Horses for Coursesowchart of a full game round.
N.-Z. Legaki, et al. International Journal of Human-Computer Studies 144 (2020) 102496
lowest mean performance, 36.13 out of 100, but also the lowest stan-
dard deviation (SD=11.57).
Based on the Shapiro-Wilk test on the ANOVA residuals, the as-
sumption of normality was violated. In order to study the signicant
dierences in the average values of all the groupsperformances, we ran
the non-parametric Kruskal-Wallis rank sum test (Kruskal and
Wallis, 1952). The null hypothesis of equal dierences is rejected (chi-
squared=70.842, df=3, p<0.001) and we can therefore establish
signicant dierences between the groups. Furthermore, we conducted
pairwise multiple comparisons without making assumptions about
normality, using the Dunn procedure (Dinno, 2015; Dunn, 1961; Zar
et al., 1999), with a condence interval equal to 95%. Kruskal-Wallis
with Dunns post-test was chosen to test the signicant dierences be-
cause data was not normally distributed in all cases. Table 4 presents
the outcomes concluding that all treatment groups resulted in sig-
nicantly higher performances compared to Group Control (p.
adj<0.001, Kruskal-Wallis with Dunns post test). Additionally, Table 4
displays the respective eect size of these treatments compared to
Group Control, based on non-parametric Clis Delta estimator (Cli,
2014; Macbeth et al., 2011; Wilcox, 2006). The only pairwise com-
parison without statistically signicant dierences in studentsperfor-
mances is Group Play versus Group Read&Play. Finally, Group
Read&Play outperformed all the other groups. This group noted the
highest improvement regarding the mean values of performances of
Group Control equal to 60.67%. Group Play and Group Read follow
with improvement equal to 45.45% and 27.68% respectively.
The performances of all groups were compared directly, focusing on
the assessment of questions in the evaluation form, despite that treat-
ments did not have the same duration. In order to deal with this
limitation of our study, we used independent binary variables for the
tasks Read and Play respectively. The value of the variable Read is
equal to 1 if the respective group completed the task, and 0 otherwise.
The same applies for the variable Play. Then, the Scheirer-Ray-Hare test
was performed (Scheirer et al., 1976), using studentsperformances and
these variables. Results show that each of the tasks: Read the research
(H=16.014, p<0.001) or Play with Horses for Courses application
(H=52.81, p<0.001) had a signicant impact on studentsperfor-
mances, but their interaction was not signicant (H=2.019, p=0.156).
Along with the impact of dierent treatments on studentsperfor-
mances, the independent variables gender, academic major and edu-
cational level were examined using the Scheirer-Ray-Hare test. While
the studentsgender and their academic major appeared to have a great
impact on their performances, based on the results presented in Table 5,
the interactions between them and the respective treatments they un-
derwent did not have signicant impact. The studentseducational level
was not an important variable, nor was its interaction with the treat-
Regarding the impact of additional variables on a subset of our
sample, only the impact of the studentsexpertise in English resulted in
statistically signicant dierences in studentsperformances. The last
rows of Table 5 show the results of the Scheirer-Ray-Hare test for the
respective samples. Results about the mean values and the standard
deviations of these extra variables are presented in Table 6. However,
these variables will not be further analyzed because students reported
their answers only in the recent experiments.
Table 7 demonstrates the impact of dierent treatments on students
performances, regarding the statistically signicant variables of gender
and academic major. We calculated the improvement of more specied
Fig. 3. View of 1
level challenges of Horses for Courses application.
N.-Z. Legaki, et al. International Journal of Human-Computer Studies 144 (2020) 102496
groups regarding these variables compared to the mean value of the
studentsperformances of Group Control (equal to 36.13) as a bench-
mark value. Students at the ECE, NTUA have noted the highest im-
provement regarding the mean values of their performances in the
evaluation form. Furthermore, female students, independently of their
major, who had used the gamied application beneted more from
gamication; their mean performances are higher from those of the
respective groups composed of male participants. These ndings do not
apply in non-gamied groups.
To conclude, we divided all the data of studentsperformances into
two larger groups, instead of four: the non-gamied group, composed of
165 students (M=41.04, SD=16.22), who did not use the Horses for
Courses application (Group Control and Group Read) and 200 students
(M=55.30, SD=18.58) who used it (Group Play and Group
Read&Play), the gamied group. We adopted this approach in order to
examine the overall impact of the challenge-based gamication on
studentslearning outcomes. Fig. 6 illustrates the studentsperfor-
mances for each group in percentiles with box-plot diagrams. Normality
is not conrmed, thus Wilcoxon-Mann-Whitney rank sum test was
performed, with a condence interval equal to 95%. The null hypoth-
esis of equal dierences in means is rejected (W=23821, p<0.001),
while the use of Horses for Courses presents a moderate level of impact
Fig. 4. View of 4
level of Horses for Courses application.
Table 3
The challenge-based gamication results per treatment and variable.
Variable Group Control Group Read Group Play Group Read&Play
n M SD n M SD n M SD n M SD
Female 15 27.08 10.07 19 35.14 19.19 28 49.08 20.46 33 57.81 18.75
Male 69 38.10 10.98 62 49.49 17.32 72 53.90 19.43 67 58.16 16.21
Academic major
ECE, NTUA 61 39.56 10.58 65 50.56 16.55 74 59.78 15.67 79 61.53 15.28
Business Administration 23 27.04 8.97 16 28.13 16.18 26 31.97 15.19 21 44.94 17.07
Educational Level
UG 71 36.28 11.87 68 47.59 19.10 85 52.45 21.03 83 56.00 16.90
MBA 13 35.34 10.16 13 38.46 14.62 15 53.13 10.02 17 68.01 14.00
Total 84 36.13 11.57 81 46.13 18.68 100 52.55 19.74 100 58.05 17.00
N.-Z. Legaki, et al. International Journal of Human-Computer Studies 144 (2020) 102496
based on non-parametric Clis Delta estimator (delta estimate=0.44
(medium)) and an improvement regarding mean values of perfor-
mances, equal to 34.75%.
Studentsperformances in the nal evaluation form (questionnaire)
should not be confused with their game performances. Weak positive
correlation was found between studentsperformances at the nal
evaluation form and their game performances for students who ex-
perienced the challenge-based gamication (r(146)=0.339, p <0.001).
However, the value of the Pearsons Correlation Coecient is calcu-
lated only for a subset of students (N=148), who used the gamied
application, since there was no specic instruction to students to use
the same personal details in the gamied application and in the eva-
luation form.
Fig. 5. Studentsperformances per treatment and variable.
Table 4
Pairwise multiple comparisons among the groups based on Kruskal-Wallis with
Dunns post test and CliDelta eect size.
Groups Z P.adj Delta
Improvement (%)
Control vs. Read -3.70 0.001 0.35
Control vs. Play -6.16 <0.001 0.51 (large) 45.45%
Control vs. Read&Play -8.04 <0.001 0.69 (large) 60.67%
Play vs. Read 2.25 0.049 - -
Play vs. Read&Play -1.96 0.05 - -
Read&Play vs. Read 4.10 <0.001 - -
N.-Z. Legaki, et al. International Journal of Human-Computer Studies 144 (2020) 102496
5. Discussion of results
Overall, the results suggest that the challenge-based gamication
improves learning outcomes in a forecasting course. This type of ga-
mication presents the greatest improvement in studentsperformances
when it is combined with traditional teaching methods. However, our
results show that even this gamied application alone, integrated in a
lecture, may have a positive impact on learning outcomes.
In general, groups who experienced the challenge-based gamica-
tion have greater performances than the groups that only participated
in traditional teaching methods such as only attending the lecture
(Group Control) or reading the paper (Group Read). More precisely, the
group whose participants read the respective paper and used the ga-
mied application had the highest performance regardless of their
gender, academic major and level of studies. However, the mean value
of the performances of this group is not statistically signicant dierent
from the group who only used the gamied application. Despite the fact
that Group Read&Play had extra 15 minutes to read the respective re-
search and then use the gamied application, the interaction of these
two tasks did not seem to have a great impact on studentsperfor-
mances; while each of the tasks, Read or Play did. According
to Fisher et al. (1981), the amount of time that students are focused or
engaged in an activity is generally positively associated with their
learning outcomes. Given that, we might speculate that the students
who had to complete two tasks may not have been fully engaged
throughout the duration of the tasks. Nevertheless, the aim of our study
is to investigate the impact of gamication on studentsperformances.
So, based on our analysis the use of this gamied application presents
an improvement regarding the mean values of performances, equal to
34.75%. In addition, the challenge-based gamication may improve
studentsperformance by up to 89.45% compared to only being present
at a lecture. Under certain conditions, the use of gamication within
less time, may have almost the same impact as reading and using the
gamied application, as far as learning outcomes in forecasting are
Moreover, we can state that a gamied application combined in a
lecture may improve learning outcomes at both schools. However, its
impact is even more important in the case of engineering students,
where both female and male participants had signicantly better per-
formances. This nding is in agreement with the fact that gamication
has already been incorporated into software engineering and math/
science education to a greater degree (Alhammad and Moreno, 2018;
Dicheva et al., 2015; Pedreira et al., 2015). Dicheva et al. (2015) argue
that one possible explanation could be the lack of fully customized
gamied applications in a variety of educational elds or the fact that
instructors in these schools are more qualied to develop such appli-
cations. Another explanation might be that gamication helps to in-
crease student interest in dicult concepts in
engineering (Markopoulos et al., 2015). Thus, engineering students
may benet more by experiencing these subjects as more manageable.
Although the research in this eld is at a preliminary stage (Alhammad
and Moreno, 2018; Pedreira et al., 2015), Pedreira et al. (2015) support
that gamication has great potential in software engineering education
mainly because of the nature of tasks, which demand high motivation
Table 5
Impact of the treatments, variables and their interactions.
Variables Gender Academic major Educational Level
Groups (N=365) df H Sig. df H Sign. df H Sign.
Treatment (df=3) 1 7.74 0.005 1 73.41 <0.001 1 0.20 0.657
(H=70.84, p<0.001)
Interaction 3 5.10 0.164 3 7.02 0.071 3 7.63 0.054
Variables English Prociency PC Expertise Game Expertise
Groups (n=146) df H Sig. df H Sign. df H Sign.
Treatment (df=3) 4 15.26 0.004 4 7.83 0.098 4 0.51 0.972
(H=24.77, p<0.001)
Interaction 9 2.58 0.995 9 5.41 0.797 12 11.31 0.502
Table 6
Studentsperformances per treatment and extra variables.
Group Performance English Prociency PC Expertise Game Expertise
Control M=32.3 M=3.36 M=3.43 M=3.29
(n=28) SD=11.4 SD=1.37 SD=1.14 SD=1.49
Read M=44.4 M=3.59 M=4 M=3.37
(n=27) SD=18.4 SD=1.39 SD=0.83 SD=1.04
Play M=43.2 M=3.77 M=3.96 M=3.26
(n=47) SD= 20.6 sd=1.37 SD=1.02 SD=1.15
Read&Play M=56.7 M=4.23 M=4.07 M= 3.27
(n=44) SD= 20.0 SD=0.96 SD=0.90 SD=1.34
Table 7
Improvement of studentsperformances per treatment, gender and academic major.
Group Academic major Gender n M SD Improvement (%) Delta Est.
ECE, NTUA Female 4 32.81 5.41 -9.18 -0.173 (small)
Control (9.49%) Male 57 40.03 10.72 10.80 0.195 (small)
(0%) Business Administration Female 11 25.00 10.73 -30.81 -0.524 (large)
(-25.17%) Male 12 28.91 6.94 -20.00 -0.388 (medium)
ECE, NTUA Female 12 44.44 16.74 23.00 0.283 (small)
Read (39.93%) Male 53 51.94 16.35 43.76 0.594 (large)
(27.66%) Business Administration Female 7 19.20 11.04 -46.87 -0.731 (large)
(-22.16%) Male 9 35.07 16.59 -2.94 -0.193 (small)
ECE, NTUA Female 15 59.95 16.07 65.91 0.775 (large)
Play (65.45%) Male 59 59.74 15.70 65.34 0.779 (large)
(45.44%) Business Administration Female 13 36.54 17.97 1.13 -0.068 (negligible)
(-11.51%) Male 13 27.40 10.61 -24.15 -0.424 (medium)
ECE, NTUA Female 16 68.45 13.70 89.45 0.935 (large)
Read&Play (70.30%) Male 63 59.77 15.26 65.43 0.764 (large)
(60.65%) Business Administration Female 17 47.79 17.53 32.28 0.400 (medium)
(24.38%) Male 4 32.81 7.86 -9.18 -0.179 (small)
N.-Z. Legaki, et al. International Journal of Human-Computer Studies 144 (2020) 102496
by students (Pedreira et al., 2015). Our results strengthen this state-
ment, indicating that challenge-based gamication was even more ef-
fective in engineering studentsperformances, without neglecting its
potential in business schools, as well.
Another nding, based on the results presented in Table 7, is that
female users of this gamied application, independently of their edu-
cational background, had higher performances and a higher level of
improvement compared to their male counterparts. However, this
nding does not apply in non-gamied groups (Group Control and
Group Read). More specically, the female participants of gamied
groups at the ECE, NTUA have achieved the highest performances and
the highest improvement. Dierences in motivations (Carr, 2005) and
activities that engage players (Codish and Ravid, 2017; Koivisto and
Hamari, 2014), within genders have already been mentioned. Thus,
since female participants are generally more motivated by challenge
than by competition (McDaniel et al., 2012), females students would
probably be even more motivated by challenge-based gamication,
which would lead to better performance. Another possible explanation
could be that female students receive higher levels of playfulness in a
gamied educational content (Codish and Ravid, 2015), which in-
creases their motivation and improves their learning outcomes. It is
important to note that while the dierence in the sample size of the
genders may be the cause of the results, this assumption requires fur-
ther exploration, and for the time being we may conclude that both
female and male students appear to benet from
gamication (ODonovan et al., 2013).
Last but not least, it is surprising that the variables of students
expertise in use of personal computers and games were not statistically
signicant for the subset of the participants who reported this data
(engineering students=60, business school students=66 and MBA
students=20). However, based on the results of Table 6, these groups
have similar mean values regarding these variables, sharing probably
common characteristics. The studentsexpertise in English is another
parameter which had a statistically signicant impact on their perfor-
mances, since the slides, the research and the interface of the applica-
tion are all in English. The dierence between the mean values of this
variable, achieved by the engineering students (M=4.45, SD=0.86)
and business school students (M=3, SD=1.29) could justify the lower
performances of the business school students in the evaluation form for
all groups.
5.1. Limitations
While challenge-based gamication has a positive impact on
learning outcomes for both engineering and business school students,
some limitations should be acknowledged. We gathered and compared
performances of all groups although the treatments did not have the
same duration as the tasks did. However, this study focuses on the as-
sessment of the evaluation form (questionnaire), which was the same
for all groups and experiments, in order to evaluate the challenge-based
gamication impact. Although conducting both pre- and post-test
would provide further methodological rigor, in the scope of the present
study, the extant knowledge of the participants could be assumed to be
homogeneous because of the specialized content of the lecture. The
content of the lecture, and consequently the topic of the gamied ap-
plication and the evaluation form are focusing on the specic topic of
Method Selection Protocols(Petropoulos et al., 2014). Thus, students
in any of the stages of their studies would have not otherwise learned
this topic. Therefore, the present studys design was economized by
conducting only the post-test of knowledge on the topic. Furthermore,
engineering students noted better performances than business school
students. This fact is probably due to the discrepancy between en-
gineering and business school students in their prociency in English
and in their years of studies.
Additionally, another possible limitation is that there is a dierence
between the two schools in terms of the incentives to participate in our
experiments. Finally, although non-parametric tests have been con-
ducted because of the fact that data was non-normal or heteroscedastic
or both, the dierences in the sample size of dierent groups should be
considered as a limitation (i.e. 307 undergraduate students versus
58 MBA students). Thus, further experiments could contribute to the
conclusions of this study.
6. Conclusion
Overall this study contributes to the core literature of how gami-
cation aects desired outcomes (i.e. skills, knowledge, motivations and
behavior). According to several state-of-the-art analyses of the
eld (Koivisto and Hamari, 2019; Majuri et al., 2018; Nacke and
Deterding, 2017; Rapp et al., 2019), there has been a relative gap of
randomized controlled experiments that could reliably show eects of
gamication. Therefore, this study contributes to the corpus via such an
Fig. 6. Studentsperformances of non-gamied and gamied groups.
N.-Z. Legaki, et al. International Journal of Human-Computer Studies 144 (2020) 102496
experiment by showing that challenge-based gamication (i.e. im-
plementation including points, levels, challenges and a leaderboard)
improves learning outcomes. Our ndings contribute to the literature of
serious games (Connolly et al., 2012) and game-based learning (Hamari
et al., 2016; Squire, 2003), as well. More specically, the study informs
the area of scientic and statistics education (Chu, 2007; Love and
Hildebrand, 2002), because the object of learning in the experiment
was in forecasting techniques. The contribution in this area gets more
importance by considering the need for updating and making more
engaging the traditional teaching methods (Love and Hildebrand, 2002;
Surendeleg et al., 2014), since the statistical skills are fundamentals to
get insight of the available data in order to increase awareness towards
social problems and understanding our world.
In this study, we conducted a factorial design experiment, using a
developed gamication approach named Horses for Courses, which
provides valuable empirical evidence on how challenge-based gami-
cation and reading dierently inuence learning performance. The
ndings of our empirical study, based on a quantitative analysis of our
results, are in line with the positive eects of gamication on
learning (Buckley and Doyle, 2016a; Hamari et al., 2016; Kuo and
Chuang, 2016; Maican et al., 2016; da Rocha Seixas et al., 2016; Simões
et al., 2013; Yildirim, 2017) as well as on software engineering edu-
cation (Alhammad and Moreno, 2018; Pedreira et al., 2015). The results
demonstrate that the challenge-based gamication improves students
performances by 34.75% regarding a statistical forecasting topic and
that the eect was larger for females or engineering students. The
greatest improvements take place when gamication is combined with
traditional methods such as reading, however even simply integrating a
gamied application into a lecture benets students.
This research sheds light upon the eect of challenge-based gami-
cation on statistics education by demonstrating improvement in
learning outcomes. Apart from the theoretical contribution, this study
also provides practical implications to gamication designers and
educators. Challenge-based gamication (i.e. points, levels, challenges
and leaderboard), can be eectively combined with traditional teaching
methods such as lectures and reading in order to improve the learning
outcomes in a variety of educational elds related to statistics and stem
education. Finally, gamication designers should take into account
studentsproles, since our results show that benets dier across
With our study, we position challenge-based gamication as a useful
educational tool in statistics education in dierent academic majors
under certain circumstances, but also we acknowledge its limitations.
Further investigation of the eects of individual game elements or
dierent gamied approaches in statistics or data-related courses with a
larger sample is necessary, in order to enhance the scope of the research
and further rene its ndings. An extension of our research could be to
investigate the impact of additional motivational aordances combined
or compared with challenge-based gamication, under proper and
cautious design. These additional aordances could be related to the
actual content of the course or the actual behaviors that the instructors
want to promote, e.g. social sharing or responding to forum questions.
CRediT authorship contribution statement
Nikoletta-Zampeta Legaki: Conceptualization, Methodology,
Software, Formal analysis, Investigation, Writing - original draft,
Writing - review & editing, Visualization, Validation, Funding acquisi-
tion. Nannan Xi: Conceptualization, Writing - original draft, Writing -
review & editing. Juho Hamari: Conceptualization, Methodology,
Writing - original draft, Writing - review & editing, Supervision,
Funding acquisition. Kostas Karpouzis: Methodology,
Conceptualization. Vassilios Assimakopoulos: Conceptualization,
Methodology, Supervision.
Declaration of Competing Interest
The authors declare that they have no known competing nancial
interests or personal relationships that could have appeared to inu-
ence the work reported in this paper.
Special thanks to the students of the forecasting courses at the
School of Electrical and Computer Engineering of the National
Technical University of Athens, Greece and to the students at the
Business Administration Department in the School of Business and
Economics of the University of Thessaly, Greece who participated in our
experiments and helped to investigate the eect of challenge-based
gamication on student learning.
An early version of this study was presented at the International
GamiFIN Conference 2018, at the HICSS-52 2019 52nd Hawaii
International Conference on System Sciences and at the International
GamiFIN Conference 2019. This work was supported by the European
Union's Horizon 2020 research and innovation program under the
Marie Sklodowska-Curie [grant agreement ID 840809]; Business of
Finland [Grant No. 5654/31/2018]; Business of Finland [Grant No.
4708/31/2019]; Liikesivistysrahasto [Grant No. 14-7798].
Albritton, M.D., McMullen, P.R., 2006. Classroom integration of statistics and manage-
ment science via forecasting. Decision Sci. J. Innovat. Educ. 4 (2), 331336.
Alcivar, I., Abad, A.G., 2016. Design and evaluation of a gamied system for erp training.
Comput. Human. Behav. 58, 109118.
Alhammad, M.M., Moreno, A.M., 2018. Gamication in software engineering education:
asystematic mapping. J. Syst. Softw. 141, 131150.
Attali, Y., Arieli-Attali, M., 2015. Gamication in assessment: do points aect test per-
formance? Comput. Educ. 83, 5763.
Barnett, B., Kirtland, M., Ganapathy, M., 2005. An architecture for distributed applica-
tions on the internet: overview of the microsoft?. net framework. J. STEM Educ. 4 (1).
Buckley, P., Doyle, E., 2016. Gamication and student motivation. Interact. Learn.
Environ. 24 (6), 11621175.
Buckley, P., Doyle, E., 2016. Using web-based collaborative forecasting to enhance in-
formation literacy and disciplinary knowledge. Interact. Learn. Environ. 24 (7),
Buckley, P., Doyle, E., 2017. Individualising gamication: an investigation of the impact
of learning styles and personality traits on the ecacy of gamication using a pre-
diction market. Comput. Educ. 106, 4355.
Buckley, P., Garvey, J., McGrath, F., 2011. A case study on using prediction markets as a
rich environment for active learning. Comput. Educ. 56 (2), 418428.
Bunchball, I., 2010. Gamication 101: an introduction to the use of game dynamics to
inuence behavior. White paper 9.
Caponetto, I., Earp, J., Ott, M., 2014. Gamication and education: A literature review.
European Conference on Games Based Learning. 1. Academic Conferences
International Limited, pp. 50.
Carr, D., 2005. Contexts, gaming pleasures, and gendered preferences. Simulat. Gaming
36 (4), 464482.
Christy, K.R., Fox, J., 2014. Leaderboards in a virtual classroom: a test of stereotype
threat and social comparison explanations for womens math performance. Comput.
Educat. 78, 6677.
Chu, S., 2007. Some initiatives in a business forecasting course. J. Stat. Educ. 15 (2).
Cli, N., 2014. Ordinal Methods for Behavioral Data Analysis. Psychology Press.
Cobb, G., 1992. Teaching statistics. Heed. Call Change 22, 343.
Codish, D., Ravid, G., 2015. Detecting playfulness in educational gamication through
behavior patterns. IBM J. Res. Dev. 59 (6), 611.
Codish, D., Ravid, G., 2017. Gender Moderation in Gamication: Does One Size Fit All?.
Connolly, T.M., Boyle, E.A., MacArthur, E., Hainey, T., Boyle, J.M., 2012. A systematic
literature review of empirical evidence on computer games and serious games.
Comput. Educ. 59 (2), 661686.
Craighead, C.W., 2004. Right on target for time-series forecasting. Decis. Sci. J. Innovat.
Educ. 2 (2), 207212.
Debnath, S.C., Tandon, S., Pointer, L.V., 2007. Designing business school courses to
promote student motivation: an application of the job characteristics model. J.
Manag. Educ. 31 (6), 812831.
Deterding, S., Dixon, D., Khaled, R., Nacke, L., 2011. From game design elements to
gamefulness: Dening gamication. Proceedings of the 15th International
Academic MindTrek Conference: Envisioning Future Media Environments. ACM, New
York, NY, USA, pp. 915.
Dichev, C., Dicheva, D., 2017. Gamifying education: what is known, what is believed and
what remains uncertain: a critical review. Int. J. Educ. Technol. Higher Educ. 14
(1), 9.
N.-Z. Legaki, et al. International Journal of Human-Computer Studies 144 (2020) 102496
Dicheva, D., Dichev, C., Agre, G., Angelova, G., 2015. Gamication in education: a sys-
tematic mapping study. Educ. Technol. Soc. 18 (3), 7589.
Dinno, A., 2015. Nonparametric pairwise multiple comparisons in independent groups
using dunns test. Stata. J. 15 (1), 292300.
DomíNguez, A., Saenz-De-Navarrete, J., De-Marcos, L., FernáNdez-Sanz, L., PagéS, C.,
MartíNez-HerráIz, J.-J., 2013. Gamifying learning experiences: practical implications
and outcomes. Comput. Educ. 63, 380392.
Donihue, M.R., 1995. Teaching economic forecasting to undergraduates. J. Econ. Educ.
26 (2), 113121.
Dunn, O.J., 1961. Multiple comparisons among means. J. Am. Stat. Assoc. 56 (293),
Fisher, C.W., Berliner, D.C., Filby, N.N., Marliave, R., Cahen, L.S., Dishaw, M.M., 1981.
Teaching behaviors, academic learning time, and student achievement: an overview.
J. Classroom Interaction 17 (1), 215.
Gallaugher, J.M., Ramanathan, S.C., 1996. Choosing a client/server architecture: a
comparison of two-and three-tier systems. Inf. Syst. Manag. 13 (2), 713.
Gardner, L., 2008. Using a spreadsheet for active learning projects in operations man-
agement. INFORMS Trans. Educ. 8 (2), 7588.
Gareld, J., Ben-Zvi, D., 2007. How students learn statistics revisited: a current review of
research on teaching and learning statistics. Int. Stat. Rev. 75 (3), 372396.
Gavirneni, S., 2008. Teaching the subjective aspect of forecasting through the use of
basketball scores. Decision Sci. J. Innovat. Educ. 6 (1), 187195.
Gel, Y.R., OHara Hines, R.J., Chen, H., Noguchi, K., Schoner, V., 2014. Developing and
assessing e-learning techniques for teaching forecasting. J. Educ. Bus. 89 (5),
Giullian, M.A., Odom, M.D., Totaro, M.W., 2000. Developing essential skills for success in
the business world: a look at forecasting. J. Appl. Bus. Res. 16 (3), 5162.
González, C., Area, M., 2013. Breaking the rules: Gamication of learning and educa-
tional materials. Proceedings of the 2nd international workshop on interaction design
in educational environments. pp. 753.
Hamari, J., 2013. Transforming homo economicus into homo ludens: aeld experiment
on gamication in a utilitarian peer-to-peer trading service. Electron. Commer. Res.
Appl. 12 (4), 236245.
Hamari, J., Sherno, D.J., Rowe, E., Coller, B., Asbell-Clarke, J., Edwards, T., 2016.
Challenging games help students learn: an empirical study on engagement, ow and
immersion in game-based learning. Comput. Human. Behav. 54, 170179.
Hamari, J., Tuunanen, J., 2014. Player Types: A Meta-synthesis.
Hanke, J., 1984. The teachers/practitioners corner forecasting in business schools: a
survey. J. Forecast. 3 (2), 229234.
Hanke, J., 1989. Forecasting in business schools: a follow-up survey. Int. J. Forecast. 5
(2), 259262.
Hanus, M.D., Fox, J., 2015. Assessing the eects of gamication in the classroom: a
longitudinal study on intrinsic motivation, social comparison, satisfaction, eort, and
academic performance. Comput. Educ. 80, 152161.
Hunicke, R., LeBlanc, M., Zubek, R., 2004. Mda: a formal approach to game design and
game research. Proceedings of the AAAI Workshop on Challenges in Game AI. 4. pp.
Huotari, K., Hamari, J., 2017. A denition for gamication: anchoring gamication in the
service marketing literature. Electron. Markets 27 (1), 2131.
Johnson, D., Deterding, S., Kuhn, K.-A., Staneva, A., Stoyanov, S., Hides, L., 2016.
Gamication for health and wellbeing: a systematic review of the literature. Internet
Intervent. 6, 89106.
Kapp, K.M., 2013. The Gamication of Learning and Instruction Fieldbook: Ideas into
Practice. John Wiley & Sons.
Karpouzis, K., Caridakis, G., Fotinea, S.-E., Efthimiou, E., 2007. Educational resources and
implementation of a greek sign language synthesis architecture. Comput. Educ. 49
(1), 5474.
Kasurinen, J., Knutas, A., 2018. Publication trends in gamication: a systematic mapping
study. Comput. Sci. Rev. 27, 3344.
Koivisto, J., Hamari, J., 2014. Demographic dierences in perceived benets from ga-
mication. Comput. Human. Behav. 35, 179188.
Koivisto, J., Hamari, J., 2019. The rise of motivational information systems: areview of
gamication research. Int. J. Inf. Manage. 45, 191210.
Kros, J.F., Rowe, W.J., 2016. Business school forecasting for the real world. Advances in
Business and Management Forecasting. Emerald Group Publishing Limited, pp.
Kruskal, W.H., Wallis, W.A., 1952. Use of ranks in one-criterion variance analysis. J. Am.
Stat. Assoc. 47 (260), 583621.
Kuo, M.-S., Chuang, T.-Y., 2016. How gamication motivates visits and engagement for
online academic dissemination - an empirical study. Comput. Human. Behav. 55,
Lambruschini, B.B., Pizarro, W.G., 2015. Tech-gamication in university engineering
education: Captivating students, generating knowledge. 2015 10th International
Conference on Computer Science & Education (ICCSE). IEEE, pp. 295299.
Landers, R.N., Auer, E.M., Collmus, A.B., Armstrong, M.B., 2018. Gamication science, its
history and future: Denitions and a research agenda. Simulat. Gaming.
Lewandowski, S.M., 1998. Frameworks for component-based client/server computing.
ACM Comput. Surv. (CSUR) 30 (1), 327.
Loomis, D.G., Cox, J.E., 2000. A course in economic forecasting: rationale and content. J.
Econ. Educ. 31 (4), 349357.
Loomis, D.G., Cox Jr, J.E., 2003. Principles for teaching economic forecasting. Int. Rev.
Econ. Educ. 2 (1), 6979.
Love, T.E., Hildebrand, D.K., 2002. Statistics education and the making statistics more
eective in schools of business conferences. Am. Stat. 56 (2), 107112.
Macbeth, G., Razumiejczyk, E., Ledesma, R.D., 2011. CliS delta calculator: a non-
parametric eect size program for two groups of observations. Universitas Psychol.
10 (2), 545555.
Maican, C., Lixandroiu, R., Constantin, C., 2016. - a study of a gamication
framework using zero-cost tools. Comput. Human. Behav. 61, 186197.
Majuri, J., Koivisto, J., Hamari, J., 2018. Gamication of education and learning: A re-
view of empirical literature. Proceedings of the 2nd International GamiFIN
Conference, GamiFIN 2018. CEUR-WS.
Makridakis, S., Wheelwright, S.C., Hyndman, R.J., 2008. Forecasting methods and ap-
plications. John wiley & sons.
Markopoulos, A.P., Fragkou, A., Kasidiaris, P.D., Davim, J.P., 2015. Gamication in en-
gineering education and professional training. Int. J. Mech. Eng. Educ. 43 (2),
McDaniel, R., Lindgren, R., Friskics, J., 2012. Using badges for shaping interactions in
online learning environments. 2012 IEEE International Professional Communication
Conference. IEEE, pp. 14.
McEwen, B.C., 1994. Teaching critical thinking skills in business education. J. Educ. Bus.
70 (2), 99103.
Mendez-Carbajo, D., 2018. Experiential learning in intermediate macroeconomics
through fredcast. Int. Rev. Econ. Educ.
Morschheuser, B., Hamari, J., Koivisto, J., Maedche, A., 2017. Gamied crowdsourcing:
conceptualization, literature review, and future agenda. Int. J. Hum. Comput. Stud.
106, 2643.
Morschheuser, B., Hassan, L., Werder, K., Hamari, J., 2018. How to design gamication? a
method for engineering gamied software. Inf. Softw. Technol. 95, 219237.
Nacke, L.E., Deterding, C.S., 2017. The maturing of gamication research. Comput. Hum.
Behav. 450454.
Nah, F.F.-H., Zeng, Q., Telaprolu, V.R., Ayyappa, A.P., Eschenbrenner, B., 2014.
Gamication of education: a review of literature. International Conference on hci in
Business. Springer, pp. 401409.
ODonovan, S., Gain, J., Marais, P., 2013. A case study in the gamication of a university-
level games development course. Proceedings of the South African Institute for
Computer Scientists and Information Technologists Conference. ACM, pp. 242251.
Osatuyi, B., Osatuyi, T., de la Rosa, R., 2018. Systematic review of gamication research
in is education: a multi-method approach. CAIS 42, 5.
Pedreira, O., García, F., Brisaboa, N., Piattini, M., 2015. Gamication in software en-
gineering - a systematic mapping. Inf. Softw. Technol. 57, 157168.
Petropoulos, F., Makridakis, S., Assimakopoulos, V., Nikolopoulos, K., 2014. Horses for
coursesin demand forecasting. Eur. J. Oper. Res. 237 (1), 152163.
Pfeer, J., Fong, C.T., 2002. The end of business schools? less success than meets the eye.
Acad. Manag. Learn. Educ. 1 (1), 7895.
Rapp, A., Hopfgartner, F., Hamari, J., Linehan, C., Cena, F., 2019. Strengthening gami-
cation studies: current trends and future opportunities of gamication research. Int.
J. Hum. Comput. Stud. 127, 16.
Reiners, T., Wood, L.C., Chang, V., Gütl, C., Herrington, J., Teräs, H., Gregory, S., 2012.
Operationalising Gamication in an Educational Authentic Environment.
da Rocha Seixas, L., Gomes, A.S., de Melo Filho, I.J., 2016. Eectiveness of gamication
in the engagement of students. Comput. Human. Behav. 58, 4863.
Sánchez-Martín, J., Dávila-Acedo, M.A., et al., 2017. Just a game? Gamifying a general
science class at university: collaborative and competitive work implications. Think.
Skill. Creat. 26, 5159.
Scheirer, C.J., Ray, W.S., Hare, N., 1976. The analysis of ranked data derived from
completely randomized factorial designs. Biometrics 32 (2), 429434.
Seaborn, K., Fels, D.I., 2015. Gamication in theory and action: a survey. Int. J. Hum.
Comput. Stud. 74, 1431.
Simões, J., Redondo, R.D., Vilas, A.F., 2013. A social gamication framework for a k-6
learning platform. Comput. Hum. Behav. 29 (2), 345353.
Snider, B.R., Eliasson, J.B., 2013. Beat the instructor: an introductory forecasting game.
Decis. Sci. J. Innovat. Educ. 11 (2), 147157.
Snodgrass, J.G., Dengah, H.F., Lacy, M.G., Fagan, J., 2013. A formal anthropological view
of motivation models of problematic mmo play: achievement, social, and immersion
factors in the context of culture. Transcult. Psychiatry 50 (2), 235262.
de Sousa Borges, S., Durelli, V.H., Reis, H.M., Isotani, S., 2014. A systematic mapping on
gamication applied to education. Proceedings of the 29th annual ACM symposium
on applied computing. ACM, pp. 216222.
Spithourakis, G.P., Petropoulos, F., Nikolopoulos, K., Assimakopoulos, V., 2015.
Amplifying the learning eects via a forecasting and foresight support system. Int. J.
Forecast. 31 (1), 2032.
Squire, K., 2003. Video games in education. Int. J. Intell. Games Simulat. 2 (1), 4962.
Surendeleg, G., Murwa, V., Yun, H.-K., Kim, Y.S., 2014. The role of gamication in
education-a literature review. Contempor. Eng. Sci. 7 (29), 16091616.
Tabatabai, M., Gamble, R., 1997. Business statistics education: content and software in
undergraduate business statistics courses. J. Educ. Bus. 73 (1), 4853.
Torres, C., Babo, L., Mendonça, J., 2018. Engaging students to learn forecasting methods.
10th annual International Technology, Conference on Education and New Learning
Technologies. INTED, pp. 1033710343.
Wilcox, R.R., 2006. Graphical methods for assessing eect size: some alternatives to co-
hens d. J. Exp. Educ. 74 (4), 351367.
Xi, N., Hamari, J., 2019. Does gamication satisfy needs? a study on the relationship
between gamication features and intrinsic need satisfaction. Int. J. Inf. Manage. 46,
Xi, N., Hamari, J., 2020. Does gamication aect brand engagement and equity? a study
in online brand communities. J. Bus. Res. 109, 449460.
Yee, N., 2006. Motivations for play in online games. CyberPsychol. Behav. 9 (6),
Yee, N., Ducheneaut, N., Nelson, L., 2012. Online gaming motivations scale: development
and validation. Proceedings of the SIGCHI conference on human factors in computing
N.-Z. Legaki, et al. International Journal of Human-Computer Studies 144 (2020) 102496
systems. ACM, pp. 28032806.
Yildirim, I., 2017. The eects of gamication-based teaching practices on student
achievement and studentsattitudes toward lessons. Internet Higher Educ. 33, 8692.
Zar, J.H., et al., 1999. Biostatistical analysis. Pearson Education India.
Zichermann, G., Cunningham, C., 2011. Gamication by design: Implementing game
mechanics in web and mobile apps. OReilly Media, Inc..
Nikoletta-Zampeta Legaki is a Marie - Curie Researcher at
Gamication Group, Tampere University. She pursuits her
PhD in School of Electrical and Computer Engineering of
National Technical University of Athens (NTUA), Greece.
She has been a researcher and teaching assistant in
Forecasting and Strategy Unit, School of Electrical and
Computer Engineering since 2012. During this period, she
has participated in various research projects about fore-
casting and data analytics and she has worked as a con-
sultant in Financial Services and Risk Management, in EY,
Greece. Her research interests lie on time series forecasting,
business forecasting information systems, gamication and
educational methods in teaching forecasting.
Dr. Nannan Xi is a Postdoctoral researcher in Gamication
Group. She got her PhD in marketing management from
Zhongnan University of Economic and Law (ZUEL), China
and holds M.Sc. in International Business from the
University of Lincoln, UK. Her dissertation was awarded as
the excellent Ph.D. dissertation in ZUEL. Xis current re-
search focuses on gamication in marketing, especially in
gamied interaction in brand management. In addition, her
research interests include customer management in gami-
cation and virtual reality/augmented reality/mixed rea-
lity in business and sharing economy.
Dr. Juho Hamari is a Professor of Gamication at the
Faculty of Information Technology and Communications,
Tampere University. He leads the Gamication Group. His
research covers several forms of information technologies
such as games, motivational information systems, new
media (social networking services, eSports), peer-to-peer
economies (sharing economy, crowdsourcing), and virtual
economies. Dr. Hamari has authored several seminal em-
pirical, theoretical and meta-analytical scholarly articles on
these topics from perspective of consumer behavior,
human-computer interaction, game studies and information
systems science. His research has been published in a
variety of prestigious venues.
Dr. Kostas Karpouzis is currently an Associate Researcher
at the Institute of Communication and Computer Systems
(ICCS) of the National Technical University of Athens
(NTUA) in Greece. His research interests lie in the areas of
human computer interaction, emotion understanding, af-
fective and natural interaction, serious games and games
based assessment and learning. He has participated in more
than twenty research projects at Greek and European level.
He is also a member of the Student Activities Chair for the
IEEE Greece Section and a member of the Editorial Board
for international journals.
Prof. Vassilios Assimakopoulos is a professor of
Forecasting Systems at the School of Electrical and
Computer Engineering of the National Technical University
of Athens (NTUA). He is the author of over than 60 original
publications and papers in international journals and con-
ferences. Moreover, he has conducted research on in-
novative tools for management support and decision sys-
tems design. He has participated and led numerous projects,
funded by National and European institutes. He specializes
in various elds of Strategic Management, Design and
Development of Information systems, Business Resource
Management, Statistical and Forecasting Techniques using
time series.
N.-Z. Legaki, et al. International Journal of Human-Computer Studies 144 (2020) 102496
... Several studies in the academic literature have emphasized the advantages of incorporating gamification into the educational process, such as an enhancement of students' capacity to acquire new competencies, attendance, motivation, and participation in undergraduate course activities by using points, leaderboards, levels, and badges [30][31][32][33]. Although various gamification methods have been applied and implemented in educational contexts during the last decade to achieve certain goals, their effect on student performance and motivation is still questionable [18,19,[34][35][36]. ...
... Leaderboards provide comparative feedback to the students of a gamified course about their performance against other students using points and ranks. Many studies have reported that leaderboards can positively impact the achievements of learners [18,53,54], engagement [30], and the amount of completed work [21,36,55], while maintaining performance [56] and course attendance [57]. While gamifying a course with leaderboards, the aforementioned favorable verdicts come with some concerns that need to be handled. ...
Full-text available
Currently, academia is grappling with a significant problem—a lack of engagement. Humankind has gone too far into exploring entertainment options, while the education system has not really kept up. Millennials love playing games, and this addiction can be used to engage and motivate them in the learning process. This study examines the effect of digital game-based learning on student engagement and motivation levels and the gender differences in online learning settings. This study was conducted in two distinct phases. A game-based and traditional online quizzing tools were used to compare levels of engagement and motivation, as well as to assess the additional parameter of gender difference. During the first phase of the study, 276 male and female undergraduate students were recruited from Sophomore Seminar classes, and 101 participated in the survey, of which 83 were male and 18 were female. In the second phase, 126 participants were recruited, of which 107 (63 females and 44 males) participated in the anonymous feedback surveys. The results revealed that digital game-based learning has a more positive impact on student engagement and motivation compared to traditional online activities. The incorporation of a leaderboard as a gaming element in the study was found to positively impact the academic performance of certain students, but it could also demotivate some students. Furthermore, female students generally showed a slightly higher level of enjoyment toward the games compared to male students, but they did not prefer a comparison with other students as much as male students did. The favorable response from students toward digital game-based activities indicates that enhancing instruction with such activities will not only make learning an enjoyable experience for learners but also enhance their engagement.
... Different groups were exposed to the experiment and the total number of participants was 365. The study included analysis of descriptive Statistics to validate that the performance of students with gamification in the course was higher than that of students without gamification [4]. Another study aimed to improve the "Hypothesis testing". ...
... Although studies have been reported where the object of study is engagement in diverse academic courses, the herein investigation is limited to the study of engagement only in Statistics courses. Regarding video games, Legaki et al. [4] predefined quests and a changing difficulty level was included as the integrated motivational affordance to keep participants engaged and trying to win more points. Another reported effort to increase the level of engagement in learning Statistics through the use of video games. ...
Conference Paper
Teaching Statistics can be a challenging endeavor. Gamification possesses immense potential to improve acquisition and retention of knowledge in general by way of making learning fun and offering the ability to embed and contextualize concepts in the game design and simulation. The breadth of technology available today presents the possibility of designing, developing, using, and/or adapting existing technologies to tailor budget-friendly gamified simulations to meet curricular needs. This research work proposes to complement the gamification approach with the design and implementation of the video game by the participating students. The Game Builder Garage (GBG) is a programming game intended for the video game tablet console called Nintendo Switch. The GBG is a visual programming language that uses characters that are controlled using the console, and as such, is intuitive and does not require a programming background. This study presents a literature review of the use of gamification in an educational setting with the intent of providing a way forward on its incorporation into a Statistics course. Besides, the degree of motivation and the knowledge acquired will be measured with a Motivated Strategies for Learning Questionnaire (MSLQ), and a standardized exam, respectively. Furthermore, including the design of educational video games in the learning of Statistics could improve motivation, engagement and, consequently, performance, and be an option to be implemented in other engineering courses.
... In chess, sports, and leisure activities, researchers reported a positive association between challenges and enjoyment and between challenges and intrinsic goal orientations (15). In higher education, the challenge-based learning approach has been proven useful in fostering learning and learning outcomes (16)(17)(18). In some workplace training, challenge-based teaching was superior to lecture-based teaching, with more participant interaction and better learning outcomes (19). ...
Full-text available
Background Little is known about the public health professionals engaged in educating and training new or future researchers in public health. Research in this direction identifies their issues, concerns, challenges, and needs. This study focused on the professional development challenges of Chinese public health professionals. Methods Snowball sampling was utilized. A total of 265 public health professionals participated. An instrument of 6 dimensions (burnout, sleep issue, mood issue, friends’ support, exercise, and challenges) was developed, revised, and administered online. Two different approaches, the conventional and data screening approaches, were applied. The former started with item quality analyses, whereas the latter began with data quality checks. The chi-square tests of associations and logistic regressions were performed on both approaches. Results and discussion 19.25% of the participants were detected and deleted as careless respondents. Using both approaches, six professional development challenges except one (“Multidisciplinary learning”) were significantly associated with various demographic features. The two approaches produced different models though they converged sometimes. The latent variables of exercise predicted professional development challenges more frequently than other latent variables. Regarding correct classification rates, results from the data screening approach were comparable to those from the conventional approach. Conclusion The latent variables of exercise, such as “Exercise effects,” “Expectations of exercise,” and “Belief in exercise,” might be understudied. More research is necessary for professional development challenges using exercise as a multidimensional construct. Based on the current study, screening and deleting careless responses in survey research is necessary.
The education system has been changing continuously with several new strategies for effective student learning. E‐learning has provided the opportunity for students to learn from anywhere and from anyone. To motivate the learners and improve their performance in e‐learning courses, different pedagogical activities have been practiced among which gamification has a good edge. Gamification is one of the upcoming fields in education for getting students motivated and engaged through game‐like activities with various gamification elements in nongame settings. Gamification can be further improved by dynamically adapting the mechanics and dynamics of gamification to a variety of factors such as the personality, needs, values, performance, and motivations of learners. The advancement of artificial intelligence (AI) enables the development of dynamically adapting gamification environments with established AI techniques. The application of AI in adaptive gamification needs to be further explored to discover new approaches. This study provides a comprehensive report of adaptive gamification from the perspective of the existing studies, which map different gamification elements, user profile elements, and intelligent techniques through a systematic literature analysis. Research questions have been formulated and the findings have been presented by exploring 29 focused studies from a selected pool of 622 research articles. The findings have identified the commonly used gamification elements and presented the methods of incorporating them with various AI techniques based on different user profile elements.
Full-text available
This review paper examines a worldwide phenomenon, gamification, in the educational context. After the high popularity of video games, the integration of gamification in the learning context has become one of the practices that have a considerable impact on learning. It is important to contribute to the literature on the educational value of gamification by providing a literature review of some aspects of the studies that have emerged in the field of educational gamification (EG) over a five-year period. It is crucial to have a more complete understanding of educational gamification without restricting emerged literature to defined criteria so that it can be discussed in relation to how much of an impact it has on learning outcomes. The current study systematically reviewed four well-known social science databases for studies on gamification in education published between January 2013 and April 2018. The study yielded 141 relevant papers that focused on only educational gamification concept and excluded other game related approaches. The findings indicate that gathering data regarding learner opinions on the application of gamification is the most frequent (N=34) study goal of these studies. Although the majority of papers highlight that gamification increases learner motivation and engagement, there is not yet enough data in this large body of research to conclusively state that using gamification would enhance learning and academic performance at the same level in every context. The findings imply that careful design based on instructional goals and context, as well as the needs of the students, is necessary for integration of gamification to serve its purpose effectively. To sum up, ensuring the description of the most accurate implementation process for specific education contexts and subject fields may still be challenging, given the use of gamification in many distinct disciplines. Furthermore, educational gamification is studied in many contexts with various expectations, and the outcomes would inevitably be different. This review categorized the numerous educational advantages and challenges of gamification.
Full-text available
The purpose of this study was to review the directions and tendencies of the conducted research on the application of gamification in education, and more specifically, to provide insight and understanding on which game elements are mostly used, results generated, predominance context, region, and research approach. A total of 43 articles were filtered from three databases: Scopus, Science Direct, and Jstor. Mapped results show that badges, points, levels, and leaderboards are the predominant game mechanics employed by researchers. In addition, the most used game dynamics in educational context are visual status, social engagement, rapid feedback, freedom to choose, and freedom to fail. Further result shows that Information Technology (IT), Engineering and Education were the predominant context in which gamification research is applied. SouthEast Asia was found to have received many gamification studies. The most used research approach is quantitative study, involving experiments and surveys. Results were generally mixed, but most results were positive, suggesting that gamification improves learning. From these reviews, a number of gaps were found. It is still not clear which game elements are responsible for the positive impact of gamification. Thus, research is needed to compare the impact of various game elements. The review recommends among other things, that game elements should be integrated into the school curriculum as an alternative to learning different topics, there is also the need to conduct more studies on gamification in different settings to come up with more findings.
Full-text available
Gamification has recently been touted as an effective user engagement method with the ability to enhance online education. Even though there has been more research on gamification in recent decades, however, there is still no taxonomy of its concrete impacts. The aim of this study was to fill this gap by identifying the effects that gamification has on both students and teachers engagement in online learning. This study combined a systematic literature review methodology and PRISMA processes to analyze high-quality articles on gamification in education from the last ten years (2012-2022) as extracted from three databases like Scopus, ScienceDirect, and Web of Science. The evaluation and analysis of the 40 articles included in the study summarized and categorized the benefits that the deployment of gamification offered to student engagement, motivation, creativity, and overall performance as well as to teachers as motivation, engagement, or feedback and evaluation. The result of the systematic literature review found that the educational levels that frequently incorporate gaming into their curricula are higher education, but it also shed light on the challenges that come with implementing gamification in the classroom. We hope the study’s findings assist educators and students in using gamification as a successful intervention technique by providing them with pertinent information that can influence outcomes and knowledge of the educational content and establish the right conditions for an effective learning process.
Gamification can increase the efficiency and the benefits to be received from an educational content; therefore, it could be used as a novel tool to help the child to engage more with the content. Because studies have shown that through gamification and gamified educational content, children with ASD's motor, behavioural, and communication skills can improve. However, selecting a tool that is suited for the child's unique needs, and instruments that can be adjusted along the way depending on the child's progress, plays a key role to achieve this goal. Different from the traditional methods, gamification-based approaches are accessible to children with ASD both at home and in school environments. Through gamification and under the guidance of professionals, traditional methods can be delivered in a non-traditional way. This way parents can be involved in their children's education and can keep track of their children's progress through the measurement tools encompassed in the gamification method. This chapter focuses on the abovementioned issues and presents a literature survey.
Full-text available
This Toolkit is designed for teachers committed to creatively, attractively, and effectively introducing students to the social world.1 It is part of a collection of texts looking to popularize the mobile approach to social education. The Toolkit is also a part of the broader international research and education project “Social Education on the Move (SoMoveED),”2 aiming to develop and popularize a mobile approach to social education.We described the theoretical and methodological foundations of this approach in the bookKnowledge on the Move. Studies on Social Mobile Education. Some of the experiences we recorded as educational videos, and for students we developed a textbook Learning on the Move. Social Education Handbook for Students. The book is something of a companion volume to this Toolkit.
Full-text available
Gamification has become a popular technique in marketing. Many companies believe that gamification can potentially increase the engagement, awareness and loyalty of consumers with respect to the brand. However, there is current dearth of empirical evidence supporting these beliefs beyond the pervasive hype. In this study we investigated the relationships between gamification, brand engagement and brand equity among consumers (N = 824) of two online gamified brand communities. The results showed that achievement and social interaction -related gamification features were positively associated with all three forms of brand engagement (emotional, cognitive and social). Immersion -related gamification features were only positively associated with social brand engagement. Additionally, brand engagement was further positively associated with brand equity. The results imply that gamification can positively affect brand engagement and further increase brand equity, and that gamification appears to be an effective technique for brand management.
Full-text available
Gamification is now a well-established technique in Human-Computer Interaction. However, research on gamification still faces a variety of empirical and theoretical challenges. Firstly, studies of gamified systems typically focus narrowly on understanding individuals. short-term interactions with the system, ignoring more difficult to measure outcomes. Secondly, academic research on gamification has been slow to improve the techniques through which gamified applications are designed. Third, current gamification research lacks a critical lens capable of exploring unintended consequences of designs. The 14 articles published in this special issue face these challenges with great methodological rigor. We summarize them by identifying three main themes: the determination to improve the quality and usefulness of theory in the field of gamification, the improvements in design practice, and the adoption of a critical gaze to uncover side-effects of gamification designs. We conclude by providing an overview of the questions that we feel must be addressed by future work in gamification. Gamification studies would benefit from a wider use of theories to account for the complexity of human behavior, a more thorough exploration of the many opportunities coming from the world of games, and an ethical reflection on the use of game design elements in serious domains.
Full-text available
Background Compared to traditional persuasive technology and health games, gamification is posited to offer several advantages for motivating behaviour change for health and well-being, and increasingly used. Yet little is known about its effectiveness. Aims We aimed to assess the amount and quality of empirical support for the advantages and effectiveness of gamification applied to health and well-being. Methods We identified seven potential advantages of gamification from existing research and conducted a systematic literature review of empirical studies on gamification for health and well-being, assessing quality of evidence, effect type, and application domain. Results We identified 19 papers that report empirical evidence on the effect of gamification on health and well-being. 59% reported positive, 41% mixed effects, with mostly moderate or lower quality of evidence provided. Results were clear for health-related behaviours, but mixed for cognitive outcomes. Conclusions The current state of evidence supports that gamification can have a positive impact in health and wellbeing, particularly for health behaviours. However several studies report mixed or neutral effect. Findings need to be interpreted with caution due to the relatively small number of studies and methodological limitations of many studies (e.g., a lack of comparison of gamified interventions to non-gamified versions of the intervention).
Full-text available
Today, our reality and lives are increasingly game-like, not only because games have become a pervasive part of our lives, but also because activities, systems and services are increasingly gamified. Gamification refers to designing information systems to afford similar experiences and motivations as games do, and consequently, attempting to affect user behavior. In recent years, popularity of gamification has skyrocketed and manifested in growing numbers of gamified applications, as well as a rapidly increasing amount of research. However, this vein of research has mainly advanced without an agenda, theoretical guidance or a clear picture of the field. To make the picture more coherent, we provide a comprehensive review of the gamification research (N = 819 studies) and analyze the research models and results in empirical studies on gamification. While the results in general lean towards positive findings about the effectiveness of gamification, the amount of mixed results is remarkable. Furthermore, education, health and crowdsourcing as well as points, badges and leaderboards persist as the most common contexts and ways of implementing gamification. Concurrently, gamification research still lacks coherence in research models, and a consistency in the variables and theoretical foundations. As a final contribution of the review, we provide a comprehensive discussion, consisting of 15 future research trajectories, on future agenda for the growing vein of literature on gamification and gameful systems within the information system science field.
Full-text available
Background. Definitions of gamification tend to vary by person, both in industry and within academia. One particularly popular lay interpretation, introduced and popularized by Ian Bogost, and reiterated by Jan Klabbers, is that gamification is “bullshit” and “exploitationware.” They describe gamification as a marketing term or business practice invented to sell products rather than to represent a real and unique phenomenon relevant to a nascent game science. However, this view is an oversimplification, one which ignores a growing body of theory development and empirical research on gamification within a post-positivist epistemology. In fact, because gamification is so much more outcome-focused than general game design, current gamification research in many ways has a stronger footing in modern social science than much games research does. Aim. In this article, to address common misunderstandings like these, we describe the philosophical underpinnings of modern gamification research, define the relationship between games and gamification, define and situate gamification science as a subdiscipline of game science, and explicate a six-element framework of major concerns within gamification science: predictor constructs, criterion constructs, mediator constructs, moderator constructs, design processes, and research methods. This framework is also presented diagrammatically as a causal path model. Conclusion. Gamification science refers to the development of theories of gamification design and their empirical evaluation within a post-positivist epistemology. The goal of gamification scientist-practitioners should be to understand how to best meet organizational goals through the design of gamification interventions, drawing upon insights derived from both gamification science and games research more broadly.
Conference Paper
Presently, higher education institutions are faced with the challenge of developing student’s abilities and skills needed in their future workplace. In what concerns technological skills, the use of spreadsheets for calculations, analysis of data and forecasting is a common and important practice in companies. Particularly, the MS Excel software is widely used by professionals from all fields. In this sense, Economic, Business and Marketing graduates need competencies in forecasting methods, extremely useful for decision-making processes. It thus becomes imperative to implement pedagogical practices to encourage students to use these technological tools. The later will promote the development of competencies in forecasting methods to solve future real problems. In this paper, we attempt to address these issues by analysing the MS Excel software capabilities as a teaching tool in a forecasting methods course. It was proposed to the students to carry out a learning project involving statistical concepts, namely linear regression, performed in MS Excel. We examine the performance and engagement of two samples of students with different backgrounds and from distinct realities. One group is composed of ERASMUS’ students from several nationalities and fields of study and other group consists of Portuguese students of the Marketing Bachelor degree, both enrolled in an optional course of the bachelor degree in Marketing taught at Porto Accounting and Business School from Polytechnic of Porto. The effectiveness of this approach is shown through the analysis of results of students’ projects. We verify that students in both groups achieved the task proposed goals and applied appropriately the required concepts in an engaged way. Keywords: Higher education, Excel, forecasting, linear regression.
This article discusses how to use FREDcast, an interactive economic forecasting game from the Federal Reserve Bank of St. Louis, to teach macroeconomics. It provides instructions on how to set the game up and organize related teaching materials such as graphs and data points through a FRED (Federal Reserve Economic Data) Dashboard. It also puts forward a series of classroom discussion questions and suggested assignments. We argue that this pedagogical strategy provides an engaging context for the discussion of core topics in macroeconomics while contributing to the development of data-related expected student proficiencies.