Content uploaded by Lisa-Maria Putz
Author content
All content in this area was uploaded by Lisa-Maria Putz on Jan 13, 2019
Content may be subject to copyright.
Content uploaded by Horst Treiblmaier
Author content
All content in this area was uploaded by Horst Treiblmaier on Nov 22, 2018
Content may be subject to copyright.
Increasing Knowledge Retention through Gamified Workshops: Findings
from a Longitudinal Study and Identification of Moderating Variables
Lisa-Maria Putz
University of Applied Sciences, Upper Austria
lisa-maria.putz@fh-steyr.at
Horst Treiblmaier
MODUL University Vienna
horst.treiblmaier@modul.ac.at
Abstract
Apathetic and poorly motivated students require
educators to redesign their educational measures in
order to create inspiring learning environments. One
such educational measure is gamification, a new tool
for active learning to improve students’ motivation,
with the ultimate goal of increasing knowledge
retention. In this paper we investigate the effects of
gamification on short- and long-term knowledge
gains. Moreover, the moderating effects of gender
and school type are scrutinized. We conducted a
longitudinal study with 384 students using three
assessments at different times and compared the
results from gamified and non-gamified workshops.
Our findings indicate that gamification is an effective
tool to increase students’ knowledge retention in the
short term, but not necessarily in the long term.
There was no significant effect of gender, but we
found some preliminary evidence that school type
might have a moderating effect on knowledge
retention.
1. Introduction
Previously, researchers have lamented that “the
challenge of teaching has increased when faced with
increasingly apathetic students and diverse learning
styles” (p.101) [54]. In particular, the so-called
generation Y, referring to people who were born
between 1980 and 2000, is hard to motivate with
traditional teaching methods. Despite continuous
efforts by education professionals to select novel and
innovate teaching methods, many students perceive
traditional schooling as boring and ineffective [24]
and prefer engaging and interactive learning activities
[46] [35].
In 1995, [14] found that traditional teaching does
not foster students’ capability to solve problems
autonomously or transfer learning content to new
situations. Educators have consequently identified a
need for novel teaching methods which tackle the
challenge of efficient knowledge transfer, student
engagement, and transferability of the course
contents. Furthermore, they should also equip
students with problem solving skills. Experiential
learning is considered as a potential solution to
address students’ motivational problems and to
increase students’ problem solving skills by
increasing their level of involvement [14] [24][39].
Games and gamified activities represent a form of
experiential learning and constitute an alternative to
traditional forms of learning. Gamification is a
promising approach to fostering intrinsic motivation
[16], making learning more engaging [48] and
increasing students’ knowledge retention [1][27][39].
By applying game design elements in non-game
contexts [8] [37] [49], gamification takes advantage
of the growing popularity of playing games in
different settings [45]. Previous studies have shown
that the educational use of games and game elements
triggers students’ active learning processes, which
subsequently improves their knowledge retention
[21].
In recent years, gamification has been frequently
applied to create highly motivating learning
environments that help to overcome the lack of
student interest and to increase students’ knowledge
retention levels [11]. However, rigorous empirical
research on the effectiveness of gamification in
educational environments and its impact on
knowledge retention is still scarce [11] [18] [27].
This especially pertains to the question of whether
gamification has the potential to positively influence
students’ knowledge retention. Moreover, the
potential moderating effects of gender and school
type are often neglected.
In this paper, we examine the potential of
gamification to make education more appealing and
effective. In particular, we strive to answer the
question of whether students are better able to recall
course content after participating in gamified
workshops as opposed to their traditional
counterparts. To this end, an experiment was
conducted comparing the knowledge retention
performance of two groups of workshop participants,
Proceedings of the 52nd Hawaii International Conference on System Sciences | 2019
URI: hps://hdl.handle.net/10125/59586
ISBN: 978-0-9981331-2-6
(CC BY-NC-ND 4.0) Page 1456
as well as examining the moderating effects of
gender and school type.
The remainder of the paper is organized as
follows. First, the existing literature on the
application and effects of gamification in education is
reviewed. Next, we briefly outline our hypotheses,
describe the methodology, and present the results.
Finally, the paper ends with a discussion, several
limitations, as well as some concluding thoughts.
2. Gamification and Education
The term “gamification” was first used in 2008,
but only gained widespread adoption in academia and
the industry from the year 2010 onwards [45].
Gamification is defined as “the use of game-design
elements in any non-game system context to achieve
one or more of the following: intrinsic and extrinsic
user motivation, facilitated information processing,
better goal achievement, and behavioral changes”
[49].
Popular game design elements—so-called
motivational affordances—include, for example,
points, badges, leaderboards, competition, immediate
feedback, and time constraints [9][28][49]. The
concept of gamification substantially differs from
playing games. Whereas the former employs specific
game elements in contexts which are otherwise
unrelated to play, the latter describes full-fledged
games for all kinds of purposes (e.g., education) [11].
Positive effects of gamification have been
identified in fields such as diverse as health [42][44],
crowdsourcing [30], and technology adoption [48].
Additionally, the concept has been previously applied
for educational purposes [21][27][38]. A review
paper conducting a systematic mapping study of
empirical papers on gamification found that most
published studies had been conducted in the context
of education [17].
Previous research shows mixed, but
predominantly positive effects of gamification on
cognitive and behavioral outcomes. Gamification
supports learning processes and offers great potential
to improve students’ motivation as well as to enhance
engagement [15]. According to [43] (p. 347) the aim
of gamification in education is to “extract the game
elements that make good games enjoyable and fun to
play, adapt them and use those elements in the
teaching processes. […] Learning must not be a
boring activity while gaming is fun. Learning can be
fun if students learn as if they were playing a game.”
In order to achieve this, game elements need to be
deployed in such a way that students are able to
retain the learned content and apply the learning
experience outside the game context [29]. One
approach to gamifying the teaching experience is to
include elements of games that have been part of the
students’ lives since early childhood [2].
Additionally, gamification has been shown able to
increase students’ engagement as well as extrinsic
and intrinsic motivation [1]. This has important
implications for motivational research, since students
that are intrinsically motivated tend to have a higher
level of in-class participation and achieve better
results [6].
Existing empirical studies on gamification in
education focus mainly on engagement and
motivation as outcomes for learners [34]. Although
gamified teaching techniques have been shown to be
suitable in areas such as the military, retail
organizations, computer service providers and
manufacturing organizations [21], little research has
been conducted on gamification and knowledge
retention [1][27]. Research specifically addressing
the question of whether gamification can lead to
increased knowledge retention remains scarce to this
day. A notable exception is an empirical study with
more than 100 participants which found that
gamification has a positive impact on students’
knowledge retention [1].
In addition, research on the effects of
gamification applied in a workshop setting is fairly
new. A notable exception, [20], compared gamified
workshops with non-gamified ones in terms of their
capacity to generate innovation. The results indicate a
higher capability for self-learning in the gamified
group.
3. Hypotheses
A sustained increase in knowledge is the main
goal of all educational measures. It is therefore highly
desirable that the substantive content is fully
understood and retained by the students for as long as
possible after the educational event [13][32]. In this
paper “short term” refers to the period of about 20
minutes immediately following the workshops and
“long term” refers to the point in time two weeks
after the workshops [13].
The forgetting curve [13] has been the subject of
intense scholarly discussions regarding the
measurement of knowledge retention [32]. It
postulates a 100% recall immediately after a learning
event and indicates that memory retention
corresponds to about 58% of the total content after 20
minutes, which corresponds to the second assessment
in our study. After two weeks the retention rate is
predicted to be about 25% [13][32]. By providing an
Page 1457
indication of students’ ability to recall content over
time, the learning curve provides a benchmark in the
learning literature against which to assess actual
observations.
Gamification is frequently applied in marketing
and education with the aim of encouraging specific
behaviors and increasing engagement and motivation.
It has previously been used for teaching purposes to
help educators broaden the variety of teaching
methods to motivate students [19][38]. Gamification
is intended to engage students in an interactive
setting, leading to better memorization [5][16][21]
[38].
A literature review and a systematic mapping
study on gamification in education identified a
positive impact in the majority of studies [11]. A
particular finding was higher levels of student
engagement and motivation in gamified settings.
Gamified teaching resulted in active participation,
more project engagement, increased attendance, and
a higher proportion of students passing the course.
An experiment comparing a non-gamified and a
gamified group found that gamification led to
positive results including an increase in class
participation and course success [10]. Moreover,
students evaluated the gamified course better than its
non-gamified counterpart.
Summarizing the findings from previous research,
we postulate that gamification is a suitable tool for
enriching exhausting full-day workshops since
students face a huge amount of learning content and
need to stay concentrated and focused over an
extended period of time. More specifically, we
hypothesize that students in the gamified group are
able to retain knowledge better than students in the
non-gamified group.
H1: In the short term the gamified group achieves
better knowledge scores than the non-gamified group
H2: In the long term the gamified group achieves
better knowledge scores than the non-gamified group
The study of gender differences in cognitive
functioning and knowledge gains has a long tradition,
with partially conflicting results. [7], for example,
emphasize the need for the design and use of gender-
specific didactics to better meet the different learning
requirements of female and male students. [51] show
that female students have a higher study-oriented
learning culture than male students, which positively
influences knowledge accumulation. Several studies
suggest that female students perform better in terms
of knowledge gains than their male counterparts
[22][36][51]. Gender-specific performance, however,
seems to depend on subject matter: while males
outperform females on tasks of visuospatial ability
and mathematical reasoning, females show better
results in tests involving memory and language use
[12][26]. [25] conclude that male students are more
sensitive to resources that create a learning-oriented
environment and that these resources can facilitate
their commitment in learning. [23] find that female
students appreciate the social benefits arising from
gamification more than males. Since there is no
conclusive evidence of the moderating effect of
gender on memory retention through gamification,
we hypothesize:
H3: Male and female students perform equally
Previous research has found significant
differences in students’ learning orientation by school
type. Students attending vocational schools achieve
lower scores than students attending other school
types, which might be caused by the learning
environment [22][51]. Compared to students engaged
in higher education, vocational students spend less
time on theoretical learning and instead focus on
advancing their practical abilities. A self-selection
process may also play a role, since better grades are
needed for entry into higher educational programs
than vocational schools [3]. It is therefore
hypothesized:
H4: Higher educational students outperform
vocational school students in terms of knowledge
retention
4. Methodology
We used an experimental design in order to
investigate whether significant differences in
knowledge retention exist between students who
participated in a gamified full-day workshop and
students who were not exposed to gamification
during their workshop participation. The
questionnaire used to measure students’ knowledge—
Table 4 in the appendix—featured questions of
varying complexity, and was developed in close
cooperation with the industry to ensure its practical
applicability. The scales were pre-tested in three
workshops with a total of 131 students to ensure their
understandability and validity.
Our pilot study further showed that the separation
of students coming from the same class into different
testing groups led to social interaction threats and
resentful demoralization [50]. In order to achieve a
high level of internal validity in this study, it was
Page 1458
essential that students and teachers did not know that
there was another group which received an
alternative treatment [50]. Classes as a whole were
therefore randomly assigned to either the non-
gamified or the gamified group.
To ensure comparability in terms of educational
levels, all participants were recruited from the second
year of tertiary educational programs at Austrian
institutions (i.e., vocational schools and higher
educational institutions). A substantial number of
students attend these school types at a later career
stage, which explains the broad age range of the
sample.
The study included a gamified and a non-
gamified workshop group. Measurements were taken
at three points in time using identical questions:
immediately before (observation 1: O1), 20 minutes
after (O2), and two weeks after (O3) the workshops.
The aim of the workshops was to train logistics
students on sustainable transport by combining
theoretical and practical knowledge. The gamified
and non-gamified workshops had the same duration,
identical learning goals and educational material.
Both workshops were organized as full-day events
lasting from 9:45 am until 3:45 pm. The instructors
of the workshops stayed the same in order to
eliminate any moderating impact by the instructor.
The program and the interactive tasks were also
identical in both workshops. Whereas the gamified
workshops included multiple motivational
affordances (e.g., competition, leaderboards, badges,
time constraints, storytelling, immediate feedback,
rewards, clear goals, social interaction) [49], the non-
gamified workshops did not include any game
elements. For example, the students had to do the
same calculations in each group, but only received
points for correct solutions in the gamified
workshops. Students in the gamified group also
received bonus points for accomplishing each task,
such as finding the correct solution in a simulation
game. All tasks were embedded in a “story” (i.e., a
case) to use the motivational advantages of
storytelling as a game element [21]. Competition
between the gamified groups was encouraged by
leaderboards. Grouping students into teams served to
reduce the negative effects of competition at an
individual level and to support social interaction
[40][41]. The detailed gamified and non-gamified
workshop programs can be found in the appendix in
Table 5. Table 6 details the game elements that were
used during the gamified workshop. Some elements,
such as leaderboards, immediate feedback, clear
goals, competition and cooperation, were used
throughout the whole day, while others (i.e., time
constraints, storytelling, rewards) were only applied
in specific sessions.
5. Results
In total, 384 students participated in the study,
with 261 assigned to the gamified group and 123 to
the non-gamified group. The latter simultaneously
served as a control group in our experimental design,
since their “treatment” resembles common teaching
practices in educational settings. The students were
slightly older in the non-gamified group (m = 23.88,
sd = 9.47) than in the gamified group (m = 18.73, sd
= 4.15). Gender distribution in the total sample was
balanced with 194 female and 190 male students. In
the non-gamified group, 73 students were from
vocational schools and 50 students from higher
educational institutions. Due to large class sizes in
higher educational institutions, it was not feasible to
include these students in the gamified workshops,
which required more interaction than the non-
gamified events.
A Mann-Whitney U test showed that knowledge
levels did not differ significantly between the
gamified and the non-gamified group prior to the
workshops (p = .39, U = 7,383). Additionally, we
found that the age difference had no significant effect
on the level of prior knowledge. The demographics
are shown in Table 1.
Table 1: Demographics
Gamified
Non-
gamified
Total
Age
m = 18.73
(sd = 4.15)
m = 23.88
(sd = 9.47)
m = 20.38
(sd = 6.79)
Gender
male
161
29
190
female
100
94
194
School type
vocational
261
73
334
higher
education
50
50
In the following sections we use several non-
parametric tests for hypothesis testing in order to
accommodate the properties of our data (e.g.,
distributions, skewness, ordinal level of
measurement, unequal sample sizes). In order to
verify the robustness of our findings we also applied
parametric tests and obtained identical results. The
latter are not reported in this paper due to lack of
space.
Table 2 shows the descriptive results of the
knowledge measurements for vocational school
students only. Since not all students completed the
Page 1459
assessments, the total numbers do not fully
correspond to the number of all workshop
participants in Table 1.
We conducted the first analysis with vocational
students only in order to avoid a distorting effect of
the school type and to account for our relatively small
sample of higher educational students.
The maximum score attainable for the knowledge
questions was 11. We found that the mean score in
the first assessment was quite low (m = 3.59, sd =
1.55), which confirmed the novelty of the topic for
both groups. The scores improved substantially
immediately after the workshops (m = 6.74, sd =
2.90). As expected, knowledge levels had declined
after two weeks (m = 5.33, sd = 2.08), but the scores
were still significantly better than those of the initial
assessment. The comparison between the gamified
and the non-gamified group showed that the values in
the gamified group increased from 3.62 ( sd = 1.48)
in O1 to 7.10 ( sd = 2.19) in O2 and decreased to 5.39
( sd = 2.13) in O3. In contrast, the scores in the non-
gamified group were 3.47 ( sd = 1.77) in O1, 5.49 (
sd = 2.17) in O2 and 4.97 ( sd = 1.77) in O3.
To test whether the knowledge improvements
between O1 – O2 (short term) and O1 – O3 (long
term) were significant, we used a dependent samples
Wilcoxon signed-rank test. The results show a
significant difference between O1 and O2 (Z = -1.97,
p<.01) and between O1 and O3 (Z = -9.13, p<.01) for
the gamified group. Similarly, the results for the non-
gamified group were significant (Z = -6.38, p<.01 for
O1 to O2 and Z = -3.50, p<.01 for O1 to O3).
Table 2: Knowledge mean values and standard
deviations across groups (max = 11)
Gamified
mean, sd, no
Non-gamified
mean, sd, no
Total
mean, sd, no
O1
3.62 (1.48),
240
3.47 (1.77),
66
3.59 (1.55),
306
O2
7.10 (2.19),
234
5.49 (2.17),
68
6.74 (2.9),
302
O3
5.39 (2.13),
207
4.97 (1.77),
37
5.33 (2.08),
244
m
5.37
5.24
5.30
In order to test H1 and H2, which postulate a
significant positive effect of gamification in the short
and long term, we used a non-parametric independent
samples Mann-Whitney U test. This test showed that
the scores in the first assessment were not
significantly different between the gamified and non-
gamified group (U = 7,883, p =.390). In the second
assessment, the gamified group outperformed the
non-gamified group, corroborating H1 (U = 4,583, p
< .01). In the third assessment, the mean value in the
gamified group (m = 5.39) was higher than that in the
non-gamified group (m = 4.97), but no significant
difference between the groups was found (U = 3,357,
p = .114, H2 rejected).
A frequency analysis revealed that 69.7% of the
students in the gamified group achieved more than
six points out of eleven in assessment 2, as opposed
to 35.5% of the non-gamified group. 9.9% of the
gamified group achieved ten or eleven points, as
opposed to 1.5% in the non-gamified group. In O3,
no student of the non-gamified group achieved nine
or more points whereas 6.4% of the students from the
gamified group achieved nine or more points. 29.5%
of the gamified and 24.3 % of the non-gamified
group achieved more than six points in O3.
Figure 1 illustrates the effect of the gamified and
non-gamified workshops and shows that both
workshops clearly outperform the benchmark from
the forgetting curve postulating that after two weeks
the retention rate will hover around 25% [13] [32].
Figure 1: The impact of gamification on
knowledge retention
In order to test for significant differences between
genders we used a non-parametric independent
samples Mann-Whitney U test. We found no
significant differences in any of the three
assessments, thereby corroborating H3. In the first
assessment, the male students slightly outperformed
the female students, but the results were not
significant (gamified: U = 6,320, p = .292; non-
gamified: U = 314.50, p = .624). In the second
assessment the male students had higher scores in the
gamified group while the female students had higher
scores in the non-gamified group, but, again, the
results were not significant (gamified: U = 5,813, p =
.148; non-gamified: U = 366, p = .909). As was
predicted by [13], the amount of memorized
knowledge had decreased after two weeks for both
groups, with no significant gender effect (gamified:
Page 1460
U = 5,620, p = .300, non-gamified: U = 518, p =
.409). The results are summarized in Table 3.
Table 3: Knowledge across genders
Gamified
mean
Non-gamified
mean
female
male
female
male
O1
3.51
3.68
3.43
3.62
O2
6.88
7.25
5.51
5.39
O3
5.58
5.26
5.15
4.50
Finally, we analyzed the differences between
school types in the non-gamified workshop to explore
whether school type might qualify as a potential
moderator in future research. In the first assessment
(O1), no significant difference was found between
higher educational and vocational school students (U
= 6,660, p = .20). However, immediately after the
workshops (O2, U = 4,888, p < .01) and two weeks
later (O3, U = 3,197, p < .01), students from higher
educational schools exhibited a significantly better
performance than those from vocational schools. The
results are supported by previous empirical studies,
determining that higher educational school students
have different learning orientations than vocational
school students [51]. Thus, H4 is supported.
Figure 2: Knowledge retention between school
types
6. Discussion, Limitations & Conclusion
In this study, we investigate whether the use of
game elements in educational workshops leads to
improved knowledge retention. Furthermore, we
tested gender and school type as potential moderating
variables.
We compared a gamified with a non-gamified
group, both of which attended workshops with
identical content, but which featured different
designs. Prior to the workshops, no significant
knowledge differences were found based on gender
or school type. However, several differences emerged
in the second and third assessment, both between and
within the groups.
We found that both the gamified and non-
gamified workshops led to a significant increase in
short-term and long-term knowledge. Student
knowledge had increased substantially directly after
the events and remained at a significantly higher
level after two weeks as compared to the initial state.
When it comes to comparing the gamified and the
non-gamified group, the former outperformed the
latter in the short term, indicating that gamification
can be suitable to improve short-term memory
retention. A descriptive frequency analysis indicates
that the participants of the gamified workshops were
also slightly better at recalling knowledge in the
long-term (i.e., after two weeks), but no statistically
significant effect was found.
No significant differences were found between
male and female students in any of the comparisons,
even though the female students had slightly higher
scores in the long term than the male students.
Finally, our experiment showed that higher
educational school students achieved better
knowledge retention than vocational school students.
This effect was predicted by existing literature and
previous empirical studies [3]. However, since the
school types were only compared within the non-
gamified group, with no data being available for the
gamified group, further research is needed to
investigate if the results also hold when educational
measures are gamified.
It is important to mention that we treated
gamification in this study as a collection of game
elements and did not investigate the specific
underlying mechanisms that explain its effects on
knowledge retention. For example, [31] argue that
gamification can trigger emotions which in turn exert
positive effects on knowledge retention.
As a practical implication, we strongly encourage
educators to incorporate game elements into their
teaching routine in order to foster students’
knowledge retention levels. When designing
gamified workshops, the didactical methods for
covering content have to be chosen carefully with
consideration of the demographics of the target
group. According to flow theory [4], for example,
information and learning aims must be adapted to the
age and educational background of the participants
with the goal of achieving an adequate level of
difficulty that is neither too easy nor too hard.
This study has several limitations. We randomly
assigned classes, rather than individuals, to the
gamified and non-gamified groups, since our pre-test
showed that the division of existing groupings
Page 1461
influences student behavior and produces social
threats to the study’s validity [50].
Furthermore, this study was conducted in Austria
and various cultural differences might influence the
results [47]. Moreover, as the focus of the workshops
was specifically to create awareness for sustainable
transportation, further research is needed to
investigate whether the application of gamification to
workshops also improves knowledge retention in
other areas.
Additionally, the composition of the gamified and
non-gamified samples was not equal because of
class-wise assignment. Due to organizational reasons,
students from higher educational schools were only
included in the non-gamified workshops and
therefore our hypothesis regarding the moderating
effect of school type was only tested within the non-
gamified setting.
The focus of this study was exclusively on
knowledge gains pertaining to memorization skills.
Other types of capabilities such as mathematics,
language, or social skills were not measured. Also,
different learning types or gaming types were not
considered.
According to [33] and [41], individual game
elements should be analyzed and not the gamification
concept as a whole. Since the experiment was
designed using gamified full-day workshops as
whole, the investigation of isolated game elements
could not be implemented. Finally, several of the
measurement scale items were slightly modified from
their original and demonstrated formats in order to fit
the specific purposes of this study. We used several
rounds of pretesting to ensure the content validity of
the items, but further research is needed to assess the
psychometric properties of our scales [53].
Our study reveals multiple opportunities for
future research. We treat gamification as a black box,
since we only consider gamification as input and
knowledge retention as the educational output. Future
studies may delve deeper and strive to find more
detailed explanations. A possible approach would be
to include and test the effect of gamification on
hedonic outcomes as mediating factors. Additionally,
it might be worthwhile to closely investigate the
motivational and learning effects over a longer time
period. Qualitative interviews can be used to get a
better understanding of the underlying drivers.
Moreover, investigating differences between
learning types (e.g., [55]) or gaming types is another
promising direction for further research. The focus of
our study was exclusively on knowledge gains
depending on memorization skills. Other types of
capabilities such as social skills or general
intelligence were not measured. Further empirical
research is needed to ensure the generalizability of
the results and to better understand how knowledge
acquisition using gamification differs from traditional
learning techniques. We also suggest the thorough
investigation of the effect of school type in a
gamified setting.
Finally, the positive effects of gamification in
education are not only limited to its ability to
improve knowledge memorization, but also to
enhance social and practical skills such as problem
solving, collaboration, and communication. Thus,
future research might also have a closer look at how
gamification can facilitate social dynamics [29].
Finally, a strong theoretical background for
gamification research, as is suggested in [49], will
help to better integrate new research with previous
findings and to create an incremental research
agenda.
7. References
[1] Buckley, P. and E. Doyle, "Individualising
gamification: an investigation of the impact of learning
styles and personality traits on the efficacy of gamification
using a prediction market", Computers & Education, 106,
2017, pp. 43–55.
[2] Çakıroğlu, Ü., B. Başıbüyük, M. Güler, M. Atabay, and
B. Yılmaz Memiş, "Gamifying an ICT course: Influences
on engagement and academic performance", Advanced
Human-Computer Interaction, 69, 2017, pp. 98–107.
[3] Çolak, E. and D. Kaya, "Learning approaches of
vocational high school students: Grade level and school
type influences", Procedia - Social and Behavioral
Sciences, 116, 2014, pp. 1556–1561.
[4] Csikszentmihalyi, M., Flow and the Psychology of
Discovery and Invention, Harper Collins, New York, NY,
1996.
[5] de Sousa Borges, S., V.H.S. Durelli, H.M. Reis, and S.
Isotani, "A systematic mapping on gamification applied to
education", in Y. Cho, S.Y. Shin, S. Kim, C.-C. Hung, and
J. Hong (eds.), 29th Annual ACM Symposium, Republic of
Korea, 16 - 19 October. 2014.
[6] Deci, E.L., R.J. Vallerand, L.G. Pelletier, and R.M.
Ryan, "Motivation and education: the self-determination
perspective", Educational Psychologist, 26(3-4), 1991, pp.
325–346.
[7] Derichs-Kunstmann, K., "Elemente einer
geschlechtergerechten Didaktik", in Ordnung muss sein!:
Pädagogische Inszenierungen. 2002. Olympe,
Feministische Arbeithefte zur Politik: Zürich.
Page 1462
[8] Deterding, S., D. Dixon, R. Khaled, and L. Nacke,
"From game design elements to gamefulness: defining
gamification", in 15th International Academic MindTrek
Conference: Envisioning Future Media, Tampere, Finland,
28 - 30 September. 2011.
[9] Deterding, S., M. Sicart, L. Nacke, K. O'Hara, and D.
Dixon, "Gamification: using game-design elements in non-
gaming contexts", CHI 2011, May 7–12, 2011, Vancouver,
2011, pp. 2425–2428.
[10] Dias, J., "Teaching operations research to
undergraduate management students: The role of
gamification", International Journal of Management
Education, 15(1), 2017, pp. 98–111.
[11] Dicheva, D., C. Dichev, G. Agre, and G. Angelova,
"Gamification in education: A systematic mapping study",
Journal of Educational Technology & Society, 18(3), 2015,
pp. 75–88.
[12] Downing, K., S.‐W. Chan, W. K. Downing, T.
Kwong, and T.‐F. Lam, "Measuring gender differences in
cognitive functioning", Multicultural Education &
Technology Journal, 2(1), 2008, pp. 4–18.
[13] Ebbinghaus, H., Memory: A Contribution to
Experimental Psychology, Teachers College, Columbia
University, New York City, 1913.
[14] Grabinger, R.S. and J.C. Dunlap, "Rich environments
for active learning: A definition", ALT-J Association for
Learning Technology Journal, 3(2), 1995, pp. 5–34.
[15] Hakulinen, L., T. Auvinen, and A. Korhonen,
"Empirical study on the effect of achievement badges in
TRAKLA2 online learning environment", in Learning and
Teaching in Computing and Engineering, Macau, China, 21
- 24 March. 2013.
[16] Hamari, J. and L. Keronen, "Why do people play
games? A meta-analysis", International Journal of
Information Management, 37(3), 2017, pp. 125–141.
[17] Hamari, J., J. Koivisto, and H. Sarsa, "Does
gamification work? A literature review of empirical studies
on gamification", 47th Hawaii International Conference on
System Sciences, Waikoloa, HI. 2014.
[18] Hamari, J., D.J. Shernoff, E. Rowe, B. Coller, J.
Asbell-Clarke, and T. Edwards, "Challenging games help
students learn: An empirical study on engagement, flow
and immersion in game-based learning", Computers in
Human Behavior, 54, 2016, pp. 170–179.
[19] Huang, W.H.-Y. and D. Soman, Gamification of
Education, Rotman School of Management, Toronto, ON,
2013.
[20] Johnsson, M., "Prisoner's dilemma as a workshop
tool?", 2nd International GamiFIN Conference, Pori,
Finland, 21-23 May. 2018.
[21] Kapp, K.M., The Gamification of Learning and
Instruction: Game-Based Methods and Strategies for
Training and Education, Pfeiffer, San Francisco, CA, 2012.
[22] Kılıç, D. and N. Sağlam, "Investigating the effects of
gender and school type on students’ learning orientations",
Procedia - Social and Behavioral Sciences, 2(2), 2010, pp.
3378–3382.
[23] Koivisto, J. and J. Hamari, "Demographic differences
in perceived benefits from gamification", Advanced
Human-Computer Interaction, 35, 2014, pp. 179–188.
[24] Lee, J.J. and J. Hammer, "Gamification in education:
What, how, why bother", Academic Exchange Quarterly,
15(2), 2011, pp. 146–151.
[25] Legewie, J. and T.A. DiPrete, "School context and the
gender gap in educational achievement", American
Sociological Review, 77(3), 2012, pp. 463–485.
[26] Maccoby, E.E. and C.N. Jacklin, The Psychology of
Sex Differences, Stanford University Press, Stanford, CA,
1974.
[27] Majuri, J., J. Koivisto, and J. Hamari, "Gamification of
Education and Learning: A Review of Empirical
Literature", 2nd International GamiFIN Conference, Pori,
Finland, 21-23 May. 2018.
[28] Monu, K. and P. Ralph, "Beyond gamification:
implications of purposeful games for the information
systems discipline", arXiv preprint arXiv:1308.1042, 2013.
[29] Moore-Russo, D., A. Wiss, and J. Grabowski,
"Integration of gamification into course design: A noble
endeavor with potential pitfalls", College Teaching, 66(1),
2017, pp. 3–5.
[30] Morschheuser, B., J. Hamari, J. Koivisto, and A.
Maedche, "Gamified crowdsourcing: conceptualization,
literature review, and future agenda", International Journal
of Human-Computer Studies, 106, 2017, pp. 26–43.
[31] Mullins, J.K. and S. Rajiv, "Beyond enjoyment: A
cognitive-emotional perspective of gamification", 51st
Hawaii International Conference on System Sciences, Big
Island, HI, 3 - 6 January. 2018.
[32] Murre, J.M.J. and J. Dros, "Replication and analysis of
Ebbinghaus' forgetting curve", PloS One, 10(7), 2015, pp.
1–23.
[33] Nacke, L.E. and S. Deterding, "The maturing of
gamification research", Advanced Human-Computer
Interaction, 71, 2017, pp. 450–454.
Page 1463
[34] Nah, F.F.-H., V.R. Telaprolu, S. Rallapalli, and P.R.
Venkata, "Gamification of education using computer
games", in S. Yamamoto (ed), Human Interface and the
Management of Information. Information and Interaction
for Learning, Culture, Collaboration and Business, 2013.
Springer: Berlin, Heidelberg.
[35] Neale, L., Treiblmaier, H., Henderson, V., Hunter, L.,
Hudson, K., and Murphy, J. (2009) "The Google Online
Marketing Challenge and Research Opportunities", Journal
of Marketing Education, 31(1), pp. 76-85.
[36] OECD, The ABC of Gender Equality in Education:
Attitude, Behaviour, Confidence, OECD Publications,
Paris, 2015.
[37] Putz, L.-M. and Treiblmaier, H. "Creating a Theory-
Based Research Agenda for Gamification", Americas
Conference on Information Systems (AMCIS), August 13-
15, Puerto Rico, 2015.
[38] Putz, L.-M. and H. Treiblmaier, "Gamified workshops
as drivers for attitudinal and behavioral shifts toward
sustainable business practices: the role of enjoyment,
curiosity and external regulation", 51st Hawaii
International Conference on System Sciences, Big Island,
HI, 3 - 6 January. 2018.
[39] Putz, L.-M., H. Treiblmaier, and S. Pfoser, "Field trips
for sustainable transport education: impact on knowledge,
attitude and behavioral intention", The International Journal
of Logistics Management, 9(3), 2018.
[40] Sailer, M., J. Hense, H. Mandl, and M. Klevers,
"Psychological perspectives on motivation through
gamification", Interaction Design and Architecture(s)
Journal, 19, 2013, pp. 28–37.
[41] Sailer, M., J.U. Hense, S.K. Mayr, and H. Mandl,
"How gamification motivates: An experimental study of the
effects of specific game design elements on psychological
need satisfaction", Advanced Human-Computer
Interaction, 69, 2017, pp. 371–380.
[42] Schmidt-Kraepelin, M., S. Thiebes, M.C. Tran, and A.
Sunyaev, "What’s in the Game? Developing a taxonomy of
gamification concepts for health apps", 51st Hawaii
International Conference on System Sciences, Big Island,
HI, 3 - 6 January. 2018.
[43] Simões, J., R.D. Redondo, and A.F. Vilas, "A social
gamification framework for a K-6 learning platform",
Advanced Human-Computer Interaction, 29(2), 2013, pp.
345–353.
[44] Spil, T., A. Sunyaev, S. Thiebes, and R. van Baalen,
"The adoption of wearables for a healthy lifestyle: can
gamification help ?", 50th Hawaii International Conference
on System Sciences, Big Island, HI, 4 - 7 January. 2017.
[45] Thiebes, S., S. Lins, and D. Basten, "Gamifying
information Systems - a synthesis of gamification
mechanics and dynamics", Twenty Second European
Conference on Information Systems, Tel Aviv. 2014.
[46] Treiblmaier, H., Neale, L., and Murphy, J. "Action
Learning of Online Marketing via a Global Contest",
Central Asia Business Journal, 2, 2009, pp. 47-50.
[47] Treiblmaier, H., Neale, L. and Chong, S. "The Web
Site and Brand Trust as Antecedents of Online Loyalty:
Results from Four Countries", International Journal of
Online Marketing, 1(2), pp. 24-42
[48] Treiblmaier, H. and Zeinzinger, T. "Understanding the
Blockchain through a Gamified Experience: A Case Study
from Austria", 25th European Conference on Information
Systems, June 23-28, Portsmouth: UK, 2018.
[49] Treiblmaier, H., Putz, L.-M. and P. B. Lowry,
"Setting a definition, context, and research agenda for the
gamification of non-gaming systems", Transactions on
Human-Computer Interaction (THCI), forthcoming.
[50] Trochim, W.M.K. and J.P. Donnelly, Research
Methods Knowledge Base, Atomic Dog/Cengage Learning,
Mason, Ohio, 2008.
[51] van Houtte, M., "Why boys achieve less at school than
girls: The difference between boys' and girls' academic
culture", Educational Studies, 30(2), 2004, pp. 159–173.
[52] Warmelink, H., J. Koivisto, I.S. Mayer, M. Vesa, and
J. Hamari, "Gamification of the work floor: A literature
review of gamifying production and logistics operations",
51st Hawaii International Conference on System Sciences,
Big Island, HI, 3 - 6 January. 2018.
[53] Wieland, A., Durach, C. F., Kembro, J., and
Treiblmaier, H. "Statistical and Judgemental Criteria for
Scale Purification", Supply Chain Management: An
International Journal, 22(4), 2017, pp. 321-328.
[54] Wood, L. C. and T. Reiners, "Gamification in
logistics and supply chain education: Extending active
learning". In P. Kommers, T. Issa, & P. Isaías (eds.), IADIS
International Conference on Internet Technologies &
Society 2012, pp. 101–108.
[55] Yanuschik, O.V., E.G. Pakhomova, and K. Batbold,
"E-learning as a way to improve the quality of educational
for International students", Procedia - Social and
Behavioral Sciences, 215, 2015, pp. 147–155.
Page 1464
Appendix
Table 4: Questionnaire
Question
Grading
Which is the largest European
inland port in terms of total
cargo volume?
(single choice)
1 point for correct
answer
What percentage of the modal
split was used for inland
waterway transport in Europe
in 2014? (open question: 0-
100%)
1 point for correct
answer
Which of the following types
of goods are appropriate for
inland waterway transport?
(multiple choice)
1 point = one answer
correct
2 points = two
answers correct;
3 points = all answers
right (all scores = 3
points)
1 point deduction for
every incorrect answer
What percentage of the
potential cargo volume of the
Danube are currently used for
freight transport? (open
question: 0-100%)
1 point for correct
answer
Which of the following key
characteristics describe the
new logistics concept of
‘synchromodality’? (multiple
choice)
1 point = one answer
correct
2 points = two
answers correct;
3 points = all answers
right (all scores = 3
points)
1 point deduction for
every incorrect answer
What was the total cargo
volume transported in 2014 in
the European Union on inland
waterways? (single choice)
1 point for correct
answer
How many trucks are
substituted by one common
inland vessel of the Danube?
(open question)
1 point for correct
answer
Table 5: Schedules of the gamified &
non-gamified workshops
Non-gamified Workshop
Gamified Workshop
09:45-
10:00
Assessment (O1)
09:45-
10:00
Assessment
(O1)
10:00-
12:00
Traditional
Lecture ‘Inland
Waterway
Transport’
10:00-
11:00
Interactive
Lecture ‘Inland
Waterway
Transport’
11:00-
11:30
Transport
Calculation
11:30-
12:00
LEGO
Simulation
Game
12:00-
12:15
Container Quiz
12:00-
12:15
Container Quiz
12:15-
13:00
Break
12:15-
13:00
Break
13:00-
13:30
Transport
Calculation
13:00-
14:00
Future Transport
Ideas
Award
Ceremony
13:30-
14:00
Demonstration of
Simulation game
14:00-
15:00
Company Visit,
Port visit
14:00-
15:00
Company Visit,
Visit of the Port
15:00-
15:20
Break
15:00-
15:20
Break
15:20-
15:45
Assessment (O2)
15:00-
15:45
Assessment
(O2)
Table 6: Game elements in the gamified workshop
Gamified Workshop
Game Elements
10:00-
11:00
Interactive Lecture
Time Constraint
11:00-
11:30
Calculation
Time Constraint
11:30-
12:00
LEGO Simulation
Game
Storytelling
Time Constraint
12:00-
12:15
Container Quiz
Time Constraint
13:00-
14:00
Future Transport
Ideas
Award Ceremony
Storytelling
Time Constraint
Rewards: Price & Badges
Whole day
(in each exercise)
Leaderboard
Immediate feedback
Clear goals
Competition & Cooperation
Page 1465
