Conference PaperPDF Available

For whom should we gamify? Insights on the users' intentions and context towards gamification in education

Authors:

Abstract and Figures

Gamification design in educational environments is not trivial and many variables need to be considered to achieve positive outcomes. Often, educators and designers do not know when the students' intentions on the use of gamified environments might influence their experience. Based on this premise, this paper describes an exploratory study on the users' intention to use gamification, focusing on its influence in the field of education. We conducted a survey study with participants (N=1.692) and analysed their answers using unsupervised data mining techniques. As a result, we obtained empirical evidence showing that demographic and contextual variables influence (positively and negatively) people's intention to use gamification. This evidence can support designers and educators better understand whether and when they should or should not gamify a learning environment.
Content may be subject to copyright.
For whom should we gamify? Insights on the users’
intentions and context towards gamification in education
Armando M. Toda¹, Filipe D. Pereira³, Ana C. T. Klock4, Wilk Oliveira¹, Paula T.
Palomino¹, Luiz Rodrigues¹, Elaine H. T. Oliveira6, Isabela Gasparini5, Alexandra
I. Cristea², Seiji Isotani¹
1 Universidade de São Paulo (ICMC - USP) São Carlos, SP, Brazil
2 Durham University - Durham, U.K
3 Universidade Federal de Roraima (UFRR) Boa Vista, RR, Brazil
4 Universidade Federal do Rio Grande do Sul (UFRGS) Boa Vista, RR, Brazil
5 Universidade do Estado de Santa Catarina (UDESC) Joinville, SC, Brazil
6 Universidade Federal do Amazonas (UFAM) Manaus, AM, Brazil
armando.toda@usp.br
Abstract. Gamification design in educational environments is not trivial and
many variables need to be considered to achieve positive outcomes. Often,
educators and designers do not know when the students’ intentions on the use
of gamified environments might influence their experience. Based on this
premise, this paper describes an exploratory study on the users’ intention to use
gamification, focusing on its influence in the field of education. We conducted
a survey study with participants (N=1.692) and analysed their answers using
unsupervised data mining techniques. As a result, we obtained empirical
evidence showing that demographic and contextual variables influence
(positively and negatively) people’s intention to use gamification. This evidence
can support designers and educators better understand whether and when they
should or should not gamify a learning environment.
1. Introduction
The use of gamification in education has become a trend in the last decade [Deterding et
al. 2011; Klock et al. 2020]. Recent literature studies indicate that gamification in the
education domain has mixed results. From positive effects, such as increasing students’
motivation and engagement, to negative outcomes, such as undesired behaviours and loss
of motivation [Dichev and Dicheva 2017]. Many researcher have pointed out that these
mixed effects are tied to the gamification design and context it is used [Dichev and
Dicheva 2017; Klock et al. 2018; Toda et al. 2018; Pereira et al. 2020b].
The positive outcomes of gamification attracted educators’ attention.
Nevertheless, due to lack of knowledge, time, and resources, these educators are often
discouraged to pursue a good design of gamification and apply it adequately together with
their current pedagogical practices [An et al. 2020]. Furthermore, gamification is context-
aware, which means that it is necessary to understand the contextual factors that permeate
the users’ routine to design gamification in their environment [Klock et al. 2020; Seaborn
and Fels 2014]. According to Savard and Mizoguchi (2019), context can be either
constructed of mental representations (internal context), or environment and
circumstances (external). Internal context reflects personal characteristics that could
IX Congresso Brasileiro de Informática na Educação (CBIE 2020)
Anais do XXXI Simpósio Brasileiro de Informática na Educação (SBIE 2020)
471DOI: 10.5753/cbie.sbie.2020.471
impact the learning process, e.g., people experience and knowledge within a certain
subject may influence their perception on understanding certain situations.
In previous studies, intentions on the use of gamification have been explored in
different contexts and through different perceptions. Hamari and Koivisto (2015)
analysed why people use health gamified applications and pointed out that usefulness,
ease of use, enjoyment and playfulness are associated with a positive intention to use. In
Rodrigues et al. (2016), the authors investigated the intention of use in e-banking context.
According to their results, socialness leads to a positive intention of use, and the intention
of use has positive influence in the users’ perception. As we can observe, in both studies,
the positive intention of use leads to positive attitudes towards a certain field.
To date, we did not find any studies that analysed the intention of use in the field
of education, nor studies that analysed how previous knowledge and context influence
towards that intention. This information is important to educators, to understand when
and for whom they should gamify learning environments, since these design decisions
might influence the students’ perception when interacting with the learning environment
[Klock et al. 2020].
Thus, we aim at providing insights to the existing body of knowledge on
gamification by pursuing the following research question: How users’ demographics and
contextual characteristics influence the positive intention towards gamification in
education? To answer this question, we conducted a survey study (N = 1.692 people) and
analysed through a quantitative approach applying unsupervised data mining methods
namely, Association rules (AR) and clustering, to find patterns within the dataset. AR
analyses the relations between variables and clusters can provide an overall analysis that
can be translated into patterns [Agrawal et al. 1993; MacQueen and Others 1967].
Through these algorithms we can understand how these variables might influence the
gamification intention of use. Our findings include empirical evidence based on real data
that can support the decision-making process of educators to know when and for whom
to gamify learning environments. We also provide insights on how users’ perceptions can
be explored to further increase the acceptance of gamified systems.
2. Methods and tools
To conduct this research, we opted to follow an exploratory approach, since the objective
is to verify the possible relations between the users’ intentions and their context. Through
this approach, we might provide new research questions to be explored in future studies.
We conducted this approach using a survey, since it allows us to gather a considerable
amount of user answers and is also a low-cost solution [Lazar et al. 2017]. We divided
this approach in three steps, considering: data collection; analysis; and report.
For the data collection, we designed a questionnaire containing 12 questions that
aimed to collect demographic (e.g., gender, age and country in which the respondent
resides), and contextual variables (concerned with the users’ background with
gamification applications and games), as well as the intentions of using gamification in
different fields (work environment, routine, health and education).
These intentions were chosen based on the popularity of gamification in those
fields [Vargas-Enriquez et al. 2015]. The intention of using gamification questions
followed a template of What would be your intention in using gamification in your
[field]” using a 5-point Likert scale [Likert 1932] from 1 Would not use at all” (negative
IX Congresso Brasileiro de Informática na Educação (CBIE 2020)
Anais do XXXI Simpósio Brasileiro de Informática na Educação (SBIE 2020)
472
intention) to 5 Would definitely use(positive intention). We opted to analyse four
different fields where gamification is usually applied and/or studied [Klock et al. 2020].
In this paper, our focus is on analysing the relations in the field of education. The
recruitment of participants was carried out through Amazon Mechanical Turk (which has
been considered a reliable platform for this kind of study [Bentley et al. 2020]) and social
networks.
In the contextual variables, we consider the experience of the user by asking what
they know about the concept of gamification, since the way an individual understands the
environment (in this case, gamification) may influence the way they perceive the context
[Savard and Mizoguchi 2019]. Concerning the concepts, we adopted three different
concepts of gamification, an Other field, and one I don’t know answer. For the
concepts, we adopted as the main definition It is the use of game elements outside of a
game[Deterding et al. 2011], another definition that is a partial concept “It is a process
to put games in non-gaming context”, and a misconception “It is the process of making
games[Deterding et al. 2011]. The “Other” concept could be defined by the participant.
Concerning other contextual variables, we have also asked the participants if they usually
play games, how many years they had contact with gamification, and which gamified
applications they might have used. This questionnaire was created under supervision of 3
experts in survey design.
To analyse the data, we used AR and clusters since these methods are used to find
patterns within a dataset and have been used in recent exploratory research concerned
with gamification and data-driven methods [Palomino et al. 2019; Pereira et al. 2020a].
AR were used to find the relations between the intentions and demographic/contextual
variables. These rules were measured and analysed based on their confidence, lift and
support, following previous studies found in the literature [Palomino et al. 2019]. Clusters
were used to identify general patterns; the number of clusters was defined by the knee
point detection. Clustering can be used to analyse the intra- and inter-distance between
cluster values marking the point of maximum curvature. To find this point, we used the
K-means algorithm in a range of values from 2 to 12 (we assume 12 can be our upper
boundary considering our data are on a Likert scale from 1 to 5) [Satopaa et al. 2011].
Moreover, with the goal of grouping similar individuals together into clusters, we
use the popular unsupervised machine-learning algorithm K-means, which can be used to
find subgroups with different profiles on Likert scale [1-5] data, as our intention variables.
To choose the best number of clusters (k), we employed, as said, the knee point detection
algorithm, which is a technique the can be used for automatic detection of the optimal k
by analysing the maximum curvature [Satopaa et al. 2011] for each k point. According to
Satopaa et al. [2011], the automatic k point detection algorithm is more appropriate than
the common (and sometimes misleading) selection by visual inspection (ad-hoc analysis).
As such, we fitted the K-means model with k values ranging from 2 to 12. Figure 1 shows
k on the x axis, whist on the y axis we show the distortion, which represents groups’
density (intra-cluster distance). As a result, the point with maximum curvature is five
(dashed vertical line).
We also calculated the silhouette coefficient (which is the mean ratio of intra-
cluster and nearest-cluster distance) using the same range for k (2-12). Despite it seeming
that k=2 or k=3 (highest values for silhouette score) might be the best values, again, five
was found as the optimal value using the knee point detection (dashed line in Figure 1b).
IX Congresso Brasileiro de Informática na Educação (CBIE 2020)
Anais do XXXI Simpósio Brasileiro de Informática na Educação (SBIE 2020)
473
In addition, as explained before, we used on the intention variables a Likert scale [1-5],
which likely would lead to one cluster for each Likert value and, hence, five subgroups
of different profiles. Thus, we opted to use k=5 based on our maximum curvature analysis
and because it seems more appropriate for our data scale and, hence, gives us more
nuances for analysis (five clusters instead of only two or three).
Figure 1. (a) On left, distance score; (b) On right, sillhouette score
Finally, to report the finding, we provided the complete data and steps of pre-
processing used in this study, alongside descriptive statistics, association rules and
clusters at the following link https://bit.ly/2EqVl8E .
4. Results and Discussions
4.1. Descriptive statistics
Initially, we collected 1.692 answers in 3 months (December 2019 February 2020).
Following, we pre-processed the data by: (a) arranging the ages in groups; (b)
standardising the concepts of gamification; (c) arranging the countries by continent. In
step (b), we identified 42 different concepts given by the users, that were analysed by two
independent judges to verify if these concepts fall into one of the previous categories or
Other”. The judges were both experts in the field of gamification, with more than 5 years
of experience. In the initial analysis, using a Cohen’s Kappa κ [Cohen 1960], the judges
achieved a low agreement (κ = 0,3) then, a third judge was invited. Based on the third
judge decision, 27 concepts were classified into one of the existing concepts and 15 were
considered outliers, then removed. After removing the outliers, we analysed a total of
1.631 valid answers. Concerning the demographic variables, users reported 7 different
genders, with the majority of individuals identifying as either Female (N = 838 | 51,4%)
or Male (N = 778 | 47,7%), followed by Prefer not to say (N = 8 | 0,5%), Genderqueer (N
= 1), and Non-binary (N = 6). The average age of our sample is 33,5 years (SD
1
= 10,5),
minimum age being 14 and maximum being 75. For the countries, the majority (66,5%)
were from North America. In cluster analysis, genders that were not Female nor Male
were considered as NaN due to the low sample that impacted significantly on the cluster
formation (less than 1%). In the same way, both Africa and Oceania were also removed
for cluster analysis, due the sample being less than 1% total.
1
Standard deviation
IX Congresso Brasileiro de Informática na Educação (CBIE 2020)
Anais do XXXI Simpósio Brasileiro de Informática na Educação (SBIE 2020)
474
As for the contextual variables, most (N = 1410 | 86,4%) of our sample usually
play games, while a few (N = 221 | 13,6%) stated they do not. Considering the concepts
of gamification, few respondents (N = 248 | 15,2%) assumed they did not know what the
definition was, while 226 (13,9%) respondents stated that gamification is the process of
making games (misconception). Following, 391 (24%) respondents believe in the partial
definition (process of putting games in non-gaming contexts) and a majority (N = 766,
47%) answered with the correct definition. Thus, in general, we can observe that most
of our respondents do not know the correct definition. When asked about previous
contact with gamification, we found a duality between their knowledge definition and
usage, since 740 (45,4%) respondents stated that they did not have a previous contact
with gamification, while 772 (47,3%) stated they had, and 119 (7,3%) affirmed they might
have had contact. In other words, this led us to believe that people that know the
concept of gamification might not know how to recognise a gamified application,
reinforcing the previous finding. We had added an optional question that aimed to
established which gamified applications these respondents might have used and the
majority (approx. 321 entries) answered Duolingo, an educational platform, followed by
TripAdvisor, a touristic guide (approx. 104 entries). Finally, concerning their experience
with gamification (in years), the average is 4 years (SD = 4,1), minimum being 0 and
maximum 30 years. Concerning the experience, the concept was coined in 2011 but
studies have reported that gamification is influenced by past events and practices
that go decades before 2011 [Nelson 2012].
Finally, considering the intention of use, we observed that education (ED) led to
a higher intention of use (63,5%, when summing scales 4 and 5 that are tied to positive
intention). In contrast, work environment (WE) translated into the higher negative
intention to use (25,9%, when summing scales 1 and 2). A summary of these findings can
be seen in Table 1.
Table 1. Intention of use. DR = Daily Routine; WE = Work Environment; ED = Education; HE =
Health
Intention (Scale | Proportion)
Field
1
2
%
3
%
4
%
5
%
DR
157
201
11,9
459
27,1
501
29,6
374
22,1
WE
224
215
12,7
406
24
472
27,9
375
22,2
ED
145
148
8,7
325
19,2
489
28,9
585
34,6
HE
196
177
10,5
379
22,4
466
27,5
474
28
4.2. Association Rules and Clusters
To mine the AR, we used the R package arules [Hahsler et al. 2007]. Using a minimum
support and confidence of 0,1 we found 723 rules: maximum support of 0,54 rule 243
(when the user is from the United States, they usually play games); maximum
confidence of 0,96 rule 491 (when the user gender is male, and they have positive
intention in using gamification in daily routine, they usually play games); and
maximum lift of 3,78 rule 610 (when the user has a maximum intention to use
gamification in their work environment, health and education, they also have
maximum intention to use gamification in their daily routine).
IX Congresso Brasileiro de Informática na Educação (CBIE 2020)
Anais do XXXI Simpósio Brasileiro de Informática na Educação (SBIE 2020)
475
Concerned with the intention to use in education, we found 239 rules. Considering
the positive intention (Likert scale 4 or 5), support (> 0,1), confidence (> 0,8) and lift
(>1,3) we can summarise the number of rules to 16 (Rules 608, 254, 617, 621, 263, 629,
633, 296, 626, 273, 642, 309, 306, 650, 23 and 276). Through these rules, we can find
contextual variables linked to the intention to use in education, according to our data,
people who usually play games, had previous contact with gamified applications and
had previous knowledge on what gamification is have a positive intention to use it in
education (Likert scale = 5). In fact, the positive intention of use in other fields also
impact the intention to use in education.
Concerned with the neutral or negative intention, we also found 16 rules related
to the neutral intention (Likert scale = 3), but none of these rules followed the previous
values for confidence (>0,8) and lift (1,3). The information presented in these rules is that
people who usually play games but did not have previous contact with gamification
have neutral intention to use it in education.
In the five clusters that were generated, we analysed the Mean and SD observing
some profiles within our sample: Those who are indifferent (In white, Mean = 3) towards
the use of gamification in education (Cluster 1); those who have positive intentions (In
blue, Mean > 3) to use gamification in education (Clusters 2, 4 and 5); and those who
have negative intentions (In red, Mean < 3) to use gamification in education (Cluster 3).
The summary of the results can be seen in Table 2, and a summary of the Clusters can be
seen on Figure 2.
Table 2. Cluster Analysis. DR = Daily Routine; ED = Education; HE = Health; WE = Work
environment. In RED: Lowest value(s); In BLUE: Highest value(s).
Variables
Cluster Labels
Intentions
C1
SD
C2
SD
C3
SD
C4
SD
C5
SD
DR
2,84
0,74
4,7
0,49
1,5
0,71
3,75
0,73
3,61
0,9
ED
3,06
0,85
4,84
0,42
1,6
0,8
4,19
0,75
4,24
0,67
HE
2,69
0,85
4,88
0,33
1,43
0,71
3,63
0,74
4,3
0,65
WE
2,87
0,8
4,7
0,51
1,28
0,52
4,11
0,5
2,37
0,73
In Cluster 1 (C1), indifferent intentions can be observed; we can also observe that
people in this group tend to have a negative intention to use gamification in other fields.
Most of these people usually play games, know what gamification is, but believe they did
not have a previous contact with gamification. On demographics, gender distribution is
almost equal, they are between 20 and 30 years and the majority lives in North America.
For the positive intentions, we can observe that Clusters 2 and 4 (C2 and C4) have
similar analysis. Both clusters consider a positive intention to use gamification in other
fields alongside education (Cluster 4 having a lesser positive intention in DR and HE).
Considering their contexts, both clusters are composed of people who usually play games
and know what gamification is; however Cluster 2 has more people that had previous
contact with gamification; while Cluster 4 is almost balanced between people who had
and did not have previous contact with gamification. For the demographics, both clusters
are also remarkably similar in gender distribution, differing slightly in the age groups and
IX Congresso Brasileiro de Informática na Educação (CBIE 2020)
Anais do XXXI Simpósio Brasileiro de Informática na Educação (SBIE 2020)
476
continent, where Cluster 4 has the smallest ratio of North Americans and highest rate of
South Americans.
Figure 2. Clusters variables distribution
Considering Cluster 5 (C5, also positive intention to use gamification in
education), we can observe a negative intention towards WE. The context of this cluster
is similar to Clusters 2 and 4, with people who usually play games, know what
gamification is and had previous contact with gamification. Although, when analysing
the demographics, we can observe this cluster has more female respondents than males.
This cluster has also the slightest rate of people above 40 years. Geographical distribution
is similar to the previous clusters.
Finally, considering the negative intentions towards education it is possible to
observe that the whole Cluster 3 (C3) replicates this negative intention towards other
fields. In other words, this Cluster is composed of people who do not want to use
gamification at all, it is composed of people who usually play games, but do not know
well what gamification is about (highest rate of people who assumed they do not know
the concept of gamification or knew it partially). They also believe they have had no
previous contact with gamification. Considering their demographics, we can observe an
equal gender distribution, with people from all age groups and a majority of North
Americans.
In summary, AR and clustering provided similar information towards the context
of our sample, which means that previous contact with gamification, knowledge of the
concept and habit of playing games do influence the intention to use in educational
environments. This information can be used by teachers, instructors, and other educators
to know when to gamify. Our demographic analysis did not present significant differences
except for Cluster 5, in which most of the sample is composed by female respondents.
These results might have influenced users’ response towards previous used applications,
where most of the respondents (N > 300) used Duolingo as an example of a gamified
application, which is an educational environment.
IX Congresso Brasileiro de Informática na Educação (CBIE 2020)
Anais do XXXI Simpósio Brasileiro de Informática na Educação (SBIE 2020)
477
4.3. Discussion
This work provides insights on how the users’ context and demographics influence their
intention of use in gamification in education. Through this study, we add other variables
(previous knowledge on gamification, previous use of gamified applications, and playing
habits) that might be important to be considered when the designer and/or educator think
about gamifying their learning environments, which is not often considered when
designing gamification, but do impact on the users’ experience [Hamari 2015; Rodrigues
et al. 2016].
In future studies, researchers might ask the students about their intentions,
knowledge and/or playing habits, to understand if that really influences and has a positive
or negative impact on gamification. Another future research proposal would be
identifying how culture (in this case, the country where the person resides in) is related
to these factors as well, since culture is not a variable that is considered too often in the
gamification empirical literature [Klock et al. 2020].
4.4. Limitations
During the design and implementation of this work we faced some limitations. Some of
these limitations are concerned with the way we collected the users’ intention of use,
which could have been done through validated instruments, such as the Technology
Acceptance Model [Davis 1989]. However, due its complexity and aiming at a broader
public, we opted to use a single question self-assessing the intention of use through a
Likert Scale, which is used to measure abstract ideas. Another limitation is the
geographical distribution of our work, which might have been influenced by using
Amazon Mechanical Turk; we could not control this variable without increasing the
overall cost of this research. This could be enhanced or explored in future works.
5. Conclusion and Future Works
In this work, we focused on exploring and analysing the influence of contextual variables
over intention to use gamification in educational environments. Through the data
collected in our survey, we provided the following empirical contributions: (I) evidence
that context (previous knowledge, habit of playing games, and contact with gamification)
influence the intention to use; (II) and evidence that specific demographic characteristics
do not play a major role in the intention to use.
We believe this analysis could be further explored in future works by increasing
the number of respondents from different countries/continents, as well as different
genders, to increase diversity. Finally, another work would be exploring these contextual
variables within the design of gamification, as something to aid in the decision-making
process by designers and other people who want to gamify a learning environment.
Acknowledgements
The authors would like to thank FAPESP (Projects 2016/02765- 2; 2018/11180-3;
2018/15917-0; 2018/07688-1), Capes and CNPq for the funding provided. This research,
was carried out within the scope of the Samsung-UFAM Project for Education and
Research (SUPER), according to Article 48 of Decree nº 6.008/2006 (SUFRAMA), was
partially funded by Samsung Electronics of Amazonia Ltda., under the terms of Federal
Law nº 8.387/1991, through agreements 001/2020 and 003/2019, signed with the Federal
IX Congresso Brasileiro de Informática na Educação (CBIE 2020)
Anais do XXXI Simpósio Brasileiro de Informática na Educação (SBIE 2020)
478
University of Amazonas and FAEPI, Brazil. The authors also would like to thank
FAPESC (public call FAPESC/CNPq No. 06/2016 support the infrastructure of CTI for
young researchers, project T.O. No.: 2017TR1755 - Ambientes Inteligentes Educacionais
com Integração de Técnicas de Learning Analytics e de Gamificação).
References
Agrawal, R., Imieliński, T. and Swami, A. (1993). Mining association rules between sets
of items in large databases. http://portal.acm.org/citation.cfm?doid=170035.170072,
[accessed on Oct 22].
An, Y., Zhu, M., Bonk, C. J. and Lin, L. (6 jun 2020). Exploring instructors’ perspectives,
practices, and perceived support needs and barriers related to the gamification of MOOCs.
Journal of Computing in Higher Education, p. 121.
Bentley, F., Neill, K. O., Quehl, K. and Lottridge, D. (21 apr 2020). Exploring the Quality
, Efficiency , and Representative Nature of Responses Across Multiple Survey Panels. In
Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems. .
ACM. https://dl.acm.org/doi/10.1145/3313831.3376671, [accessed on Jun 3].
Cohen, J. (1 apr 1960). A coefficient of agreement for nominal scales. Educational and
Psychological Measurement, v. 20, n. 1, p. 3746.
Davis, F. D. (sep 1989). Perceived Usefulness, Perceived Ease of Use, and User
Acceptance of Information Technology. MIS Quarterly, v. 13, n. 3, p. 319.
Deterding, S., Sicart, M., Nacke, L., O’Hara, K. and Dixon, D. (2011). From Game
Design Elements to Gamefulness: Defining “Gamification.” Proceedings of the 2011
annual conference extended abstracts on Human factors in computing systems - CHI EA
’11, p. 2425.
Dichev, C. and Dicheva, D. (20 dec 2017). Gamifying education: what is known, what is
believed and what remains uncertain: a critical review. International Journal of
Educational Technology in Higher Education. Nature Publishing Group.
http://educationaltechnologyjournal.springeropen.com/articles/10.1186/s41239-017-
0042-5, [accessed on Apr 17].
Hahsler, M., Grün, B. and Hornik, K. (2007). Introduction to arules--mining association
rules and frequent item sets. SIGKDD Explor, v. 2, n. 4, p. 128.
Hamari, J. (2015). Why Do People Buy Virtual Goods ? Attitude towards Virtual Good
Purchases versus Game Enjoyment.
Hamari, J. and Koivisto, J. (1 aug 2015). Why do people use gamification services?
International Journal of Information Management, v. 35, n. 4, p. 419431.
Klock, A. C. T., Gasparini, I., Pimenta, M. S. and Hamari, J. (13 jun 2020). Tailored
gamification: A review of literature. International Journal of Human-Computer Studies,
p. 102495.
Klock, A. C. T., Ogawa, A. N., Gasparini, I. and Pimenta, M. S. (2018). Does
gamification matter? A systematic mapping about the evaluation of gamification in
educational environments. In Proceedings of the 33rd Annual ACM Symposium on
Applied Computing. . ACM Press.
http://dl.acm.org/citation.cfm?doid=3167132.3167347, [accessed on Jan 8].
IX Congresso Brasileiro de Informática na Educação (CBIE 2020)
Anais do XXXI Simpósio Brasileiro de Informática na Educação (SBIE 2020)
479
Lazar, J., Feng, J. H. and Hochheiser, H. (2017). Research methods in human-computer
interaction. 2nd. ed. Morgan Kaufmann.
Likert, R. A. (1932). A technique for the measurement of attitudes. Archives of
Psychology, v. 140, n. 140, p. 4453.
MacQueen, J. and Others (1967). Some methods for classification and analysis of
multivariate observations. In Proceedings of the fifth Berkeley symposium on
mathematical statistics and probability.
Nelson, M. J. M. (2012). Soviet and American precursors to the gamification of work.
Proceeding of the 16th international academic MindTrek conference (pp. 23-26), p. 23
26.
Palomino, P. T., Toda, A., Oliveira, W., et al. (2019). Exploring Content Game Elements
to Support Gamification Design in Educational Systems : Narrative and Storytelling. In
Proceedings of the SBIE 2019.
Pereira, F. D., Oliveira, E. H. T., Oliveira, D. B. F., et al. (2020a). Using learning analytics
in the Amazonas: understanding students’ behaviour in introductory programming.
British Journal of Educational Technology, p. bjet.12953.
Pereira, F. D., Toda, A., Oliveira, E. H., Cristea, A. I., Isotani, S., Laranjeira, D., Almeida,
A., Mendonça, J. (2020b). Can We Use Gamification to Predict Students’ Performance?
A Case Study Supported by an Online Judge. In International Conference on Intelligent
Tutoring Systems (pp. 259-269). Springer, Cham.
Rodrigues, L. F., Oliveira, A. and Costa, C. J. (1 oct 2016). Playing seriously - How
gamification and social cues influence bank customers to use gamified e-business
applications. Computers in Human Behavior, v. 63, p. 392407.
Satopaa, V., Albrecht, J., Irwin, D. and Raghavan, B. (2011). Finding a" kneedle" in a
haystack: Detecting knee points in system behavior. In 2011 31st international
conference on distributed computing systems workshops.
Savard, I. and Mizoguchi, R. (18 dec 2019). Context or culture: what is the difference?
Research and Practice in Technology Enhanced Learning. SpringerOpen.
https://telrp.springeropen.com/articles/10.1186/s41039-019-0112-5, [accessed on May
25].
Seaborn, K. and Fels, D. I. (2014). Gamification in Theory and Action: A Survey.
Internatoinal Journal of Human-Computer Studies, v. 74, p. 1431.
Toda, A. M., Valle, P. H. D. D. and Isotani, S. (20 mar 2018). The Dark Side of
Gamification: An Overview of Negative Effects of Gamification in Education. In
Communications in Computer and Information Science. . Springer, Cham.
http://link.springer.com/10.1007/978-3-319-97934-2_9, [accessed on Oct 1].
Vargas-Enriquez, J., Garcia-Mundo, L., Genero, M. and Piattini, M. (sep 2015). A
Systematic Mapping Study on Gamified Software Quality. 2015 7th International
Conference on Games and Virtual Worlds for Serious Applications (VS-Games), p. 18.
IX Congresso Brasileiro de Informática na Educação (CBIE 2020)
Anais do XXXI Simpósio Brasileiro de Informática na Educação (SBIE 2020)
480
... However, studies on CS applications often focus on gamification's impact on behavioral learning outcomes, such as student performance (e.g., [5,8,20,22]), neglecting learners' motivation and context of use, despite those are inherently connected to learning [1,36,38,42]. Thus, there is a need to understand how gamification affects programming learning while also considering motivation, as well as the role of context, corroborating recent calls for better understanding how gamification works and the context's role aiming to improve gamification's positive outcomes [15,31,35]. ...
... The fact that the same gamification is unlikely to work for all users has been recently discussed, calling for the need of tailored gamification [40]. Within this context, recent studies have called for considering aspects related to the context and learning activities when tailoring gamification [9,10,30,32,38]. Our findings' implication to this vein is that learners' previous familiarity with the learning activity, which relates to the context, moderates how gamification impacts their intrinsic motivation. ...
Conference Paper
Full-text available
Programming is a complex, not trivial to learn and teach task, which gamification can facilitate. However, how gamification affects learning and the influence of context-related aspects on that effect demand research to better understand how and to whom gamification enhances programming learning. Therefore, we conducted an experimental study analyzing how gamification worked and the role of context-related aspects in terms of intervention duration and learners' familiarity with programming (i.e., the task's topic). It was a six-week study with 19 undergraduate students from an Algorithms class that measured their learning gains, intrinsic motivation, and number of completed quizzes. Mainly, we found gamification affected learning via intrinsic motivation, effect that depended on intervention duration and learners' familiarity with programming. That is, intrinsic motivation strongly predicted learning gains and gamification's effect on intrinsic motivation changed over time, decreasing from positive to negative as learners had less familiarity with programming. Thus, showing gamification can positively impact programming learning by improving students' intrinsic motivation, although that effect changes over time depending on one's previous familiarity with programming.
... Despite past research has studied similar questions Hanus and Fox (2015); Sanchez et al. (2020); Tsay et al. (2020), this one differs in two main perspectives, according to our best knowledge. First, research context: The literature indicates gamification's effect varies depending on the context Hamari et al. (2014); Liu et al. (2017); Toda et al. (2020), and a single research analyzed how gamification's effect changes over time when applied to CS1 students . Second, the gamification design. ...
Article
Full-text available
There are many claims that gamification (i.e., using game elements outside games) impact decreases over time (i.e., the novelty effect). Most studies analyzing this effect focused on extrinsic game elements, while fictional and collaborative competition have been recently recommended. Additionally, to the best of our knowledge, no long-term research has been carried out with STEM learners from introductory programming courses (CS1), a context that demands encouraging practice and mitigating motivation throughout the semester. Therefore, the main goal of this work is to better understand how the impact of a gamification design, featuring fictional and competitive-collaborative elements, changes over a 14-week period of time, when applied to CS1 courses taken by STEM students (N = 756). In an ecological setting, we followed a 2x7 quasi-experimental design, where Brazilian STEM students completed assignments in either a gamified or non-gamified version of the same system, which provided the measures (number of attempts, usage time, and system access) to assess user behavior at seven points in time. Results indicate changes in gamification’s impact that appear to follow a U-shaped pattern. Supporting the novelty effect, the gamification’s effect started to decrease after four weeks, decrease that lasted between two to six weeks. Interestingly, the gamification’s impact shifted to an uptrend between six and 10 weeks after the start of the intervention, partially recovering its contribution naturally. Thus, we found empirical evidence supporting that gamification likely suffers from the novelty effect, but also benefits from the familiarization effect, which contributes to an overall positive impact on students. These findings may provide some guidelines to inform practitioners about how long the initial contributions of gamification last, and how long they take to recover after some reduction in benefits. It can also help researchers to realize when to apply/evaluate interventions that use gamification by taking into consideration the novelty effect and, thereby, better understand the real impact of gamification on students’ behavior in the long run.
Conference Paper
Full-text available
O interesse na gamificação por professores e outros profissionais da educação vem aumentando nos últimos anos. Contudo, a literatura indica que dependendo da forma como a gamificação é planejada e executada os resultados podem beneficiar ou prejudicar à aprendizagem e motivação dos estudantes. Neste contexto, é fundamental desenvolver técnicas de design, aplicação e avaliação da gamificação específicas para o contexto educacional com o objetivo de maximizar os benefícios e mitigar potenciais problemas. Dessa forma, este trabalho apresenta parte das contribuições alcançadas neste trabalho de doutorado focando: (a) no desenvolvimento e validação da primeira taxonomia de elementos de jogos específica para o contexto educacional; (b) no framework conceitual para gamificar ambientes de aprendizagem testado e validado por profissionais do ensino; e (c) na uso de dados empíricos de centenas de usuários (N=1692) para criar estratégias inovadoras de design de gamificação por meio técnicas de mineração de dados. Os resultados obtidos viabilizaram tanto o uso adequado dos elementos de gamificação durante o processo de planejamento da gamificação quanto a correta aplicação da gamificação em ambientes de ensino.
Article
Full-text available
Gamification is increasingly becoming a pertinent aspect of any UI and UX design. However, a canonical dearth in research and application of gamification has been related to the role of individual differences in susceptibility to gamification and its varied designs. To address this gap, this study reviews the extant corpus of research on tailored gamification (42 studies). The findings of the review indicate that most studies on the field are mostly focused on user modeling for a future personalization, adaptation, or recommendation of game elements. This user model usually contains the users’ preferences of play (i.e., player types), and is mostly applied in educational settings. The main contributions of this paper are a standardized terminology of the game elements used in tailored gamification, the discussion on the most suitable game elements for each users’ characteristic, and a research agenda including dynamic modeling, exploring multiple characteristics simultaneously, and understanding the effects of other aspects of the interaction on user experience.
Article
Full-text available
This study explored instructors’ perceptions, interest, self-efficacy, perceived barriers, and support needs regarding gamification in MOOCs. Both quantitative and qualitative data were collected from an online survey and follow-up interviews. Most participants showed interest in gamification and indicated that they would consider utilizing gaming elements in their future MOOCs. Interestingly, they wanted to gamify their MOOCs mostly to increase social interactions and student retention. Significant differences in self-efficacy and perceptions of gamification were found between younger participants and older participants. The results also revealed significant differences in interest and self-efficacy between participants with prior experience with gamification and those without prior experience. The major barriers to gamifying MOOCs included lack of time, limited knowledge, lack of funding, lack of fit between gamification and the course content, concerns about students’ perceptions of gamification, and concerns regarding the negative effects of gamification. Participants reported that they would need time and funding, guidance from gamification experts, examples of gamified MOOCs, more flexible MOOC platforms in order to successfully gamify their MOOCs.
Article
Full-text available
Abstract Literature can sometimes tend to present context and culture almost as synonyms. This creates ambiguity, which can complicate the consideration of contextual and cultural variables in instructional design, learning, and teaching. From an ontological point of view, some clarification of these two concepts is essential as each may influence learning and teaching in different ways. Moreover, since context and culture are interconnected to a certain degree, one may influence the other. It is crucial to make a clear distinction between these two concepts in the knowledge models used in intelligent tutoring systems and distance education systems if we want to facilitate (1) their consideration in pedagogical scenarios, and (2) the accumulation of knowledge about different contexts and cultures. This article offers an interpretation of the difference between these two concepts, presenting context as a substrate of culture. Contextual issues in the learning ecology are also discussed, based on this distinction.
Conference Paper
Full-text available
Several studies on gamification applied to educational systems aim at encouraging students to do certain tasks and improving their learning. According to the literature, most gamification frameworks are structural (e.g. scoring systems, ranking, etc.), with few content-related frameworks. To the best of our knowledge, there is no narrative framework available. Therefore this paper analyses data obtained from a survey about the students' preferred game elements in an educational context, with focus on Association Rules found concerning Narrative and Storytelling elements. We show that Narrative and Story-telling are tightly related and provide insights of their use in groups with other game elements, enabling the creation of gamified strategies based on these aspects .
Chapter
Full-text available
Gamification has a great number of studies in the education area since the emergence of the term. However, there is a lack of primary and secondary studies that explore the negative effects that gamification may have on learners, and lack of studies that analyze the gamification design that are linked to those negative effects. Based on this premise, we aim at answering the following research question “What are the negative effects that can occur in gamification when applied to educational contexts?”. We seek to answer this question by analyzing the negative effects that are associated to gamification and the gamified learning design that are linked with them. To answer this question, we conducted a systematic mapping study to identify these negative effects. Based on the studies that were analyzed, we identified and mapped 4 negative effects and their gameful design. Loss of Performance was the most occurring effect and Leaderboard the most cited game design element, among other 11 elements. Moreover, elements and effects were linked in order to identify how these elements may have influenced on these outcomes. Based on our results, we found out that the game design may lead to a negative impact. For instance, Leaderboards are strongly associated to many negative effects mapped in this work. This result is corroborated by the psychology literature regarding ranking systems within learning environments. We believe our work may be useful to guide gamification instructors and specialists to avoid those negative effects in education contexts, by avoiding some game design elements settings.
Conference Paper
Full-text available
Gamification is a technique that reuses game design and elements in others contexts, such as in e-commerces systems and virtual learning environments. When applied in the educational area, the gamification purpose is to promote a better user experience by improving students' motivation and engagement. There are already several studies applying gamification in the educational systems, and many ways to evaluate its effects on students. Thus, the aim of this work is to investigate how these studies evaluate gamification and compare its usage in educational environments through a systematic mapping. From the 832 studies returned by the search engines, only 20 of them met the defined selection criteria. As a result, these studies compared and evaluated gamification through students' interaction, performance, and user experience, based on their activities and answers in satisfaction surveys and tests.
Article
Full-text available
Gamification of education is a developing approach for increasing learners’ motivation and engagement by incorporating game design elements in educational environments. With the growing popularity of gamification and yet mixed success of its application in educational contexts, the current review is aiming to shed a more realistic light on the research in this field by focusing on empirical evidence rather than on potentialities, beliefs or preferences. Accordingly, it critically examines the advancement in gamifying education. The discussion is structured around the used gamification mechanisms, the gamified subjects, the type of gamified learning activities, and the study goals, with an emphasis on the reliability and validity of the reported outcomes. To improve our understanding and offer a more realistic picture of the progress of gamification in education, consistent with the presented evidence, we examine both the outcomes reported in the papers and how they have been obtained. While the gamification in education is still a growing phenomenon, the review reveals that (i) insufficient evidence exists to support the long-term benefits of gamification in educational contexts; (ii) the practice of gamifying learning has outpaced researchers’ understanding of its mechanisms and methods; (iii) the knowledge of how to gamify an activity in accordance with the specifics of the educational context is still limited. The review highlights the need for systematically designed studies and rigorously tested approaches confirming the educational benefits of gamification, if gamified learning is to become a recognized instructional approach.
Article
Tools for automatic grading programming assignments, also known as Online Judges, have been widely used to support computer science (CS) courses. Nevertheless, few studies have used these tools to acquire and analyse interaction data to better understand the students’ performance and behaviours, often due to data availability or inadequate granularity. To address this problem, we propose an Online Judge called CodeBench, which allows for fine‐grained data collection of student interactions, at the level of, eg, keystrokes, number of submissions, and grades. We deployed CodeBench for 3 years (2016–18) and collected data from 2058 students from 16 introductory computer science (CS1) courses, on which we have carried out fine‐grained learning analytics, towards early detection of effective/ineffective behaviours regarding learning CS concepts. Results extract clear behavioural classes of CS1 students, significantly differentiated both semantically and statistically, enabling us to better explain how student behaviours during programming have influenced learning outcomes. Finally, we also identify behaviours that can guide novice students to improve their learning performance, which can be used for interventions. We believe this work is a step forward towards enhancing Online Judges and helping teachers and students improve their CS1 teaching/learning practices.
Book
Research Methods in Human-Computer Interaction is a comprehensive guide to performing research and is essential reading for both quantitative and qualitative methods. Since the first edition was published in 2009, the book has been adopted for use at leading universities around the world, including Harvard University, Carnegie-Mellon University, the University of Washington, the University of Toronto, HiOA (Norway), KTH (Sweden), Tel Aviv University (Israel), and many others. Chapters cover a broad range of topics relevant to the collection and analysis of HCI data, going beyond experimental design and surveys, to cover ethnography, diaries, physiological measurements, case studies, crowdsourcing, and other essential elements in the well-informed HCI researcher's toolkit. Continual technological evolution has led to an explosion of new techniques and a need for this updated 2nd edition, to reflect the most recent research in the field and newer trends in research methodology. This Research Methods in HCI revision contains updates throughout, including more detail on statistical tests, coding qualitative data, and data collection via mobile devices and sensors. Other new material covers performing research with children, older adults, and people with cognitive impairments.