Content uploaded by Armando Toda
Author content
All content in this area was uploaded by Armando Toda on Nov 24, 2020
Content may be subject to copyright.
For whom should we gamify? Insights on the users’
intentions and context towards gamification in education
Armando M. Toda¹, Filipe D. Pereira³, Ana C. T. Klock4, Wilk Oliveira¹, Paula T.
Palomino¹, Luiz Rodrigues¹, Elaine H. T. Oliveira6, Isabela Gasparini5, Alexandra
I. Cristea², Seiji Isotani¹
1 Universidade de São Paulo (ICMC - USP) – São Carlos, SP, Brazil
2 Durham University - Durham, U.K
3 Universidade Federal de Roraima (UFRR) – Boa Vista, RR, Brazil
4 Universidade Federal do Rio Grande do Sul (UFRGS) – Boa Vista, RR, Brazil
5 Universidade do Estado de Santa Catarina (UDESC) – Joinville, SC, Brazil
6 Universidade Federal do Amazonas (UFAM) – Manaus, AM, Brazil
armando.toda@usp.br
Abstract. Gamification design in educational environments is not trivial and
many variables need to be considered to achieve positive outcomes. Often,
educators and designers do not know when the students’ intentions on the use
of gamified environments might influence their experience. Based on this
premise, this paper describes an exploratory study on the users’ intention to use
gamification, focusing on its influence in the field of education. We conducted
a survey study with participants (N=1.692) and analysed their answers using
unsupervised data mining techniques. As a result, we obtained empirical
evidence showing that demographic and contextual variables influence
(positively and negatively) people’s intention to use gamification. This evidence
can support designers and educators better understand whether and when they
should or should not gamify a learning environment.
1. Introduction
The use of gamification in education has become a trend in the last decade [Deterding et
al. 2011; Klock et al. 2020]. Recent literature studies indicate that gamification in the
education domain has mixed results. From positive effects, such as increasing students’
motivation and engagement, to negative outcomes, such as undesired behaviours and loss
of motivation [Dichev and Dicheva 2017]. Many researcher have pointed out that these
mixed effects are tied to the gamification design and context it is used [Dichev and
Dicheva 2017; Klock et al. 2018; Toda et al. 2018; Pereira et al. 2020b].
The positive outcomes of gamification attracted educators’ attention.
Nevertheless, due to lack of knowledge, time, and resources, these educators are often
discouraged to pursue a good design of gamification and apply it adequately together with
their current pedagogical practices [An et al. 2020]. Furthermore, gamification is context-
aware, which means that it is necessary to understand the contextual factors that permeate
the users’ routine to design gamification in their environment [Klock et al. 2020; Seaborn
and Fels 2014]. According to Savard and Mizoguchi (2019), context can be either
constructed of mental representations (internal context), or environment and
circumstances (external). Internal context reflects personal characteristics that could
IX Congresso Brasileiro de Informática na Educação (CBIE 2020)
Anais do XXXI Simpósio Brasileiro de Informática na Educação (SBIE 2020)
471DOI: 10.5753/cbie.sbie.2020.471
impact the learning process, e.g., people experience and knowledge within a certain
subject may influence their perception on understanding certain situations.
In previous studies, intentions on the use of gamification have been explored in
different contexts and through different perceptions. Hamari and Koivisto (2015)
analysed why people use health gamified applications and pointed out that usefulness,
ease of use, enjoyment and playfulness are associated with a positive intention to use. In
Rodrigues et al. (2016), the authors investigated the intention of use in e-banking context.
According to their results, socialness leads to a positive intention of use, and the intention
of use has positive influence in the users’ perception. As we can observe, in both studies,
the positive intention of use leads to positive attitudes towards a certain field.
To date, we did not find any studies that analysed the intention of use in the field
of education, nor studies that analysed how previous knowledge and context influence
towards that intention. This information is important to educators, to understand when
and for whom they should gamify learning environments, since these design decisions
might influence the students’ perception when interacting with the learning environment
[Klock et al. 2020].
Thus, we aim at providing insights to the existing body of knowledge on
gamification by pursuing the following research question: How users’ demographics and
contextual characteristics influence the positive intention towards gamification in
education? To answer this question, we conducted a survey study (N = 1.692 people) and
analysed through a quantitative approach applying unsupervised data mining methods
namely, Association rules (AR) and clustering, to find patterns within the dataset. AR
analyses the relations between variables and clusters can provide an overall analysis that
can be translated into patterns [Agrawal et al. 1993; MacQueen and Others 1967].
Through these algorithms we can understand how these variables might influence the
gamification intention of use. Our findings include empirical evidence based on real data
that can support the decision-making process of educators to know when and for whom
to gamify learning environments. We also provide insights on how users’ perceptions can
be explored to further increase the acceptance of gamified systems.
2. Methods and tools
To conduct this research, we opted to follow an exploratory approach, since the objective
is to verify the possible relations between the users’ intentions and their context. Through
this approach, we might provide new research questions to be explored in future studies.
We conducted this approach using a survey, since it allows us to gather a considerable
amount of user answers and is also a low-cost solution [Lazar et al. 2017]. We divided
this approach in three steps, considering: data collection; analysis; and report.
For the data collection, we designed a questionnaire containing 12 questions that
aimed to collect demographic (e.g., gender, age and country in which the respondent
resides), and contextual variables (concerned with the users’ background with
gamification applications and games), as well as the intentions of using gamification in
different fields (work environment, routine, health and education).
These intentions were chosen based on the popularity of gamification in those
fields [Vargas-Enriquez et al. 2015]. The intention of using gamification questions
followed a template of “What would be your intention in using gamification in your
[field]” using a 5-point Likert scale [Likert 1932] from 1 “Would not use at all” (negative
IX Congresso Brasileiro de Informática na Educação (CBIE 2020)
Anais do XXXI Simpósio Brasileiro de Informática na Educação (SBIE 2020)
472
intention) to 5 “Would definitely use” (positive intention). We opted to analyse four
different fields where gamification is usually applied and/or studied [Klock et al. 2020].
In this paper, our focus is on analysing the relations in the field of education. The
recruitment of participants was carried out through Amazon Mechanical Turk (which has
been considered a reliable platform for this kind of study [Bentley et al. 2020]) and social
networks.
In the contextual variables, we consider the experience of the user by asking what
they know about the concept of gamification, since the way an individual understands the
environment (in this case, gamification) may influence the way they perceive the context
[Savard and Mizoguchi 2019]. Concerning the concepts, we adopted three different
concepts of gamification, an “Other” field, and one “I don’t know” answer. For the
concepts, we adopted as the main definition “It is the use of game elements outside of a
game” [Deterding et al. 2011], another definition that is a partial concept “It is a process
to put games in non-gaming context”, and a misconception “It is the process of making
games” [Deterding et al. 2011]. The “Other” concept could be defined by the participant.
Concerning other contextual variables, we have also asked the participants if they usually
play games, how many years they had contact with gamification, and which gamified
applications they might have used. This questionnaire was created under supervision of 3
experts in survey design.
To analyse the data, we used AR and clusters since these methods are used to find
patterns within a dataset and have been used in recent exploratory research concerned
with gamification and data-driven methods [Palomino et al. 2019; Pereira et al. 2020a].
AR were used to find the relations between the intentions and demographic/contextual
variables. These rules were measured and analysed based on their confidence, lift and
support, following previous studies found in the literature [Palomino et al. 2019]. Clusters
were used to identify general patterns; the number of clusters was defined by the knee
point detection. Clustering can be used to analyse the intra- and inter-distance between
cluster values marking the point of maximum curvature. To find this point, we used the
K-means algorithm in a range of values from 2 to 12 (we assume 12 can be our upper
boundary considering our data are on a Likert scale from 1 to 5) [Satopaa et al. 2011].
Moreover, with the goal of grouping similar individuals together into clusters, we
use the popular unsupervised machine-learning algorithm K-means, which can be used to
find subgroups with different profiles on Likert scale [1-5] data, as our intention variables.
To choose the best number of clusters (k), we employed, as said, the knee point detection
algorithm, which is a technique the can be used for automatic detection of the optimal k
by analysing the maximum curvature [Satopaa et al. 2011] for each k point. According to
Satopaa et al. [2011], the automatic k point detection algorithm is more appropriate than
the common (and sometimes misleading) selection by visual inspection (ad-hoc analysis).
As such, we fitted the K-means model with k values ranging from 2 to 12. Figure 1 shows
k on the x axis, whist on the y axis we show the distortion, which represents groups’
density (intra-cluster distance). As a result, the point with maximum curvature is five
(dashed vertical line).
We also calculated the silhouette coefficient (which is the mean ratio of intra-
cluster and nearest-cluster distance) using the same range for k (2-12). Despite it seeming
that k=2 or k=3 (highest values for silhouette score) might be the best values, again, five
was found as the optimal value using the knee point detection (dashed line in Figure 1b).
IX Congresso Brasileiro de Informática na Educação (CBIE 2020)
Anais do XXXI Simpósio Brasileiro de Informática na Educação (SBIE 2020)
473
In addition, as explained before, we used on the intention variables a Likert scale [1-5],
which likely would lead to one cluster for each Likert value and, hence, five subgroups
of different profiles. Thus, we opted to use k=5 based on our maximum curvature analysis
and because it seems more appropriate for our data scale and, hence, gives us more
nuances for analysis (five clusters instead of only two or three).
Figure 1. (a) On left, distance score; (b) On right, sillhouette score
Finally, to report the finding, we provided the complete data and steps of pre-
processing used in this study, alongside descriptive statistics, association rules and
clusters at the following link https://bit.ly/2EqVl8E .
4. Results and Discussions
4.1. Descriptive statistics
Initially, we collected 1.692 answers in 3 months (December 2019 – February 2020).
Following, we pre-processed the data by: (a) arranging the ages in groups; (b)
standardising the concepts of gamification; (c) arranging the countries by continent. In
step (b), we identified 42 different concepts given by the users, that were analysed by two
independent judges to verify if these concepts fall into one of the previous categories or
“Other”. The judges were both experts in the field of gamification, with more than 5 years
of experience. In the initial analysis, using a Cohen’s Kappa κ [Cohen 1960], the judges
achieved a low agreement (κ = 0,3) then, a third judge was invited. Based on the third
judge decision, 27 concepts were classified into one of the existing concepts and 15 were
considered outliers, then removed. After removing the outliers, we analysed a total of
1.631 valid answers. Concerning the demographic variables, users reported 7 different
genders, with the majority of individuals identifying as either Female (N = 838 | 51,4%)
or Male (N = 778 | 47,7%), followed by Prefer not to say (N = 8 | 0,5%), Genderqueer (N
= 1), and Non-binary (N = 6). The average age of our sample is 33,5 years (SD
1
= 10,5),
minimum age being 14 and maximum being 75. For the countries, the majority (66,5%)
were from North America. In cluster analysis, genders that were not Female nor Male
were considered as NaN due to the low sample that impacted significantly on the cluster
formation (less than 1%). In the same way, both Africa and Oceania were also removed
for cluster analysis, due the sample being less than 1% total.
1
Standard deviation
IX Congresso Brasileiro de Informática na Educação (CBIE 2020)
Anais do XXXI Simpósio Brasileiro de Informática na Educação (SBIE 2020)
474
As for the contextual variables, most (N = 1410 | 86,4%) of our sample usually
play games, while a few (N = 221 | 13,6%) stated they do not. Considering the concepts
of gamification, few respondents (N = 248 | 15,2%) assumed they did not know what the
definition was, while 226 (13,9%) respondents stated that gamification is the process of
making games (misconception). Following, 391 (24%) respondents believe in the partial
definition (process of putting games in non-gaming contexts) and a majority (N = 766,
47%) answered with the correct definition. Thus, in general, we can observe that most
of our respondents do not know the correct definition. When asked about previous
contact with gamification, we found a duality between their knowledge definition and
usage, since 740 (45,4%) respondents stated that they did not have a previous contact
with gamification, while 772 (47,3%) stated they had, and 119 (7,3%) affirmed they might
have had contact. In other words, this led us to believe that people that know the
concept of gamification might not know how to recognise a gamified application,
reinforcing the previous finding. We had added an optional question that aimed to
established which gamified applications these respondents might have used and the
majority (approx. 321 entries) answered Duolingo, an educational platform, followed by
TripAdvisor, a touristic guide (approx. 104 entries). Finally, concerning their experience
with gamification (in years), the average is 4 years (SD = 4,1), minimum being 0 and
maximum 30 years. Concerning the experience, the concept was coined in 2011 but
studies have reported that gamification is influenced by past events and practices
that go decades before 2011 [Nelson 2012].
Finally, considering the intention of use, we observed that education (ED) led to
a higher intention of use (63,5%, when summing scales 4 and 5 that are tied to positive
intention). In contrast, work environment (WE) translated into the higher negative
intention to use (25,9%, when summing scales 1 and 2). A summary of these findings can
be seen in Table 1.
Table 1. Intention of use. DR = Daily Routine; WE = Work Environment; ED = Education; HE =
Health
Intention (Scale | Proportion)
Field
1
%
2
%
3
%
4
%
5
%
DR
157
9,3
201
11,9
459
27,1
501
29,6
374
22,1
WE
224
13,2
215
12,7
406
24
472
27,9
375
22,2
ED
145
8,6
148
8,7
325
19,2
489
28,9
585
34,6
HE
196
11,6
177
10,5
379
22,4
466
27,5
474
28
4.2. Association Rules and Clusters
To mine the AR, we used the R package arules [Hahsler et al. 2007]. Using a minimum
support and confidence of 0,1 we found 723 rules: maximum support of 0,54 – rule 243
(when the user is from the United States, they usually play games); maximum
confidence of 0,96 – rule 491 (when the user gender is male, and they have positive
intention in using gamification in daily routine, they usually play games); and
maximum lift of 3,78 – rule 610 (when the user has a maximum intention to use
gamification in their work environment, health and education, they also have
maximum intention to use gamification in their daily routine).
IX Congresso Brasileiro de Informática na Educação (CBIE 2020)
Anais do XXXI Simpósio Brasileiro de Informática na Educação (SBIE 2020)
475
Concerned with the intention to use in education, we found 239 rules. Considering
the positive intention (Likert scale 4 or 5), support (> 0,1), confidence (> 0,8) and lift
(>1,3) we can summarise the number of rules to 16 (Rules 608, 254, 617, 621, 263, 629,
633, 296, 626, 273, 642, 309, 306, 650, 23 and 276). Through these rules, we can find
contextual variables linked to the intention to use in education, according to our data,
people who usually play games, had previous contact with gamified applications and
had previous knowledge on what gamification is have a positive intention to use it in
education (Likert scale = 5). In fact, the positive intention of use in other fields also
impact the intention to use in education.
Concerned with the neutral or negative intention, we also found 16 rules related
to the neutral intention (Likert scale = 3), but none of these rules followed the previous
values for confidence (>0,8) and lift (1,3). The information presented in these rules is that
people who usually play games but did not have previous contact with gamification
have neutral intention to use it in education.
In the five clusters that were generated, we analysed the Mean and SD observing
some profiles within our sample: Those who are indifferent (In white, Mean = 3) towards
the use of gamification in education (Cluster 1); those who have positive intentions (In
blue, Mean > 3) to use gamification in education (Clusters 2, 4 and 5); and those who
have negative intentions (In red, Mean < 3) to use gamification in education (Cluster 3).
The summary of the results can be seen in Table 2, and a summary of the Clusters can be
seen on Figure 2.
Table 2. Cluster Analysis. DR = Daily Routine; ED = Education; HE = Health; WE = Work
environment. In RED: Lowest value(s); In BLUE: Highest value(s).
Variables
Cluster Labels
Intentions
C1
SD
C2
SD
C3
SD
C4
SD
C5
SD
DR
2,84
0,74
4,7
0,49
1,5
0,71
3,75
0,73
3,61
0,9
ED
3,06
0,85
4,84
0,42
1,6
0,8
4,19
0,75
4,24
0,67
HE
2,69
0,85
4,88
0,33
1,43
0,71
3,63
0,74
4,3
0,65
WE
2,87
0,8
4,7
0,51
1,28
0,52
4,11
0,5
2,37
0,73
In Cluster 1 (C1), indifferent intentions can be observed; we can also observe that
people in this group tend to have a negative intention to use gamification in other fields.
Most of these people usually play games, know what gamification is, but believe they did
not have a previous contact with gamification. On demographics, gender distribution is
almost equal, they are between 20 and 30 years and the majority lives in North America.
For the positive intentions, we can observe that Clusters 2 and 4 (C2 and C4) have
similar analysis. Both clusters consider a positive intention to use gamification in other
fields alongside education (Cluster 4 having a lesser positive intention in DR and HE).
Considering their contexts, both clusters are composed of people who usually play games
and know what gamification is; however Cluster 2 has more people that had previous
contact with gamification; while Cluster 4 is almost balanced between people who had
and did not have previous contact with gamification. For the demographics, both clusters
are also remarkably similar in gender distribution, differing slightly in the age groups and
IX Congresso Brasileiro de Informática na Educação (CBIE 2020)
Anais do XXXI Simpósio Brasileiro de Informática na Educação (SBIE 2020)
476
continent, where Cluster 4 has the smallest ratio of North Americans and highest rate of
South Americans.
Figure 2. Clusters’ variables distribution
Considering Cluster 5 (C5, also positive intention to use gamification in
education), we can observe a negative intention towards WE. The context of this cluster
is similar to Clusters 2 and 4, with people who usually play games, know what
gamification is and had previous contact with gamification. Although, when analysing
the demographics, we can observe this cluster has more female respondents than males.
This cluster has also the slightest rate of people above 40 years. Geographical distribution
is similar to the previous clusters.
Finally, considering the negative intentions towards education it is possible to
observe that the whole Cluster 3 (C3) replicates this negative intention towards other
fields. In other words, this Cluster is composed of people who do not want to use
gamification at all, it is composed of people who usually play games, but do not know
well what gamification is about (highest rate of people who assumed they do not know
the concept of gamification or knew it partially). They also believe they have had no
previous contact with gamification. Considering their demographics, we can observe an
equal gender distribution, with people from all age groups and a majority of North
Americans.
In summary, AR and clustering provided similar information towards the context
of our sample, which means that previous contact with gamification, knowledge of the
concept and habit of playing games do influence the intention to use in educational
environments. This information can be used by teachers, instructors, and other educators
to know when to gamify. Our demographic analysis did not present significant differences
– except for Cluster 5, in which most of the sample is composed by female respondents.
These results might have influenced users’ response towards previous used applications,
where most of the respondents (N > 300) used Duolingo as an example of a gamified
application, which is an educational environment.
IX Congresso Brasileiro de Informática na Educação (CBIE 2020)
Anais do XXXI Simpósio Brasileiro de Informática na Educação (SBIE 2020)
477
4.3. Discussion
This work provides insights on how the users’ context and demographics influence their
intention of use in gamification in education. Through this study, we add other variables
(previous knowledge on gamification, previous use of gamified applications, and playing
habits) that might be important to be considered when the designer and/or educator think
about gamifying their learning environments, which is not often considered when
designing gamification, but do impact on the users’ experience [Hamari 2015; Rodrigues
et al. 2016].
In future studies, researchers might ask the students about their intentions,
knowledge and/or playing habits, to understand if that really influences and has a positive
or negative impact on gamification. Another future research proposal would be
identifying how culture (in this case, the country where the person resides in) is related
to these factors as well, since culture is not a variable that is considered too often in the
gamification empirical literature [Klock et al. 2020].
4.4. Limitations
During the design and implementation of this work we faced some limitations. Some of
these limitations are concerned with the way we collected the users’ intention of use,
which could have been done through validated instruments, such as the Technology
Acceptance Model [Davis 1989]. However, due its complexity and aiming at a broader
public, we opted to use a single question self-assessing the intention of use through a
Likert Scale, which is used to measure abstract ideas. Another limitation is the
geographical distribution of our work, which might have been influenced by using
Amazon Mechanical Turk; we could not control this variable without increasing the
overall cost of this research. This could be enhanced or explored in future works.
5. Conclusion and Future Works
In this work, we focused on exploring and analysing the influence of contextual variables
over intention to use gamification in educational environments. Through the data
collected in our survey, we provided the following empirical contributions: (I) evidence
that context (previous knowledge, habit of playing games, and contact with gamification)
influence the intention to use; (II) and evidence that specific demographic characteristics
do not play a major role in the intention to use.
We believe this analysis could be further explored in future works by increasing
the number of respondents from different countries/continents, as well as different
genders, to increase diversity. Finally, another work would be exploring these contextual
variables within the design of gamification, as something to aid in the decision-making
process by designers and other people who want to gamify a learning environment.
Acknowledgements
The authors would like to thank FAPESP (Projects 2016/02765- 2; 2018/11180-3;
2018/15917-0; 2018/07688-1), Capes and CNPq for the funding provided. This research,
was carried out within the scope of the Samsung-UFAM Project for Education and
Research (SUPER), according to Article 48 of Decree nº 6.008/2006 (SUFRAMA), was
partially funded by Samsung Electronics of Amazonia Ltda., under the terms of Federal
Law nº 8.387/1991, through agreements 001/2020 and 003/2019, signed with the Federal
IX Congresso Brasileiro de Informática na Educação (CBIE 2020)
Anais do XXXI Simpósio Brasileiro de Informática na Educação (SBIE 2020)
478
University of Amazonas and FAEPI, Brazil. The authors also would like to thank
FAPESC (public call FAPESC/CNPq No. 06/2016 support the infrastructure of CTI for
young researchers, project T.O. No.: 2017TR1755 - Ambientes Inteligentes Educacionais
com Integração de Técnicas de Learning Analytics e de Gamificação).
References
Agrawal, R., Imieliński, T. and Swami, A. (1993). Mining association rules between sets
of items in large databases. http://portal.acm.org/citation.cfm?doid=170035.170072,
[accessed on Oct 22].
An, Y., Zhu, M., Bonk, C. J. and Lin, L. (6 jun 2020). Exploring instructors’ perspectives,
practices, and perceived support needs and barriers related to the gamification of MOOCs.
Journal of Computing in Higher Education, p. 1–21.
Bentley, F., Neill, K. O., Quehl, K. and Lottridge, D. (21 apr 2020). Exploring the Quality
, Efficiency , and Representative Nature of Responses Across Multiple Survey Panels. In
Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems. .
ACM. https://dl.acm.org/doi/10.1145/3313831.3376671, [accessed on Jun 3].
Cohen, J. (1 apr 1960). A coefficient of agreement for nominal scales. Educational and
Psychological Measurement, v. 20, n. 1, p. 37–46.
Davis, F. D. (sep 1989). Perceived Usefulness, Perceived Ease of Use, and User
Acceptance of Information Technology. MIS Quarterly, v. 13, n. 3, p. 319.
Deterding, S., Sicart, M., Nacke, L., O’Hara, K. and Dixon, D. (2011). From Game
Design Elements to Gamefulness: Defining “Gamification.” Proceedings of the 2011
annual conference extended abstracts on Human factors in computing systems - CHI EA
’11, p. 2425.
Dichev, C. and Dicheva, D. (20 dec 2017). Gamifying education: what is known, what is
believed and what remains uncertain: a critical review. International Journal of
Educational Technology in Higher Education. Nature Publishing Group.
http://educationaltechnologyjournal.springeropen.com/articles/10.1186/s41239-017-
0042-5, [accessed on Apr 17].
Hahsler, M., Grün, B. and Hornik, K. (2007). Introduction to arules--mining association
rules and frequent item sets. SIGKDD Explor, v. 2, n. 4, p. 1–28.
Hamari, J. (2015). Why Do People Buy Virtual Goods ? Attitude towards Virtual Good
Purchases versus Game Enjoyment.
Hamari, J. and Koivisto, J. (1 aug 2015). Why do people use gamification services?
International Journal of Information Management, v. 35, n. 4, p. 419–431.
Klock, A. C. T., Gasparini, I., Pimenta, M. S. and Hamari, J. (13 jun 2020). Tailored
gamification: A review of literature. International Journal of Human-Computer Studies,
p. 102495.
Klock, A. C. T., Ogawa, A. N., Gasparini, I. and Pimenta, M. S. (2018). Does
gamification matter? A systematic mapping about the evaluation of gamification in
educational environments. In Proceedings of the 33rd Annual ACM Symposium on
Applied Computing. . ACM Press.
http://dl.acm.org/citation.cfm?doid=3167132.3167347, [accessed on Jan 8].
IX Congresso Brasileiro de Informática na Educação (CBIE 2020)
Anais do XXXI Simpósio Brasileiro de Informática na Educação (SBIE 2020)
479
Lazar, J., Feng, J. H. and Hochheiser, H. (2017). Research methods in human-computer
interaction. 2nd. ed. Morgan Kaufmann.
Likert, R. A. (1932). A technique for the measurement of attitudes. Archives of
Psychology, v. 140, n. 140, p. 44–53.
MacQueen, J. and Others (1967). Some methods for classification and analysis of
multivariate observations. In Proceedings of the fifth Berkeley symposium on
mathematical statistics and probability.
Nelson, M. J. M. (2012). Soviet and American precursors to the gamification of work.
Proceeding of the 16th international academic MindTrek conference (pp. 23-26), p. 23–
26.
Palomino, P. T., Toda, A., Oliveira, W., et al. (2019). Exploring Content Game Elements
to Support Gamification Design in Educational Systems : Narrative and Storytelling. In
Proceedings of the SBIE 2019.
Pereira, F. D., Oliveira, E. H. T., Oliveira, D. B. F., et al. (2020a). Using learning analytics
in the Amazonas: understanding students’ behaviour in introductory programming.
British Journal of Educational Technology, p. bjet.12953.
Pereira, F. D., Toda, A., Oliveira, E. H., Cristea, A. I., Isotani, S., Laranjeira, D., Almeida,
A., Mendonça, J. (2020b). Can We Use Gamification to Predict Students’ Performance?
A Case Study Supported by an Online Judge. In International Conference on Intelligent
Tutoring Systems (pp. 259-269). Springer, Cham.
Rodrigues, L. F., Oliveira, A. and Costa, C. J. (1 oct 2016). Playing seriously - How
gamification and social cues influence bank customers to use gamified e-business
applications. Computers in Human Behavior, v. 63, p. 392–407.
Satopaa, V., Albrecht, J., Irwin, D. and Raghavan, B. (2011). Finding a" kneedle" in a
haystack: Detecting knee points in system behavior. In 2011 31st international
conference on distributed computing systems workshops.
Savard, I. and Mizoguchi, R. (18 dec 2019). Context or culture: what is the difference?
Research and Practice in Technology Enhanced Learning. SpringerOpen.
https://telrp.springeropen.com/articles/10.1186/s41039-019-0112-5, [accessed on May
25].
Seaborn, K. and Fels, D. I. (2014). Gamification in Theory and Action: A Survey.
Internatoinal Journal of Human-Computer Studies, v. 74, p. 14–31.
Toda, A. M., Valle, P. H. D. D. and Isotani, S. (20 mar 2018). The Dark Side of
Gamification: An Overview of Negative Effects of Gamification in Education. In
Communications in Computer and Information Science. . Springer, Cham.
http://link.springer.com/10.1007/978-3-319-97934-2_9, [accessed on Oct 1].
Vargas-Enriquez, J., Garcia-Mundo, L., Genero, M. and Piattini, M. (sep 2015). A
Systematic Mapping Study on Gamified Software Quality. 2015 7th International
Conference on Games and Virtual Worlds for Serious Applications (VS-Games), p. 1–8.
IX Congresso Brasileiro de Informática na Educação (CBIE 2020)
Anais do XXXI Simpósio Brasileiro de Informática na Educação (SBIE 2020)
480
