Content uploaded by Wilk Oliveira
Author content
All content in this area was uploaded by Wilk Oliveira on Jul 15, 2021
Content may be subject to copyright.
Modeling students’ flow experience through data
logs in gamified educational systems
Wilk Oliveira, Seiji Isotani
University of S˜
ao Paulo
S˜
ao Carlos, Brazil
{wilk.oliveira, sisotani}@usp.br
Olena Pastushenko, Tom´
aˇ
s Hruˇ
ska
Brno University of Technology
Brno, Czech Republic
{ipastushenko, hruska}@fit.vut.cz
Juho Hamari
Tampere University
Tampere, Finland
juho.hamari@tuni.fi
Abstract—User modeling in gamified educational systems is a
contemporary challenge. In particular, modeling the students’
flow experience (i.e., challenge-skill balance, action-awareness
merging, clear goals, unambiguous feedback, total concentration
on the task at hand, sense of control, loss of self-consciousness,
transformation of time, and autotelic experience) during a gam-
ified system usage is highly challenging. It is because mea-
surement’ instruments usually are invasive, removing the users
from the flow experience and/or cannot be applied massively
(e.g., participant observation, questionnaires or electroencephalo-
gram). We faced this challenge by conducting a data-driven
study (N = 23), where we used a robust statistical method
(i.e., partial least squares path modeling) to model the students’
flow experience, based on their interaction data (e.g., number
of mouse clicks) in a gamified educational system. The main
results indicate a relationship between the interaction logs and
four flow experience dimensions. Our finds contribute to the
area of gamified educational systems, through the students’
flow experience modeling. Finally, based on our results, we also
provided a series of recommendations for future studies.
Index Terms—Gamified educational systems, Flow Theory,
Flow experience, Students’ experience, User modeling
I. INTRODUCTION
In recent years, the number of different types of educational
systems (e.g., Massive Open Online Courses (MOOCs) [1],
intelligent tutoring systems (ITS) [2], gamified educational
systems [3], and others) has grown, attracting the attention
of teachers/instructors and students around the world. That
is causing the increase of the number of students using this
type of systems [4]. Therefore it is necessary to invest in
new methods to improve the student experience [5]. One
of the commonly used methods to improve the quality of
such educational systems is gamification (“The use of game
elements in non-game contexts” [6]) [7].
On the one hand, the design of online gamified educational
systems can help improve the teaching and learning process
[8]. On the other hand, the large number of students using
the same system at the same time tends to hinder some tasks,
such as understanding students’ behavior and assessing their
experience (e.g., engagement, motivation, and flow) during
the system usage [9]. This is highlighting the importance of
The authors would like to thank the grant provided by S˜
ao Paulo Research
Foundation (FAPESP), Project: 2018/07688-1.
modeling students’ experience in gamified educational systems
through their data logs [10].
Faced with this challenge, recent research has attempted
to take advantage of the big amounts of data logs generated
during users interaction with educational systems to model
students’ experience [11], [12]. One of the most complex pa-
rameters to be analyzed in such data sets is the flow experience
[13]. Flow is a deep engagement experience, composed of
nine associated dimensions [13]: i) challenge-skill balance, ii)
action-awareness merging, iii) clear goals, iv) unambiguous
feedback, v) total concentration on the task, vi) sense of
control, vii) loss of self-consciousness, viii) transformation of
time and ix) autotelic experience. When a student is in a state
of flow, they also tend to have a high learning experience (i.e.,
flow state positively affects the learning process) [5].
This paper addresses the described challenge by presenting
the results of a data-driven study with a sample composed of
23 university students. For this research we used structural
equation modeling (i.e., partial least squares path modeling
[14]) to associate the students’ flow experience in a gamified
system with their interaction data logs. Afterwards, we answer
the following research question: Does possible the students’
flow experience in a gamified educational system be mod-
eled based on their interaction data logs?
The study’s main results indicate a correlational model
between the group of data logs and four different flow ex-
perience dimensions (i.e., unambiguous feedback, clear goals,
loss of self-consciousness, and action-awareness merging).
Thus, our study is a step towards modeling the students’ flow
experience in gamified educational systems based on the data
logs. It also contributes to the development of computational
approaches for providing automatic students’ flow experience
identification in this kind of system. Based on the achieved
results, we proposed a series of recommendations for new
studies in the field.
II. RELATED WORKS
To identify the main related works, we analyzed the sys-
tematic literature review about Flow Theory and educational
technologies conducted by Oliveira et al. [15], the literature
review about Flow Theory and game based systems conducted
by Perttula et al. [16], and the literature review about Flow
97
2021 International Conference on Advanced Learning Technologies (ICALT)
2161-377X/21/$31.00 ©2021 IEEE
DOI 10.1109/ICALT52272.2021.00037
Theory and gamification conducted by Oliveira et al. [17]. Ini-
tially, studies that aim to model the students’ flow experience
in educational systems using data logs are relatively recent
[15]–[17]. The first studies were published in 2014 and used
statistical analysis [18]. After that, new studies used ontology
[19] and electroencephalogram [20] aiming to relate students’
flow experience and their data logs. However, in these studies
the flow experience was implemented only as challenge-skill
balance.
Most recent papers are proposing and evaluating theoretical
models that relate the students’ flow experience in educational
systems with different data logs [10], as well as conducting
qualitative studies with the same objective [21]. Finally, there
are studies which conduct similar experiments using machine
learning methods to model the students’ flow experience [22].
Analysis of the existing research indicates that one of
the remaining challenges is the need to conduct different
experiments in various contexts (e.g., gamified educational
systems), as well as data analysis using different techniques.
As far as we know, our study is the first to model the
students’ flow experience in gamified educational systems
through structural equation modeling considering the nine
original flow experience dimensions.
III. STUDY DESIGN
The main goal of this study is to model students’ flow
experience in a gamified educational system through data logs
(i.e., the student interaction data logs during the system usage).
To achieve this goal, we choose a data-drive method (i.e.,a
research based on users’ data analysis) [23].
A. Research question and hypothesis
Based on the study goal, the following research question
(RQ) was defined:
•RQ: Does possible the students’ flow experience in a
gamified educational system be modeled based on their
interaction data logs?
Since a large number of students started to use online
educational platforms in recent years, as well as due to
the large amount of data generated by these interactions,
numerous studies have sought to use this data to analyze
students’ experience [24]. In particular, concerning the flow,
some recent studies have shown that there may be a direct
relationship between different dimensions of the students’ flow
experience and the data logs [10], [21], [22]. Therefore, in this
study, we hypothesized that is possible to model students’
flow experience through data logs in gamified educational
systems.
B. Materials and procedure
To conduct this study, we used the gamified educational
system “bombsQuery”1[25], which is a tool for teaching the
basics of JavaScript/jQuery. It is a gamified system with 11
different missions, each one devoted to a different topic. Each
1https://lirael.github.io/bombsQuery/task.html
mission has some theory and examples and a free text area
where students need to insert their proposed solutions. The
playful goal of the missions is to clear the minefield from
all bombs. If the student’s solution was wrong, they have an
unlimited amount of attempts to correct it. However, if their
answer was correct, the next level starts. The students can
always come back to any of the already solved levels. This
might be useful if they want to check the accepted answer
for inspiration or go through the theory once again [26]. The
tool was chosen because it allowed the implementation of a
module to collect the students’ data logs. Moreover, it has
already been validated and used by other researchers [26].
Figure 1 illustrates an example for the mission.
Fig. 1. An example of the mission in the gamified educational system
bombsQuery
To collect the students’ data logs, we implemented a new
module in the tool (described below). Data logs were collected
based on the theoretical model proposed by Oliveira et al. [10].
The theoretical model proposed by Oliveira et al. [10] presents
a series of data logs theoretically associated with the nine flow
experience dimensions. The module proposes nine different
data logs that can be related to the nine flow experience
dimensions. The collected data logs are: ArtAF: average
students’ response time after a feedback; NumCOB: number
of mouse clicks; ProWS: proportion of wrong steps/responses;
RF: received feedback; TotUSV: total unique session views;
UsdTFS: used time to finish a step/mission; and ActTS: active
time in the system.
To analyze the students’ flow experience during working on
the assignment, we used the short flow state scale (short FSS)
proposed by Jackson and Mars [27]. This scale was chosen
because it was previously validated by Hamari and Koivisto
[28] to be used gamified settings, as well as being the most
popular scale in studies in the area of educational technologies
[15]. As the data collection was done after performing a quick
activity (with less than an hour), following the recommenda-
tion of the “The Manual for the Flow Scales” [29], we chose to
use the short scale composed of nine questions (one for each
dimension of the flow experience) presented in a five-points
Likert scale. To ensure the quality of the responses, inspired
by recent studies [30], we have included an “attention-check”
question (i.e., if you are filling out the form carefully, answer
3*) to eliminate responses from students who were not paying
98
due attention when answering the questions. This study was
organized in four different general steps: i) selection of the
gamified system, ii) students’ invitation, iii) data collection,
and iv) data analysis.
C. Participants
Our participants were 31 bachelor students of Brno Uni-
versity of Technology (Czech Republic), who volunteered to
participate in the experiment. Five responses were excluded
because students spent less than 5 minutes on the assignment,
which is an indication that they haven’t really used the system.
Three responses were excluded because the students answered
incorrectly the attention-check question. We, therefore, in-
cluded 23 participants (mean age = 21.54 years old, SD = 1.33;
6 women, 13 men, 0 non-binary, 4 preferred not to disclose
gender). To participate in the study, students received a link
to the questionnaires and the assignment, and they could work
on it online, at their pace and preferred time.
IV. RESULTS
In order to define the best strategy to analyse our data,
we analysed the data normality. As recommended by Wohlin
[31], we used the Shapiro-Wilk test to check it. The tests
showed that data is within a non-normal distribution. Thus, we
measured the internal reliability for the scale (for the overall
flow experience), using Cronbach’s alpha, thus obtain α=
0.621 (despite the low alpha value, this can occur because
the measured experiences on the scale behave independently).
Next, we measured the discriminant validity for the analyses.
The results are shown in Table I.
To answer the RQ, we modeled the students’ interaction
logs (as latent variables) and their flow experience in the
gamified educational system. Partial Least Squares Path Mod-
eling (PLS-PM) analysis [14] was used to observe the relation
between the students’ data logs and their flow experience.
PLS-PM was used because it is a reliable method for estimate
cause-effect relationship models with latent variable [14]. At
the same time, this method can be selected to the exploratory
research and to studies with small sample sizes (as such
our case) [32]. We used the software SmartPLS2to run the
analyses. Table II present the PLS-PM matrix and Figure 2
present the research path model with the results.
A. Discussion
In this study, we used PLS-PM to model the user flow
experience in the gamified educational system using gathered
data logs. The created model presented a significant relation-
ship between the data logs and four different flow experience
dimensions (see Table II): unambiguous feedback (β-0.435),
clear goals (β-0.530), loss of self-consciousness (β-0.707),
and action-awareness merging (β-0.283).
Initially, the data logs have a negative relationship (β-0.435)
with the unambiguous feedback. In this dimension, according
to Oliveira et al. [10], when a student takes a long time to
complete an activity after receiving feedback, they possibly
2https://www.smartpls.com/
Data logs
ArtNF
NumCOB
ProWS
RF
TotUSV
UsdTFS
ActTS
CSB
R2 = 0.121
MMA
R2 = 0.080
G
R2 = 0.281
F
R2 = 0.189
C
R2 = 0.005
CTRL
R2 = 0.001
LSC
R2 = 0.500
T
R2 = 0.000
A
R2 = 0.017
Fig. 2. Research path model
received ambiguous feedback. Thus, the less time the student
spends in the system, or the lesser the number of received
feedback, the unambiguous feedback experience tends to be
less. On the other hand, this result has not yet been identified
in previous studies and needs to be confirmed [18], [21], [22].
We also identified a negative relationship between the data
logs model and the the clear goals dimension (β-0.530).
This relationship can be explained based on the previous
relationship, because, once a student has not received unam-
biguous feedback, they will also possibly not be able to clearly
understand the objectives of the activity. This result can also
help to explain the results identified in the qualitative study
conducted by Oliveira et al. [21]. Their study identified that
the average of correct steps affected the sense of “clear goals”.
The highest relationship (also negative) occurred between
the data logs and loss of self-consciousness dimension (β-
0.707). According to Jackson et al. [29], when an individual
is no longer concerned with what others think of them, self-
consciousness has been lost. This statement may be key to
interpreting the results as if there is a trend in data logs (e.g.,
the proportion of wrong steps/responses, average students’
response time after feedback, and active time in the system),
is low, students can be concerned with what others think of
them, and then, to have a low loss of self-consciousness.
The last relationship in the model in represented by the
correlation between data logs and action-awareness merging
dimension (β-0.283), an experience comes about through a
total absorption in what one is doing [29]. This relationship
can also be explained according to the same explanation as
the previous experience. Our results demonstrate a tendency
regarding a relationship/pattern between students’ flow expe-
rience and their data logs in a gamified educational system.
Therefore, following the trend of previous studies. However,
the relationships identified in our study are different from the
99
TABLE I
DISCRIMINANT VALIDITY (COMPLETE BOOTSTRAPPING,SAMPLE=5000)
A C CSB CTRL Datalogs F G LSC MMA T
A1.000
C-0.112 1.000
CSB -0.135 0.030 1.000
CTRL 0.399 0.171 0.051 1.000
Datalogs 0.128 0.068 -0.348 0.035 1.000
F0.017 0.369 0.361 0.228 -0.435 1.000
G0.157 0.145 0.239 0.090 -0.530 0.536 1.000
LSC -0.014 0.064 0.181 0.072 -0.707 0.311 0.303 1.000
MMA 0.193 -0.130 0.586 0.294 -0.283 0.150 0.182 0.039 1.000
T0.644 0.209 0.002 0.138 -0.015 -0.043 0.113 0.246 -0.027 1.000
Key: CSB: challenge-skill balance, MMA: action-awareness merging, G: clear goals, F: unambiguous feedback, C:
total concentration on the task at hand, CTRL: sense of control, LSC: loss of self-consciousness, T: transformation
of time, and A: autotelic experience
TABLE II
PLS-PM MATRIX FOR DATA LOGS AND FLOW EXPERIENCE DIMENSIONS
CI
βM SD P 2.5% 97.5%
DL →A0.128 0.058 0.248 0.605 -0.604 0.397
DL →C0.068 0.141 0.220 0.756 -0.233 0.534
DL →CSB -0.348 -0.384 0.190 0.067 -0.755 -0.034
DL →CTRL 0.035 0.010 0.201 0.862 -0.496 0.372
DL →F-0.435** -0.436 0.153 0.005 -0.707 -0.097
DL →G-0.530** -0.526 0.145 0.000 -0.756 -0.227
DL →LSC -0.707* -0.584 0.279 0.011 -0.908 0.034
DL →MMA -0.283* -0.341 0.134 0.036 -0.602 -0.101
DL →T-0.015 -0.026 0.135 0.913 -0.284 0.232
Key: DL: data logs, CSB: challenge-skill balance, MMA: action-awareness
merging, G: clear goals, F: unambiguous feedback, C: total concentra-
tion on the task at hand, CTRL: sense of control, LSC: loss of self-
consciousness, T: transformation of time, and A: autotelic experience, β:
regression coefficient, M: meam, SD: standard deviation, P: p-value,CI:
confidence interval, * p<0.5, ** p<0.005
relationships observed in some other studies.
This result may indicate that the relationship between data
logs and flow experience can be different according to the type
of system, participants’ age, and other factors (since each pre-
vious study was conducted in different settings). Another way
of thinking is related to the number of participants (especially
in our study). The sample size needs to be increased to also
increase the confidence in the results. Regardless, our results
are promising for modeling the students’ flow experience
in gamified educational systems based on data logs, while
highlighting the need for further research.
B. Limitations
The study presented in this paper has some limitations,
which we seek to mitigate. The experience measured in the
study (i.e., flow experience) is a complex parameter to be
measured. To mitigate this limitation, we use only previously
validated methods (i.e., the short FSS validated by Hamari
and Koivisto [28] for the gamification domain and theoret-
ical model proposed by Oliveira [10] to collect data logs
in educational systems). At the same time, to ensure the
quality of responses and to avoid external threats (e.g., lack
of attention from students), we insert an “attention checking
question” within the scale and used other methods (e.g.,
remove responses from students who used the system for less
than five minutes) to avoid data set inconsistencies. Another
important limitation is related to our small sample size (i.e.,
23 students). To mitigate this limitation, we use a robust
statistical method capable of accurately analyzing data from
small samples (i.e., PLS-PM) [14]. However, we highlight the
importance of replicating the experiment with larger samples
to provide a greater results generalization, and we are sure that
this paper would serve as an excellent basis for such future
research.
C. Ways forward
Based on our results, it is possible to provide some rec-
ommendations for future studies. Comparing our results with
results of other studies [18], [21], [22], we can conclude that
the relationship of each flow experience dimension with the
different data logs can be changed according to the context
(i.e., with the type of educational system or even with the type
of data analysis). Therefore, we recommend that future stud-
ies conduct similar studies in different types of educational
environments (e.g., MOOCs, ITS, educational games, and
others). In the same way, we also recommend analyzing the
data through different techniques (e.g., qualitative analysis,
data mining and machine learning techniques), seeking to
carry out the experiments with larger samples, to increase the
generalization of the results.
In our study, we considered the entry model as a set of
different variables (i.e., data logs). However, we do not model
each data log individually with each flow experience dimen-
sion. Thus, we recommend that future studies can model the
relationship of each data log (individually) with the students’
flow experience dimensions. In our study, our input data
(data logs) were defined based on the study by Oliveira et
al. [10] However, not all data from the theoretical model
was considered in our study. Thus, we recommend that
future studies also include the other data logs proposed
in the theoretical model. Last but not least, we recommend
that further research improve the inputs by analysing
at whether gamification data (e.g., number of points,
badges, and leader-boards (alone)) affects students’ flow
experience.
100
V. C ONCLUDING REMARKS
Modeling students’ flow experience in gamified educational
systems is a contemporary challenge. In this study, we used
PLS-PM to model the flow experience of the students through
their data logs in a gamified educational system. Our results
demonstrate a correlation between the interaction logs and
four flow experience dimensions, moving towards modeling
the students’ flow experience in educational games using data
logs. Future research is planned to replicate this experiment
with larger numbers of participants, using new data analysis
methods, such as data mining and machine learning.
NOTES
Previous studies of this project have been published:
Oliveira et al. [15] conducted a systematic literature review
about Flow Theory and Educational Technologies; Oliveira
[33] presented the project overview; Oliveira et al. [10]
proposed a theoretical model relating students’ data logs and
their flow experience in educational systems; and Oliveira et
al. [21] conducted a qualitative study analysing students’ data
logs and their flow experience in an educational systems.
REFERENCES
[1] A. Bozkurt, E. Akg¨
un- ¨
Ozbek, and O. Zawacki-Richter, “Trends and
patterns in massive open online courses: Review and content analysis of
research on moocs (2008-2015),” International Review of Research in
Open and Distributed Learning: IRRODL, vol. 18, no. 5, pp. 118–147,
2017.
[2] E. Mousavinasab, N. Zarifsanaiey, S. R. Niakan Kalhori, M. Rakhshan,
L. Keikha, and M. Ghazi Saeedi, “Intelligent tutoring systems: a sys-
tematic review of characteristics, applications, and evaluation methods,”
Interactive Learning Environments, pp. 1–22, 2018.
[3] J. Koivisto and J. Hamari, “The rise of motivational information systems:
A review of gamification research,” International Journal of Information
Management, vol. 45, pp. 191–210, 2019.
[4] C. Jack and S. Higgins, “Embedding educational technologies in early
years education,” Research in learning technology, vol. 27, 2019.
[5] I. Buil, S. Catal´
an, and E. Mart´
ınez, “The influence of flow on learning
outcomes: An empirical study on the use of clickers,” British Journal
of Educational Technology, vol. 50, no. 1, pp. 428–439, 2019.
[6] S. Deterding, D. Dixon, R. Khaled, and L. Nacke, “From game design
elements to gamefulness: defining gamification,” in Proceedings of the
15th international academic MindTrek conference: Envisioning future
media environments. ACM, 2011, pp. 9–15.
[7] C. Hursen and C. Bas, “Use of gamification applications in science
education.” International Journal of emerging technologies in Learning,
vol. 14, no. 1, 2019.
[8] S. Bai, K. F. Hew, and B. Huang, “Is gamification “bullshit”? evidence
from a meta-analysis and synthesis of qualitative data in educational
contexts,” Educational Research Review, p. 100322, 2020.
[9] L. Shi, A. I. Cristea, A. M. Toda, and W. Oliveira, “Exploring navigation
styles in a futurelearn mooc,” in International Conference on Intelligent
Tutoring Systems. Springer, 2020, pp. 45–55.
[10] W. Oliveira, A. Toda, P. Palomino, L. Rodrigues, S. Isotani, and
L. Shi, “Towards automatic flow experience identification in educational
systems: A theory-driven approach,” in Extended Abstracts of the
Annual Symposium on Computer-Human Interaction in Play Companion
Extended Abstracts. ACM, 2019, pp. 581–588.
[11] L. Shi, A. I. Cristea, A. M. Toda, and W. Oliveira, “Social engagement
versus learning engagement an exploratory study of futurelearn learners,”
in 2019 IEEE 14th International Conference on Intelligent Systems and
Knowledge Engineering (ISKE). IEEE, 2019, pp. 476–483.
[12] ——, “Revealing the hidden patterns: A comparative study on profiling
subpopulations of mooc students,” International Conference on Infor-
mation Systems Development, 2020.
[13] M. Csikszentmihalyi, Finding flow: The psychology of engagement with
everyday life. Basic Books, 1997.
[14] J. F. Hair Jr, G. T. M. Hult, C. Ringle, and M. Sarstedt, A primer on
partial least squares structural equation modeling (PLS-SEM). Sage
publications, 2016.
[15] W. Oliveira, I. I. Bittencourt, S. Isotani, D. Dermeval, L. B. Marques, and
I. F. Silveira, “Flow theory to promote learning in educational systems: Is
it really relevant?” Brazilian Journal of Computers in Education, vol. 26,
no. 02, p. 29, 2018.
[16] A. Perttula, K. Kiili, A. Lindstedt, and P. Tuomi, “Flow experience
in game based learning–a systematic literature review,” International
Journal of Serious Games, vol. 4, no. 1, pp. 57–72, 2017.
[17] W. Oliveira, O. Pastushenko, L. Rodruigues, A. M. Toda, P. T. Palomino,
J. Hamari, and S. Isotani, “Does gamification affect flow experience? a
systematic literature review,” in GamiFIN Conference 2021: Proceedings
of the 5th International GamiFIN Conference, 2021, pp. 1–10.
[18] P.-M. Lee, S.-Y. Jheng, and T.-C. Hsiao, “Towards automatically detect-
ing whether student is in flow,” in International Conference on Intelligent
Tutoring Systems. Springer, 2014, pp. 11–18.
[19] G. C. Challco, F. R. Andrade, S. S. Borges, I. I. Bittencourt, and
S. Isotani, “Toward a unified modeling of learner’s growth process and
flow theory,” Journal of Educational Technology & Society, vol. 19,
no. 2, 2016.
[20] F. De Kock, The Neuropsychological Measure (EEG) of Flow Under
Conditions of Peak Performance. Unisa, 2014. [Online]. Available:
https://books.google.com.br/books?id=qKDFrQEACAAJ
[21] W. Oliveira, L. Rodrigues, A. Toda, P. Palomino, L. Shi, and S. Isotani,
“Towards automatic flow experience identification in educational sys-
tems: A qualitative study,” in Brazilian Symposium on Computers in
Education, vol. 31, 2020.
[22] Y. C. Semerci and D. Goularas, “Evaluation of students’ flow state in
an e-learning environment through activity and performance using deep
learning techniques,” Journal of Educational Computing Research,p.
0735633120979836, 2020.
[23] V. Dhar, “Data science and prediction,” Communications of the ACM,
vol. 56, no. 12, pp. 64–73, 2013.
[24] J. L´
opez-Zambrano, J. A. Lara, and C. Romero, “Towards portability of
models for predicting students’ final performance in university courses
starting from moodle logs,” Applied Sciences, vol. 10, no. 1, p. 354,
2020.
[25] O. Pastushenko, T. Hruˇ
ska, and J. Zendulka, “Increasing students’
motivation by using virtual learning environments based on gamification
mechanics: Implementation and evaluation of gamified assignments for
students,” in Proceedings of the Sixth International Conference on
Technological Ecosystems for Enhancing Multiculturality, 2018, pp.
755–760.
[26] O. Pastushenko, W. Oliveira, S. Isotani, and T. Hruˇ
ska, “A methodology
for multimodal learning analytics and flow experience identification
within gamified assignments,” in Extended Abstracts of the 2020 CHI
Conference on Human Factors in Computing Systems, 2020, pp. 1–9.
[27] S. A. Jackson and R. C. Eklund, “Assessing flow in physical activity:
The flow state scale–2 and dispositional flow scale–2,” Journal of Sport
and Exercise Psychology, vol. 24, no. 2, pp. 133–150, 2002.
[28] J. Hamari and J. Koivisto, “Measuring flow in gamification: Disposi-
tional flow scale-2,” Computers in Human Behavior, vol. 40, pp. 133–
143, 2014.
[29] S. Jackson, B. Eklund, and A. Martin, “The flow manual the manual for
the flow scales manual. sampler set,” Mind, pp. 1–85, 2011.
[30] R. Orji, G. F. Tondello, and L. E. Nacke, “Personalizing persuasive
strategies in gameful systems to gamification user types,” in Proceedings
of the 2018 CHI Conference on Human Factors in Computing Systems,
2018, pp. 1–14.
[31] C. Wohlin, P. Runeson, M. H¨
ost, M. C. Ohlsson, B. Regnell, and
A. Wessl´
en, Experimentation in software engineering. Springer Science
& Business Media, 2012.
[32] J. Henseler, C. M. Ringle, and R. R. Sinkovics, “The use of partial least
squares path modeling in international marketing,” in New challenges to
international marketing. Emerald Group Publishing Limited, 2009.
[33] W. Oliveira, “Towards automatic flow experience identification in edu-
cational systems: A human-computer interaction approach,” in Extended
Abstracts of the Annual Symposium on Computer-Human Interaction in
Play Companion Extended Abstracts, 2019, pp. 41–46.
101
