Content uploaded by M.José Nácher
Author content
All content in this area was uploaded by M.José Nácher on May 14, 2021
Content may be subject to copyright.
Studies in Educational Evaluation 70 (2021) 101026
Available online 5 May 2021
0191-491X/© 2021 The Author(s). Published by Elsevier Ltd. This is an open access article under the CC BY-NC-ND license
(http://creativecommons.org/licenses/by-nc-nd/4.0/).
The effectiveness of the GoKoan e-learning platform in improving
university students’ academic performance
María Jos´
e N´
acher
a
, Laura Badenes-Ribera
a
,
*, Clara Torrijos
b
, Miguel A. Ballesteros
c
,
Elena Cebadera
d
a
University of Valencia, Valencia, Spain
b
CEO of GoKoan, Valencia, Spain
c
CTO of GoKoan, Valencia, Spain
d
UX/UI of GoKoan Valencia, Spain
ARTICLE INFO
Keywords:
Online learning
Distributed learning environments
Human-computer interface
Media in education
ABSTRACT
The GoKoan e-learning platform supports face-to-face training in an educational community. Its aim is to
optimise the way and the time of study in order to improve academic performance. To evaluate the GoKoan
platform’s effectiveness as a tool for improving academic performance, an experimental study was carried out
using a sample of 171 university students enrolled in the psychology degree programme who were randomly
assigned to the two different conditions (the experimental group: traditional learning +e-learning with the
GoKoan platform; and the control group: traditional learning without e-learning). The ndings showed that using
GoKoan had a positive impact on the students’ academic performance (d =0.39, 95 % CI [0.08, 0.69]). The
results highlight the importance of blended learning in improving students’ learning performance. Other aspects
of its effectiveness (e.g, the levels achieved in student learning outcomes) will be considered in future studies.
1. Introduction
The present Covid-19 pandemic has forced universities to shift from
100 % face-to-face classes to new scenarios with online classes or
blended learning systems (Adedoyin & Soykan, 2020; Rapanta et al.,
2020). This has been a considerable challenge for the university teach-
ing system, which has had to implement blended learning (a.k.a.
b-learning, mixed learning or the hybrid model) to combine the physical
space of face-to-face learning with asynchronous or synchronous virtual
environments (Bonk & Graham, 2005; Garrison & Kanuka, 2004; Gra-
ham, 2019; Macdonald, 2008).
In this context of hybrid teaching models, the development of
effective e-learning tools becomes especially important in the virtual
environment. With this aim the present study analyses the impact on
university students’ academic performance of a new online learning
platform implemented in a blended learning system.
1.1. Blended learning
Numerous studies have shown that blended learning offers a series of
advantages over a fully face-to-face or fully online course (Graham,
2019; Harding et al., 2005; Liu et al., 2016; Macedo-Rouet et al., 2009;
Woods et al., 2004): a) offers students intellectually more interesting
and satisfactory learning (Woods et al., 2004); b) enables the concepts
learned in textbooks or classrooms to be reinforced (González-Gómez
et al., 2015; L´
opez-Ozieblo, 2018); c) sees students achieve better un-
derstanding, retain information for longer, and enjoy classes more as it
favours the construction of understanding that is more cohesive with the
interconnected facets of a discipline ( ´
Alvarez et al., 2013; Regueras
et al., 2009); d) improves student motivation and commitment to
learning (Ahmed & Osman, 2020; Gilboy et al., 2015; Street et al., 2015;
Xiuhan & Samuel Kai Wah, 2020); e) offers students a higher level of
independence in the learning process (Hung, 2015; Jebraeily et al.,
2020); and f) improves interaction between the teacher and student,
favouring knowledge exchange and learning given that teachers have
more access to students and have better supervision over their student’s
* Corresponding author at: Faculty of Psychology (University of Valencia), Avda. Blasco Ib´
a˜
nez, 21, Valencia, Spain.
E-mail addresses: mjnacher@uv.es (M.J. N´
acher), laura.badenes@uv.es (L. Badenes-Ribera), clara@gokoan.com (C. Torrijos), ma.ballesteros@gokoan.com
(M.A. Ballesteros), elena.cebadera@gokoan.com (E. Cebadera).
Contents lists available at ScienceDirect
Studies in Educational Evaluation
journal homepage: www.elsevier.com/locate/stueduc
https://doi.org/10.1016/j.stueduc.2021.101026
Received 26 September 2020; Received in revised form 27 April 2021; Accepted 30 April 2021
Studies in Educational Evaluation 70 (2021) 101026
2
progress; and students have better access to the teacher and can easily
ask their questions or provide the teacher with their suggestions
(Jebraeily et al., 2020; Makhdoom et al., 2013).
These benets certainly have a positive impact on academic per-
formance. Prior studies have shown that by applying blended learning in
different knowledge areas of university education (e.g., mathematics,
engineering, foreign languages, health sciences, etc.) academic
achievement is greater, in comparison with traditional learning (e.g.
Ahmed & Osman, 2020; ´
Alvarez et al., 2013; Corell et al., 2018; Hung,
2015; Kassem, 2016; Lança & Bjerre, 2018; Lim & Morris, 2009;
L´
opez-P´
erez et al., 2011; McLaughlin et al., 2014; Poon, 2013; Regueras
et al., 2009; Yigit et al., 2013; Zacharis, 2015). For instance, a recent
meta-analysis (Vall´
ee et al., 2020) evaluated the effectiveness of several
types of blended learning compared to traditional learning in health
education and found that all blended learning showed signicantly
better knowledge outcomes than traditional learning (d
+
=1.07, 95 %
CI [0.85, 1.28); similar results were found for online learning (d
+
=0.73,
95 % CI [0.60, 0.86] and for computer-assisted instruction (d
+
=1.13,
95 % CI [0.47–1.79] compared with traditional learning.
1.2. The GoKoan method: an e-learning platform
This study presents the development of a new e-learning tool that
supports higher education studies named the GoKoan platform. Gokoan
is a web-based software system, fully implemented and functional. The
innovation offered by the GoKoan method resides in the fact it is an
online study platform that: 1) includes all theoretical and practical
content of a subject, and 2) develops an algorithm based on articial
intelligence to help the student organise and optimise the manner of
study and study time, thus avoiding academic procrastination, which
has proven to have a negative impact on academic performance (e.g.,
Hooshyar et al., 2020; Kim & Seo, 2015).
With the aim of fully optimising the chances of success in studying,
and therefore academic performance, the platform is scientically based
on the principles and laws of memory and learning that have been
established from pioneering studies carried out in this eld by Ebbing-
haus (1885, 1913). Their results led to the denition of important as-
pects to be considered in the educational context: study material
staggered over time and the signicance of reviewing, both in terms of
acquiring, maintaining and consolidating the information.
The authors of the GoKoan method developed a system of software
that executes planning algorithms and considers: a) The Fragmentation of
the subject’s content and its study in time throughout the period of study,
given that learning is more effective if study is spaced out with a smaller
amount of information than if it is concentrated into fewer learning
sessions with a greater content load (Baddeley & Longman, 1978; Jost
et al., 2021); b) The introduction of reviews at critical moments of forget-
fulness and throughout the period of study for the material. As revealed by
the experimental studies carried out by Ebbinghaus (1885, 1913), a
signicant loss of information occurs in the rst hours after learning,
with a progressive decline over time if there are no material reviews.
Therefore, the platform introduces reviews immediately after a certain
content is learned (within 24 h) and throughout the period of study; c)
Grouping of material learned in progressively larger content blocks for re-
view and reviews spaced out over time. For example, in later phases of the
study when the material has been studied for the rst time, the contents
are grouped into larger information blocks, and the tests cover pro-
gressively more content; d) Signicant learning. The platform allows the
students to create their own materials (diagrams, mental maps, etc.),
while the tool also proposes memory strategies and techniques that
facilitate a more in-depth, elaborate processing of the material and,
therefore, its improved consolidation and recovery in the long term; e)
Multiple choice-style reviews with feedback provided to the student on the
mistakes made and the subject area related to the mistake; and f) Practice
exams with the same structure as the subject’s nal exam (number of
questions, alternative formats).For example, the format of the platform’s
tests are all exactly the same type (multiple choice tests), number of
alternatives and penalising system) that the students will face in their
nal exam.
Within the Technology Enhance Leaning (TEL) framework, the
GoKoan system focuses on areas of student’s knowledge modelling in the
study contents and planning study activities within the limits of the time
available and the time horizon (exam date). Its most innovative ele-
ments are as follows: a) fragmentation of contents and their evaluation
in the order of <1000 words, which allows learning difculties to be
detected more precisely and thus create specic reinforcement activ-
ities; b) the uniqueness of the content in the system, which allows the
student to change years (possibly for a more advanced one) without
having to repeat what he has already learned in previous years, and c)
adaptive planning under the time restrictions (availability and nal-
ization) according to the aimed-for mark.
1.2.1. GoKoan algorithm
GoKoan is based on a complex AI algorithm which creates a fully
personalised student study plan founded on the staggered study princi-
ple and the introduction of reviews, since it is possible to implement this
in e-learning. Consequently, when the students access the platform using
any electronic device with Internet access (computer, mobile or tablet),
they will nd material (the subject) that they must learn. To do so, they
specify their weekly availability for study, the deadline by which they
must have learned the subject and the desired level of retention. Then
the algorithm offers a study plan considering these variables in addition
to study phases, reviews and assessment.
The study plan is completely dynamic and can be adapted to the
evaluations that the student has to take either after learning a content or
after a periodical micro-evaluation. It also changes each time the student
modies the time restrictions such as the nal exam date or the weekly
available study time. In this way, the system sequences the tasks for all
the sections that can be included in the time available according to the
time slot indicated.
1.2.2. Description of the software
GoKoan employs Cloud software architecture with stateless servers
sheltered by a load balancer. GoKoan is a web application developed
using Angular JS technology. It accesses the servers via an HTTP
communication interface. The servers run on a Java virtual machine,
and Kotlin is the language used for the business logic. The database is
based on PostgreSQL technology, which offers transactionality and
standard consulting in data cross-referencing.
The GoKoan method’s main algorithms are found in the system’s
core: a) The scoring algorithm uses the student’s marks in each section
and constructs an aggregated view of their overall position, including
their total and partial progress in each section. For example, the topics
are composed according to the content hierarchy (section trees 1, 1.1,
1.2, etc.) and computes an aggregate/totalled score as a weighted
average of the scores of lower levels with an overall score for each
section or topic, and at a higher level, for the entire year (see Figs. 1 and
2); b) The content sequencing algorithm is based on previous scoring,
which determines the content that should be worked on next and which
tasks the user should do (study, take a test, etc.). At present there are
study tasks, tests and rests but it is open to other types of task such as
summarising texts, revision of questions and answers from other stu-
dents, etc. (see Fig. 3); c) The planning algorithm cross-references hourly/
daily availability and sequencing to generate a unique, personal calen-
dar that distributes all outstanding work as efciently as possible (see
Fig. 4); d) The test-building algorithm analyses the user’s weak points,
emphasising them to detect forgotten content and choosing questions
according to the user’s level. The test construction is based on the scope
of the test (a section, a specic topic or “everything I’ve learned up to
now” (continual evaluation) and considers the latest questions dealt
with so that they are not repeated as long as the set of available ques-
tions makes this possible (see Fig. 5); and e) The coaching algorithm,
M.J. N´
acher et al.
Studies in Educational Evaluation 70 (2021) 101026
3
based on decision trees, cross-referenced scoring, and planning and aims
to provide “human” feedback on how to optimise study time (see Fig. 6).
1.3. The purpose of the present Study
The aim of this paper is to offer data on the effectiveness of the
GoKoan platform with regard to improving academic performance,
providing evidence of its efcacy in university students using an
experimental design with two groups: an experimental group with
traditional learning FTF and e-learning with the GoKoan platform, and a
control group with traditional learning FTF without e-learning. In
agreement with previous studies that showed the positive effect of
blended learning methods on academic performance (e.g. Ahmed &
Osman, 2020; Corell et al., 2018; L´
opez-P´
erez et al., 2011; McLaughlin
et al., 2014; Vall´
ee et al., 2020), in the present study it is expected that:
Hypothesis 1. Students who attend traditional lecturer-directed FTF
classes and e-learning (e.g., the GoKoan platform) will get higher nal
exam scores (nal marks) than those who attend only traditional
lecturer-directed FTF classes.
Hypothesis 2. Students who obtain good results in the GoKoan tests
will get higher nal exam scores than those who obtain worse results in
the same tests.
2. Method
2.1. Participants
The sample size was planned by estimating that being enrolled in the
GoKoan platform would produce a medium effect size (d =0.5), a sta-
tistical power of 0.80, and an alpha value of 0.05 (two-tailed). The
required sample size was thus 128 participants according to the results
provided by the G*Power program (Faul et al., 2007).
Nonprobability or convenience sampling was used. The sample was
composed of 171 university students enrolled in the “Psychology of
Memory” subject of the Psychology degree program at the University of
Valencia (Spain), with a mean age of 21.47 years old (SD =4.59).
Fig. 1. Overall score and score according to units.
Fig. 2. Feedback on progress at the end of the session.
M.J. N´
acher et al.
Studies in Educational Evaluation 70 (2021) 101026
4
Consistent with the typical composition of psychology courses, 84.2 % of
the sample participants were women.
The participants were randomly divided into two groups: 84 were
assigned to the experimental group with traditional FTF and e-learning
(GoKoan) and 87 were assigned to the control group, with traditional
learning. There were 4 participants who dropped out (3 from the
experimental group and 1 from the control group). Table 1 shows the
descriptive demographic and academic data for the entire sample (N =
171) and those who dropped out during the course of the study (n =4).
No statistically signicant differences were found between the partici-
pants who completed the study and those who dropped out regarding
age (p =.554), gender (p =.609), academic achievement in the previous
year (p =.226), nal marks in the “Psychology of Memory” exam (p =
.982), the number of times that they sat the“Psychology of Memory”
exam (p =.517), retained students (p =.474), those in employment (p =
.615), or their ability to manage information and communication tech-
nologies (ICTs) (p =.265). No adjustments were therefore made to the
data.
2.2. Intervention
The face-to-face version of the course consisted of lessons in the
classroom with the existing curriculum: these were mostly teacher-
centered and used methods such as direct lectures, presentations
(PowerPoint slides), demonstrations and question-answer drills.
The Blended version of the course incorporated the GoKoan platform
for four months during the class period up until the exam date, which
allowed the experimental group to study the course in two phases:
1) Learning phase: the contents were presented in fragments to be
studied with assessments evaluating the level of acquisition until they
were learned. For example, one topic was divided into small study
sections. After studying each content fragment, the student took a test to
assess acquisition. If the result was 60 % or higher they could move on to
the next content fragment, otherwise the tool detected the parts of the
content where they were failing and then would re-plan the study pro-
gram and acquisition test until the student successfully passed the
fragment.
2) Review phase: re-tests of the whole course were carried out using
Fig. 3. Agenda for the day with detailed information on content to be studied and proposed activities.
Fig. 4. Personal plan editor.
M.J. N´
acher et al.
Studies in Educational Evaluation 70 (2021) 101026
5
multiple choice questions. These reviews were grouped into blocks with
a progressively higher content load and ended with a practice exam.
To optimise the learning process a series of factors were taken into
account: personalised study planning, time-based study, exhaustive
error analysis, continuous progress reports, collaborative learning, etc.
2.3. Measures
2.3.1. Demographic and academic variables
All the participants were asked to complete a demographic survey on
their age, sex, average academic achievement in the last academic year
(response scale from 0 to 10 points), the number of times they had sat
the “Psychology of Memory” exam; whether or not they were retained
students, whether or not they were in employment, their ability to
manage ICTs (on a scale of 0–10, where 0=completely incompetent and
10=completely competent), teacher’s name and class schedule (morn-
ing or afternoon).
2.3.2. Academic achievement in “Psychology of Memory”
Teachers were asked to report the student’s result in this subject’s
nal exam (June 2019), which was the same for all the participants and
consisted of multiple choice questionswith three alternatives. Students
could select the correct answer for each question by circling the asso-
ciated letter and lling in the right circle on the response sheet. Failures
(e.g., selecting an incorrect response) were penalized. The subject was
graded on a 0–10 scale. The higher the score, the higher the academic
performance.
2.3.3. Grades obtained by the tests on the GoKoan platform
The academic performance obtained on the Gokoan platform was
only assessed for students belonging to the experimental group (blended
learning version). The grade obtained was measured as the average
result of the multiple choice question tests completed by the partici-
pants. Each multiple choice question test had three alternatives of
response, where only one of them was the correct response. Students
should select the correct answer for each question. The grade was
assessed in a scale of measure: 0 to10. The higher the scores, the higher
Fig. 5. Feedback on assessment results.
Fig. 6. Banner with messages that improve motivation, and emotional graphics.
M.J. N´
acher et al.
Studies in Educational Evaluation 70 (2021) 101026
6
the performance”.
2.4. Design and procedure
The GoKoan effect on e-learning was assessed using an experimental
design with an unequal control group. The participants were randomly
divided into two conditions: the experimental group with traditional
FTF and e-learning (platform) and the control group with traditional
learning.Both groups covered the same contents and same course hours
during the same semester (from February to June 2019). The rando-
mised sampling process was carried out by a teacher who did not belong
to the "Psychology of Memory" subject and did not know the un-
dergraduates, so that the course teachers were blinded, i.e. they did not
know which students were in the experimental or control groups.
Participation in the study required the students’ informed consent.
The participants were given written consent request forms describing
the nature and objective of the study, in compliance with the ethical
code of the Declaration of Helsinki, and ethical approval to conduct the
study was obtained from the university. The forms stated that data
condentiality would be assured, participation was voluntary and the
participants could withdraw at any time.
2.5. Statistical analysis
Preliminary analyses were conducted to examine the normality of
distribution of the continuous variables. The participants’ academic and
sociodemographic characteristics were described by the means and
standard deviations of the continuous variables and frequencies and
percentages of the categorical variables. To compare signicant differ-
ences in academic and sociodemographic characteristics between the
entire sample and dropout participants, and between experimental and
control groups, Chi-squared test or Fisher’s exact test were performed on
the categorical variables, while the Student’s t-test or non-parametric
Mann-Whitney U test was used for continuous variables. Fisher’s exact
test was used when the cell count of the categorical variables was lower
than ve. The nonparametric Mann-Whitney U test was run when data
did not meet the parametric assumption of normality (e.g., age and
number of times students have sat the “Psychology of Memory” exam).
When the Student’s t-test was used the equality of variance was checked
by Levene’s test. As the two groups analysed had equal variance, no
corrections to the Student’s t-test were required.
A univariate analysis of variance (ANOVA) was then performed to
examine the GoKoan effect on the academic achievement indicator
measured as the nal marks in the "Psychology of Memory" exam. Sex
was added as a covariate to control any inuence it might have on ac-
ademic achievement scores, since the Chi-square test showed statisti-
cally signicant differences between the experimental group and the
control group in terms of sex. The equality of variance was checked
using Levene’s test. As the two groups analysed have an equal variance,
no corrections to the ANOVA test were required.
To evaluate the relationship between academic achievement and the
grades obtained in the tests on the GoKoan platform, Pearson’s corre-
lation coefcient test was performed.
Finally, to measure the magnitude of differences, Cohen’s d was used
as effect size statistics (Cohen, 1988) for the Student’s t-test and the
ANOVA test, the correlation coefcient r (r =z/root N) for the
Nonparametric Mann-Whitney U test (Clark-Carter 2009), and the Phi
Coefcient or Cramer’s V for the Chi-squared test and Fisher’s exact test.
All statistical analyses were carried out on SPSS Version 26 for Windows.
3. Results
3.1. Preliminary analyses
To ensure univariate normality, Kline (2011) suggested cut-offs of
absolute values of 3.0 and 10.0 for skewness and kurtosis, respectively.
Absolute values of skewness and kurtosis for scores on the outcome
measures (“Academic achievement in the last year”, “Academic
achievement in Psychology of Memory”, GoKoan test scores, and ICTs
management) were within the acceptable range for normal distribution.
However, distribution of the age variable and the number of times stu-
dents had sat the “Psychology of Memory” exam did not t with normal
distribution.
3.2. Pre-treatment comparisons
Table 2 shows the descriptive demographic and academic data for
the experimental and control groups, as well as comparisons between
both groups in relation to the participants’ characteristics. It can be seen
that there were no statistically signicant differences between the
experimental group and the control group in most of the demographic
and academic variables. Only one statistically signicant group differ-
ence was found in the sex variable, so that overall both groups were
homogeneous in terms of the academic and socio-demographic variables
assessed.
3.3. Effectiveness of the GoKoan Platform
The results of the ANOVA test controlling the sex of the participants
revealed a statistically signicant difference in the average nal exam
scores between the two groups (F(1, 167) =6.41, p =.012). The
experimental group students showed higher academic results (M =7.80,
SD =1.50) than control group students (M =7.24, SD =1.39), with a
small to moderate effect size (d =0.39, 95 % CI [0.08, 0.69]) according
to Cohen (1988), supporting Hypothesis H1.
The Pearson’s Correlation Coefcient test for the experimental group
students’ data revealed a positive and statistically signicant correlation
between the theGoKoan scores and the nal exam marks (r =.33, p =
.003, 95 %CI [.12, .51]), indicating that as the GoKoan test scores
improved, the students’ nal exam marks also improved. In other words,
those who had high GoKoan scores also had high scores in the nal
exam, and vice versa, supporting Hypothesis H2. Following Cohen’s
(1988) criteria, a correlation coefcient of r =.33 can be interpreted as
reecting a moderate but relevant relationship.
4. Discussion
This study analysed the effectiveness of the GoKoan platform, a new
online tool that supports university teaching while also complementing
and reinforcing face-to-face teaching in the classroom. Its aim is to
Table 1
Participants’ characteristics for the entire sample (N =171) and those who
dropped out of the study (n =4) and comparisons between both groups.
All sample Dropouts
Age, M (SD) 21.47
(4.59)
21.25
(2.06)
Sex, n (%)
Male 26 (15.8) 1 (25)
Female 141 (84.2) 3 (75)
Average academic achievement in the last academic
year, M (SD)
7.20 (0.97) 7.75 (0.5)
Number of times sat the exam, M (SD) 1.15 (0.55) 1 (0.0)
Ability to manage ICTs, M (SD) 7.92 (1.56) 7 (1.83)
Being a retained student, n (%)
Yes 19 (11.4) 0 (0.0)
No 148 (88.9) 4 (100)
Being in employment, n (%)
No 94 (57) 3 (75)
Occasionally 40 (24.2) 1 (25)
Yes 31(18.8) 0 (0.0)
Note. M =mean. SD =standard deviation. n =frequency, %=percentage. ES:
effect size. 95 % CI: condence interval for effect size statistic.
M.J. N´
acher et al.
Studies in Educational Evaluation 70 (2021) 101026
7
improve academic achievement among university students. For this
purpose, a study using an experimental design with an unequal control
group on the effect of blended learning (e.g., FTF +Gokoan platform) on
student performance measured by objective outcomes of nal course
grades was conducted. The ndings showed that being enrolled in the
GoKoan platform had positive effects on the students’ academic results.
Those who used the GoKoan platform obtained better nal exam marks
than those who attended only traditional lecturer-directed FTF classes.
The effect size was small to moderate (d =.039) according to Cohen’s
(1988) criteria. Similar ndings were obtained in the meta-analytic
study by Vo et al. (2017), who observed a small to moderate effect of
blended learing on student performance (g+ = 0.385, 95 % CI [0.239,
0.531]) over traditional teaching methods. In a meta-analysis, Ødegaard
et al. (2021) also found a small to moderate effect for ipped classrooms
on knowledge acquisition (d+ = 0.41; 95 % CI [0.20, 0.62] compared to
traditional classroom teaching.
The ndings of the present study showed that those who obtained the
highest scores in academic achievement indicators were also those with
the highest scores in the GoKoan achievement indicators, while those
with a lower index in the indicators also had lower scores on the GoKoan
platform. These results suggest that use of the GoKoan platform may
produce an improvement in students’ level of knowledge acquisition.
These ndings are in line with previous studies (e.g., Lança & Bjerre,
2018; Twigg, 2003) and demonstrate that the inclusion of ICTs in
traditional learning improves students’ academic achievement, moti-
vation and satisfaction with teachers (´
Alvarez et al., 2013; L´
opez-P´
erez
et al., 2011; Mohammad & Job, 2012).
In addition to the results of diverse studies, our experience as
teachers reveals the negative effects on academic performance of uni-
versity students’ tendency to delay studying for a subject until the days
close to the exam date (e.g., Hooshyar et al., 2020; Steel, 2007). In terms
of pedagogy, the pioneering studies by Ebbinghaus (1885, 1913)
demonstrated that study spaced out over time improves learning in
comparison to mass concentrated study periods. Indeed, distributed
practice (e.g., studying regularly) is one of the most powerful learner
behaviours that positively impact academic performance (Jost et al.,
2021). In this regard, and with the aim of improving the acquisition of
academic content and therefore student performance, the GoKoan
platform helps the student to organise and fully optimise studying. To do
so, the algorithm offers a study plan for the subject for the entire
teaching period adapted to the student’s availability.
On the other hand, the Learning Management Systems (LMS) used in
blending learning methodologies are very useful in assisting students
when it comes to managing their learning (planning the academic year,
access to educational material, multimedia resources, Moodle LMS, ac-
tivities, web applications, blogs, quizzes,etc.) and communicating with
teachers or fellow students (forums, chats, etc.) (Kraleva et al., 2019, for
a review). However, they do not offer adaptation mechanisms, nor do
they provide the students with feedback on their learning progress or on
the expected levels of acquisition regarding the subject’s contents. In
this regard, the GoKoan platform overcomes these limitations as it in-
forms the students of their progress and the contents to which they need
to dedicate more study time. Based on their mistakes in learning and
assessment tests, the algorithm allows them to know the contents that
must be improved and readjusts the study plan according to their
progress.
Not only does the platform offer the student feedback but it also
provides the teacher with feedback by implementing learning analytics
(LA) (Banihashem et al., 2018; Ferguson & Clow, 2017; Lodge & Corrin,
2017). LA are dened as the “measurement, collection, analysis and
presentation of data about students and their contexts for purposes of
understanding and optimising learning and the contexts in which it
occurs” (Long & Siemens, 2011). In this regard, all the information
collected by the platform through a process of learning analytics is
transformed into a metrics panel (business intelligence) that gives the
teacher access in real time to the students’ behaviour and progress
during the learning process as an individual and in a group. The infor-
mation collected is related to: a) study time (daily, weekly or monthly);
b) the results of the tests taken during the periodor at certain intervals; c)
the materials produced by the student (summaries, conceptual maps,
schemes, etc.); d) the number of study sessions; and e) active users each
day.
The statistical analysis of the student cohorts also allows adjustments
to estimates for subsequent years as it identies contents with a high
percentage of errors that need to be revised. The LA incorporated in the
platform also informs the teacher of how the students work asyn-
chronically, the study speed and frequency, the time periods, number of
times the platform has been accessed, the length of time used, errors, etc.
(Vela-P´
erez et al., 2017).
In this way, the teacher has at his disposal in the metrics panel in-
formation to understand and optimise the students’ learning process,
detect any problems and thus to work in the classroom on the contents
that present greater difculty or adapt them to ensure the students’
comprehension, performance and learning.
Table 2
Participants’ characteristics for the experimental (n =81) and control (n =86) groups and comparisons between both groups.
Experimental Control Statistic test p ES 95 %CI
Age, M (SD) 21.75 (4.98) 21.22 (4.30) z =-0.84 .399 d =0.11 −0.19, 0.42
Sex, n (%)
χ
2
=3.88 .049
Փ
=.15 .00, .30
Male 8 (9.9) 18 (20.9)
Female 73 (90.1) 68 (79.1)
Average academic achievement in the last academic year, M (SD) 7.25 (1.04) 7.13 (0.91) t =0.79 .431 d =0.12 −0.18, 0.43
Number of times sat the exam, M (SD) 1.10 (0.41) 1.20 (0.66) z =-0.95 .342 d =-0.18 −0.48. 0.12
Ability to manage ICTs, M (SD) 8.11 (1.48) 7.78 (1.63) t =-1.38 .170 d =0.21 −0.09, 0.52
Being a retained student, n (%)
χ
2
=0.35 .553
Փ
=.05 .00, .20
Yes 8 (9.9) 11 (12.8)
No 73 (90.1) 75 (87.2)
Being in employment, n (%)
χ
2
=1.32 .516 V =.09 .00, .21
No 45 (55.6) 49 (58.3)
Occasionally 18 (22.2) 22 (26.2)
Yes 18 (22.2) 13 (15.5)
Professor/instructor, n (%)
χ
2
=1.29 .526 V =.09 −.06, 0.24
Teacher/instructor 1 53 (65.4) 49 (57)
Teacher/instructor 2 18 (22.2) 23 (26.7)
Teacher/instructor 3 10 (12.3) 14 (16.3)
Class Schedule
χ
2
=0.16 .689
Փ
=.03 −.12, 0.18
Morning 60 (74.1) 66 (76.7)
Afternoon 20 (23.3) 20 (23.3)
Note. M =mean. SD =standard deviation. n =frequency, %=percentage. ES: effect size. 95 % CI: condence interval for effect size statistic.
M.J. N´
acher et al.
Studies in Educational Evaluation 70 (2021) 101026
8
The Gokoan platform also produces other benets for the teaching-
learning process. For instance, it promotes participation and improves
communication between students and teachers, encourages collabora-
tive learning through the community, sharing materials and resolving
doubts, favours constructive learning in which the teacher plays the role
of guide and the students have the important role and are more
responsible for their own learning process, and learning theoretical
content can be delegated to the platform so that class time can be used
for activities that facilitate more signicant, interactive learning based
on experiences, simulations and activities that encourage debate,
reection, critical thinking, the opportunity to practice and apply what
has been studied, and a focus on any parts of the subject matter that
cause difculties. In other words it provides what is known as the
“ipped classroom” (Awidi & Paynter, 2018; Dehghanzadeh & Jafar-
aghaee, 2018; Hinojo et al., 2018; Murillo et al., 2019).
4.1. Limitations and future research
The study has several limitations that must be pointed out. Firstly,
the class attendance of students participating in the study was not
considered. Several studies have shown its positive impact on nal
marks (e.g., Donnelly, 2010; L´
opez-P´
erez et al., 2011; Woltering et al.,
2009). Secondly, another factor that was not considered and that can
inuence the results is the time dedicated to studying and the number of
content reviews and may be signicant with regard to nal exam marks.
Thirdly, it would be worth considering the impact of the ability to
manage ICTs on academic performance, given that ICTs skills have been
noted as an important variable in blended learning courses (Almasi
et al., 2018). These aspects may have impacted on the results obtained
and so constitute highly interesting lines of future research that should
be investigated. For instance, future research might include these vari-
ables as covariates to control their effect on academic performance.
Future research might also include a post-experience questionnaire and
some monitoring during the experience to study how the students’
behaviour can be affected by interaction with the system.
Lastly, as the Gokoan system uses "study time plans" and multiple
choice "tests" in a way that make the effects of the two aspects of learning
indiscernible, future research should examine whether the system’s
learning results originate in the contextual use of planning and tests or
only in the availability of tests, because at present it is impossible to say
whether the results are on planning and tests or only on the latter option,
which was not considered in the study.
4.2. Conclusions
This paper has shown that the use of the e-learning GoKoan platform
as a support to traditional face-to-face classes has statistically signicant
effects on psychology degree students’ academic performance as
compared to that of a control group that followed the traditional classes.
These ndings agree with those of previous studies that combined
classroom teaching with blended teaching methods, showing that using
the GoKoan platform with traditional face-to-face classes raises stu-
dents’ marks (e.g. Ahmed & Osman, 2020; Graham, 2019; Liu et al.,
2016; Vall´
ee et al., 2020).
It should be pointed out that unlike many other platforms used in
blended learning (Kraleva et al., 2019), GoKoan incorporates a series of
characteristics that make it an innovative tool in the ITC context: 1) it
includes all the course study content in any electronic device with
Internet access; 2) to optimise the probability of successful studies and
therefore academic performance it is scientically founded on the laws
and principles of memory and the learning obtained from the tasks
carried out in this eld (e.g., Baddeley & Longman, 1978; Ebbinghaus,
1913; Jost et al., 2021). Important aspects implemented in the tool are
derived from its results and should be considered in the educational
context: the study material is spaced out over time, and importance is
given to revising when acquiring, maintaining and consolidating
information as well as at critical moments of forgetfulness; 3) the tool
also offers a learning experience that can be adapted to individual stu-
dents, with a study and revision plan that can be adjusted to their
progress, reports on the errors committed in the tests and the part of the
syllabus that deals with these errors, and 4) the students not only receive
information on their progress but also from the data analysis of the
learning analytics (LA) the teacher can: a) know each student’s progress
in real time and detect cases that are not advancing at an early stage; b)
detect the students’ frequent errors and adopt preventive strategies to
avoid academic failure, and c) improve the contents and devise new
strategies to optimise the learning process.
In conclusion, the results of the present study show that imple-
menting GoKoan in the context of higher education has important pos-
itive effects on students’ achievement. These ndings validate the
current GoKoan platform as a suitable technological complement for
blended learning methods. However, further studies are necessary to
better understand its impact on student learning outcomes. Future
research should consider students’ skills with new technology, average
class attendance, and the time and number of reviews invested in
studying.
Declaration of Competing Interest
The authors declared no potential conicts of interest with respect to
the research, authorship and/or publication of this article.
Acknowledgments
This research did not receive any specic grant from funding
agencies in the public, commercial, or not-for-prot sector. This study
has been possible thanks to the collaboration agreement signed with the
University of Valencia, Spain (Record Code: 25742). We would also like
to thank the teachers and students of the psychology degree at the
University of Valencia who participated in the GoKoan method evalu-
ation experience.
References
Adedoyin, O. B., & Soykan, E. (2020). Covid-19 pandemic and online learning: The
challenges and opportunities. Interactive Learning Environments. https://doi.org/
10.1080/10494820.2020.1813180. Advance online publication.
Ahmed, A., & Osman, M. E. (2020). The effectiveness of using Wiziq interaction
plataform on students’ achievement, motivation, and attitudes. Turkisk online Journal
of Distance Education, 21(1), 19–30.
Almasi, M., Zhu, C., & Machumu, H. (2018). Teaching, social, and cognitive presences
and their relations to students’ characteristics and academic performance in blended
learning courses in a Tanzanian University. Afrika Focus, 31(1), 73–89.
´
Alvarez, A., Martin, M., Fern´
andez-Castro, I., & Urretavizcaya, M. (2013). Blended
traditional teaching methods with learning environments: Experience, cyclical
evaluation process and impact with MAgAdI. Computers & Education, 68, 129–140.
https://doi.org/10.1016/j.compedu.2013.05.006.
Awidi, I. T., & Paynter, M. (2018). The impact of a ipped classroom approach on student
learning experience. Computers & Education, 128, 269–283. https://doi.org/
10.1016/j.compedu.2018.09.013.
Baddeley, A. D., & Longman, D. J. A. (1978). The inuence of length and frequency of
training sessions on the rate of learning to type. Ergonomics, 21, 627–635.
Banihashem, S. K., Aliabadi, K., Pourroostae, S., Delaver, A., & NiliAhmadabadi, M.
(2018). Learning analytics: A systematic literature review. Interdisciplinary Journal of
Virtual Learning in Medical Science, 9(2). https://doi.org/10.5812/ijvlms.63024.
Bonk, C. J., & Graham, C. E. (2005). The handbook of blended learning: Global perspectives,
local designs. San Francisco, CA: Pfeiffer.
Clark-Carter, D. (2009). Quantitative psychological research: The complete student’s
companion (3rd ed.). Philadelphia, Pennsylvania: Psychology Press.
Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). New
Jersey, NY: Lawrence Erlbaum.
Corell, A., Regueras, L. M., Verdú, E., Verdú, M. J., & De Castro, J. P. (2018). Effects of
competitive learning tools on medical students: A case study. PloS One, 13(3), Article
e0194096. https://doi.org/10.1371/journal.pone.0194096.
Dehghanzadeh, S., & Jafaraghaee, F. (2018). Comparing the effects of traditional lecture
and ipped classroom on nursing students’ critical thinking disposition: A quasi-
experimental study. Nurse Education Today, 71, 151–156. https://doi.org/10.1016/j.
nedt.2018.09.027.
M.J. N´
acher et al.
Studies in Educational Evaluation 70 (2021) 101026
9
Donnelly, R. (2010). Harmonizing technology with interaction in blended problem-based
learning. Computers & Education, 54, 350–359. https://doi.org/10.1016/j.
compedu.2009.08.012.
Ebbinghaus, H. (1885). Über das Ged¨
achtnis.
UntersuchungenzurexperimentellenPsychologie, Duncker and Humblot, Leipzig.
Translated to English by H.A. Ruger and C.E. Bussenius (1913). In H. Ebbingahaus
(Ed.), Memory: A contribution to experimental psychology. New York: Teachers College,
Columbia University.
Ebbinghaus, H. (1913). Memory: A contribution to experimental psychology. H.A. Ruger and
C.E. Bussenius (translators). New York: Teachers College, Columbia University.
Faul, F., Erdfelder, E., Lang, A. G., & Buchner, A. (2007). G*Power 3: A exible statistical
power analysis program for the social, behavioral, and biomedical sciences. Behavior
Research Methods, 39, 175–191. https://doi.org/10.3758/BF03193146.
Ferguson, R. Y., & Clow, D. (2017). Where is the evidence? A call to action for learning
analytics. In LAK’ 17. Proceedings of the Seventh International Learning Analytics &
Knowledge Conference, ACM International Conference Proceeding Series (pp. 56–65).
Garrison, D. R., & Kanuka, H. (2004). Blended learning: Uncovering its transformative
potential in higher education. The Internet and Higher Education, 7(2), 95–105.
https://doi.org/10.1016/j.iheduc.2004.02.001.
Gilboy, M. B., Heinerichs, S., & Pazzaglia, G. (2015). Enhancing student engagement
using the ipped classroom. Journal of Nutrition Education and Behavior, 47(1),
109–114. https://doi.org/10.1016/j.jneb.2014.08.008.
González-Gómez, D., Airado, D., Cañada-Cañada, F., & Su-Jeong, J. (2015).
A comprehensive application to assist in acid−base titration self-learning: An
approach for high school and undergraduate students. Journal of Chemical Education,
92(5), 855–863. https://doi.org/10.1021/ed5005646.
Graham, C. R. (2019). Current research in blended learning. In M. G. Moore, &
W. C. Diehl (Eds.), Handbooks of distance learning (4th, pp. 173–188). New York NY:
Routledge.
Harding, A., Kaczynski, D., & Wood, L. (2005). Evaluation of blended learning: Analysis
of qualitative data. September Paper Presented at the UniServe Science Blended
Learning Symposium.
Hinojo, F. J., Mingorance, A. C., Trujillo, J. M., Aznar, I., & C´
aceres, M. P. (2018).
Incidence of the ipped classroom in the physical education students’ academic
performance in university contexts. Sustainability, 10(5), 1–13. https://doi.org/
10.3390/su10051334.
Hooshyar, D., Pedaste, M., & Yang, Y. (2020). Mining educational data to predict
students’ performance through procrastination behavior. Entropy, 22, 12. https://
doi.org/10.3390/e22010012.
Hung, H.-T. (2015). Flipping the classroom for English language learners to foster active
learning. Computer Assisted Language Learning, 28(1), 81–96. https://doi.org/
10.1080/09588221.2014.967701.
Jebraeily, M., Pirnejad, H., Feizi, A., & Niazkhani, Z. (2020). Evaluation of blended
medical education from lecturers’ and students’ viewpoint: A qualitative study in a
developing country. BMC Medical Education, 20, 482. https://doi.org/10.1186/
s12909-020-02388-8.
Jost, N. S., Jossen, S. L., Rothen, N., & Martarelli, C. S. (2021). The advantage of
distributed practice in a blended learning setting. Education and Information
Technologies. https://doi.org/10.1007/s10639-020-10424-9. Advance online
publication.
Kassem, M. (2016). Flipping the Literature classroom: Fostering EFL students’
achievement and autonomy. Journal of Wadi El Nile for Human, Social and Educational
Research and Studies, Faculty of Arts, Cairo University, 9(1), 1–28. https://doi.org/
10.3991/ijet.v13i09.7792.
Kim, K. R., & Seo, E. H. (2015). The relationship between procrastination and academic
performance: A meta-analysis. Personality & Individual Differences, 82, 26–33.
https://doi.org/10.1016/j.paid.2015.02.038.
Kline, R. (2011). Principles and practice of structural equations modelling (2nd ed.). London,
England: Guilford Press.
Kraleva, R., Sabani, M., & Kralev, V. (2019). An analysis of some learning management
systems. International Journal on Advanced Science, Engineering and Information
Technology, 9(4), 1190–1198. https://doi.org/10.18517/ijaseit.9.4.9437.
Lança, C., & Bjerre, A. (2018). A retrospective study of orthoptic students’ and teaching
experience with the introduction of technology promoting a blended learning
environment. British and Irish Orthoptic Journal, 14(1), 56–63. https://doi.org/
10.22599/bioj.119.
Lim, D. H., & Morris, M. L. (2009). Learner and instructional factors inuencing learning
outcomes within a blended learning environment. Journal ofEducational Technology
& Society, 12(4), 282–293.
Liu, Q., Peng, W., Zhang, F., Hu, R., Li, Y., & Yan, W. (2016). The effectiveness of blended
learning in health professions: Systematic review and meta-analysis. Journal of
Medical Internet Research, 18(1), e2. https://doi.org/10.2196/jmir.4807.
Lodge, J. M., & Corrin, L. (2017). What data and analytics can and do say about effective
learning. NPJ Science of Learning, 2(5). https://doi.org/10.1038/s41539-017-0006-
5.
Long, P. H., & Siemens, G. (2011). Penetrating the fog: Analytics in learning and
education. EDUCAUSE Review, 46(5), 30–40.
L´
opez-Ozieblo, R. (2018). Independent ESP learners: The case for blended learning. In
R. Mu˜
noz-Luna, & L. Taillefer (Eds.), Integrating information and communication
technologies in English for specic purposes. English language education, 10 pp. 37–53).
Cham: Springer.
L´
opez-P´
erez, M. V., P´
erez-L´
opez, M. C., & Rodríguez-Ariza, L. (2011). Blended learning in
higher education: Students’ perceptions and their relation to outcomes. Computers &
Education, 56(3), 818–826. https://doi.org/10.1016/j.compedu.2010.10.023.
Macdonald, J. (2008). Blended learning and online tutoring (2nd ed.). Hampshire, UK:
Gower.
Macedo-Rouet, M., Ney, M., Charles, S., & Lallich-Boidin, G. (2009). Students’
performance and satisfaction with Web vs. Paper-based practice quizzes and lecture
notes. Computers & Education, 53, 375–384. https://doi.org/10.1016/j.
compedu.2009.02.013.
Makhdoom, N., Khoshhal, K. I., Algaidi, S., Heissam, K., & Zolaly, M. A. (2013). Blended
learning’ as an effective teaching and learning strategy in clinical medicine: A
comparative cross-sectional university-based study. Journal of Taibah University
Medical Sciences, 8(1), 12–17. https://doi.org/10.1016/j.jtumed.2013.01.002.
McLaughlin, J. E., Roth, M. T., Glatt, D. M., Gharkholonarehe, N., Davidson, C. A.,
Grifn, L. M., & Mumper, R. J. (2014). The ipped classroom: A course redesign to
foster learning and engagement in a health professions school. Academic Medicine, 89
(2), 236–243. https://doi.org/10.1097/ACM.0000000000000086.
Mohammad, S., & Job, M. A. (2012). Condence–motivation–satisfaction–performance
(CMSP) analysis of blended learning system in the Arab Open University Bahrain.
International Journal of Information Technology & Business Management, 3, 23–29.
Murillo, L. R., L´
opez, J. A., & Godoy, A. L. (2019). How the ipped classroom affects
knowledge, skills, and engagement in higher education: Effects on students’
satisfaction. Computers & Education, 141. https://doi.org/10.1016/j.
compedu.2019.103608. Advance on-line publication.
Ødegaard, N. B., Myrhaug, H. T., Dahl-Michelsen, T., & Røe, Y. (2021). Digital learning
designs in physiotherapy education: A systematic review and metaanalysis. BMC
Medical Education, 21, 48. https://doi.org/10.1186/s12909-020-02483-w.
Poon, J. (2013). Blended learning: An institutional approach for enhancing students’
learning experiences. Journal of Online Learning and Teaching, 9(2), 271. Retrieved
from https://search.proquest.com/docview/1500421423?accountid=14777.
Rapanta, C., Botturi, L., Goodyear, P., Gu`
ardia, L., & Koole, M. (2020). Online university
teaching during and after the Covid-19 crisis: Refocusing teacher presence and
learning activity. PostdigitalScience and Education, 2, 923–945. https://doi.org/
10.1007/s42438-020-00155-y.
Regueras, L. M., Verdú, E., Mu˜
noz, M. F., P´
erez, M. A., De Castro, J. P., & Verdú, M. J.
(2009). Effects of competitive E-Learning tools on higher education students: A case
study. IEEE Transactions on Education, 52(2), 279–285. Retrieved from http://ieeexpl
ore.ieee.org/lpdocs/epic03/wrapper.htm?arnumber=4909477.
Steel, P. (2007). The nature of procrastination: A meta-analytic and theoretical review of
quintessential self-regulatory failure. Psychological Bulletin, 133(1), 65–94. https://
doi.org/10.1037/0033-2909.133.1.65.
Street, S. E., Gilliland, K. O., McNeil, C., & Royal, K. (2015). The ipped classroom
improved medical student performance and satisfaction in a pre-clinical physiology
course. Medical Science Educator, 25(1), 35–43. https://doi.org/10.1007/s40670-
014-0092-4.
Twigg, C. A. (2003). Improving learning and reducing costs: Lessons learned from Round 1 of
the Pew grant program in course redesign. Troy, NY: Center for Academic
Transformation.
Vall´
ee, A., Blacher, J., Cariou, A., & Sorberts, E. (2020). Blended learning compared to
traditional learning in medical education: Systematic review and meta-analysis.
Journalof Medical Internet Research, 22(8), Article e16504. https://doi.org/10.2196/
16504.
Vela-P´
erez, M., Hern´
andez-Estrada, A., Tirado-Domínguez, G., et al. (2017). Learning
analytics to classify students according to their activity in moodle. EDULEARN17
Proceedings, 1166–1172.
Vo, H. M., Zhu, C., & Diep, N. A. (2017). The effect of blended learning on student
performance at course-level in higher education: A meta-analysis. Studies in
Educational Evaluation, 53, 17–28. https://doi.org/10.1016/j.stueduc.2017.01.002.
Woltering, V., Herrler, A., Spitzer, K., & Spreckelsen, C. (2009). Blended learning
positively affects students’ satisfaction and the role of the tutor in the problem-based
learning process: Results of a mixed-method evaluation. Advances in Health Sciences
Education: Theory and Practice, 14, 725–738. https://doi.org/10.1007/s10459-009-
9154-6.
Woods, R., Jason, D. B., & Hopper, D. (2004). Hybrid structures: Faculty use and
perception of web-based courseware as a supplement to face-face instruction. The
Internet and Higher Education, 7, 281–297. https://doi.org/10.1016/j.
iheduc.2004.09.002.
Xiuhan, L., & Samuel Kai Wah, C. (2020). Exploring the effects of gamication pedagogy
on children’s reading: A mixed-method study on academic performance, reading-
related mentality and behaviors, and sustainability. British Journal of Educational
Technology. https://doi.org/10.1111/bjet.13057. Advance online publication.
Yigit, T., Koyun, A., Yuksel, A., & Cankaya, I. (2013). Evaluation of blended learning
approach in computer engineering education. Procedia-Social & Behavioral Sciences,
141, 807–812. https://doi.org/10.1016/j.sbspro.2014.05.140.
Zacharis, N. Z. (2015). A multivariate approach to predicting student outcomes in web-
enabled blended learning courses. The Internet & Higher Education, 27, 44–53.
https://doi.org/10.1016/j.iheduc.2015.05.002.
M.J. N´
acher et al.
