ArticlePDF Available

How learning analytics can early predict under-achieving students in a blended medical education course

Authors:

Abstract and Figures

Aim: Learning analytics (LA) is an emerging discipline that aims at analyzing students’ online data in order to improve the learning process and optimize learning environments. It has yet un-explored potential in the field of medical education, which can be particularly helpful in the early prediction and identification of under-achieving students. The aim of this study was to identify quantitative markers collected from students’ online activities that may correlate with students’ final performance and to investigate the possibility of predicting the potential risk of a student failing or dropping out of a course. Methods: This study included 133 students enrolled in a blended medical course where they were free to use the learning management system at their will. We extracted their online activity data using database queries and Moodle plugins. Data included logins, views, forums, time, formative assessment, and communications at different points of time. Five engagement indicators were also calculated which would reflect self-regulation and engagement. Students who scored below 5% over the passing mark were considered to be potentially at risk of under-achieving. Results: At the end of the course, we were able to predict the final grade with 63.5% accuracy, and identify 53.9% of at-risk students. Using a binary logistic model improved prediction to 80.8%. Using data recorded until the mid-course, prediction accuracy was 42.3%. The most important predictors were factors reflecting engagement of the students and the consistency of using the online resources. Conclusions: The analysis of students’ online activities in a blended medical education course by means of LA techniques can help early predict underachieving students, and can be used as an early warning sign for timely intervention.
Content may be subject to copyright.
How learning analytics can early predict under-achieving students in a
blended medical education course
Mohammed Saqr
a,b
, Uno Fors
b
and Matti Tedre
b
a
College of Medicine, Qassim University, Qassim, Kingdom of Saudi Arabia;
b
Department of Computer and System Sciences (DSV),
Stockholm University, Kista, Sweden
ABSTRACT
Aim: Learning analytics (LA) is an emerging discipline that aims at analyzing studentsonline data in order to improve the
learning process and optimize learning environments. It has yet un-explored potential in the field of medical education,
which can be particularly helpful in the early prediction and identification of under-achieving students. The aim of this study
was to identify quantitative markers collected from studentsonline activities that may correlate with studentsfinal perform-
ance and to investigate the possibility of predicting the potential risk of a student failing or dropping out of a course.
Methods: This study included 133 students enrolled in a blended medical course where they were free to use the learning
management system at their will. We extracted their online activity data using database queries and Moodle plugins. Data
included logins, views, forums, time, formative assessment, and communications at different points of time. Five engage-
ment indicators were also calculated which would reflect self-regulation and engagement. Students who scored below 5%
over the passing mark were considered to be potentially at risk of under-achieving.
Results: At the end of the course, we were able to predict the final grade with 63.5% accuracy, and identify 53.9% of at-risk
students. Using a binary logistic model improved prediction to 80.8%. Using data recorded until the mid-course, prediction
accuracy was 42.3%. The most important predictors were factors reflecting engagement of the students and the consistency
of using the online resources.
Conclusions: The analysis of studentsonline activities in a blended medical education course by means of LA techniques
can help early predict underachieving students, and can be used as an early warning sign for timely intervention.
Introduction
E-learning has become an essential part of current health-
care education, and courses delivered or supported by
technology are on the rise in size, number, and scale of
adoption. (Dahlstrom et al. 2014; Liu et al. 2016). Using
technology in education extends through a wide range of
applications like e-logistics, e-administration, e-assessment,
digital course content, multimedia, simulation, collabor-
ation, communication, and e-support, to name a few. Most
of these technologies can be bundled in comprehensive
learning management systems (LMSs) (Ellaway & Masters
2008; Liu et al. 2016).
While traditional teaching methods leave little behind
to track, modern LMSs generate vast amounts of informa-
tion about students, their use of the material, and the
learning contexts in terms of records, logs, interactions,
and other digital footprints (Ferguson 2012; Siemens 2013).
The availability of these datasets, increased computer
power, skills learnt in business analytics, the pressure
toward better teaching and learning, personalization of the
content, and improving LMSs has led to increased interest
in learning analytics (LA) development and research (Brown
2011; Ferguson 2012; Dahlstrom et al. 2014; Papamitsiou &
Economides 2014).
LA is an emerging, relatively new, and rapidly develop-
ing discipline (Conde & Hern
andez-Garc
ıa2015; Rienties
et al. 2016) that aims at measurement, collection, analysis
and reporting of these data and their contexts, for
purposes of understanding and optimizing learning and
the environments in which it occurs(Siemens 2013).
Analytics have two main functions: to provide information
about the current status of the learners and their learning
process or provide an insight about what is yet to happen
on the individual or group level in the future (Ellaway et al.
2014).
Analyzing data collected from learnersinteractions
within LMSs and information systems has been the com-
mon approach to LA and the one that has proven most
promising (Ramos & Yudko 2008; Macfadyen & Dawson
2012; Lockyer et al. 2013; Agudo-Peregrina et al. 2014;
Ga
sevi et al. 2015). Course Signalsby Purdue University is
Practice points
Learning analytics (LA) is an emerging field that
uses studentsonline activities to learn about their
online behavior for the sake of improving their
outcome and optimizing learning.
Analyzing online activity can highlight active and
inactive students, which can be used as an alert to
educators and academic supervisors.
LA can be used to early predict grades and
performance.
The application of LA might help early interven-
tion that has the potential to decrease dropout
rate.
CONTACT Mohammed Saqr saqr@qumed.edu.sa College of Medicine, Qassim University, PO Box: 6655, 51452 Qassim, Kingdom of Saudi Arabia
School of Computing, University of Eastern Finland, PO Box 111, Joensuu, Finland.
ß2017 Informa UK Limited, trading as Taylor & Francis Group
MEDICAL TEACHER, 2017
VOL. 39, NO. 7, 757767
http://dx.doi.org/10.1080/0142159X.2017.1309376
an early example of building a warning and feedback sys-
tem for students and teachers using LA principles. It has
been reported to have a positive effect on student reten-
tion and teachersawareness of their students (Pistilli &
Arnold 2010).
LA has been shown to enable effective, automatic track-
ing of studentsengagement along the course (Macfadyen
& Dawson 2010,2012; Wolff et al. 2013; Cruz-Benito et al.
2015; Tempelaar et al. 2015;Ga
sevi
c et al. 2016). The
insights generated by LA can be shared by course teachers,
academic supervisors, and administrators (Arnold & Pistilli
2012; Howard et al. 2016; Rienties et al. 2016). Those
insights could help identify students at risk of under-
achievement where an early intervention can lead to a
meaningful change (Macfadyen & Dawson 2010; Lockyer
et al. 2013; Tempelaar et al. 2015;Ga
sevi
c et al. 2016;
Howard et al. 2016; Rienties et al. 2016). Although trad-
itional assessment methods offer this kind of feedback,
their results most often come too late for a possible action
or a significant intervention (Macfadyen & Dawson 2010;
Cruz-Benito et al. 2015).
Education in the healthcare sector is under a lot of pres-
sure to respond efficiently and timely to the rapidly chang-
ing scientific, societal, and social environment, as well as to
keep programs modern and connected to the communities
it serves (Ellaway & Masters 2008; Ellaway et al. 2014;
Vaitsis et al. 2014; Liu et al. 2016). Another challenge facing
medical schools is underachieving and potential attrition;
an issue that may be a symptom of preventable problems
in the medical education (in selection of students, curricula,
teaching methods, assessment or policies) (ONeill et al.
2011). While the problem of attrition in medical education
seems to incur a substantial cost, it is still poorly studied
and most of the published studies have focused on
studentsattributes at the point of admission (ONeill et al.
2011; Stegers-Jager et al. 2015), which only explained 30%
of variance in performance, recent research indicates that
LA can significantly improve the predictability of academic
performance and hence can help solve the problem
(Macfadyen & Dawson 2010; Agudo-Peregrina et al. 2014;
Papamitsiou & Economides 2014; Wolff et al. 2014;
Tempelaar et al. 2015).
Although the benefits of using LA in education have
been conceptually justified (Richards 2011; Ellaway et al.
2014;Ga
sevi et al. 2015; Tempelaar et al. 2015; Rienties
et al. 2016) and the need was recently recognized in the
medical education literature (Doherty et al. 2014; Ellaway
et al. 2014), research in medical setting is rare. Based on
results from other fields, we expect that LA can have a con-
siderable a considerable, yet unexplored potential for
healthcare education. Ellaway et al. (2014) summarized this
need as follows health professional educators will need to
be ready to deal with the complex and compelling dynam-
ics of analytics and Big Data. We therefore need to explore,
discuss, and critique these emerging techniques to develop
a robust understanding of their strengths and limitations in
the service of health professional education.
This research study was performed to analyze data of
studentsonline activity in a blended medical education
course in Saudi Arabia in order to identify quantitative
markers that correlate with studentsperformance and
might be used as early warning signs for possible data
driven measures.
The research questions of this study are:
1. Which tracking variables best correlate with student
performance?
2. To what extent can the analysis of studentsonline activ-
ities be used to predict student grades, and identify the
potential risk of a student failing or dropping a course?
Methodology
The analysis of student data followed a standard procedure
used in data mining research and analytics (Dean 2014;
Gandomi & Haider 2014; Wolff et al. 2014):
Acquisition and recording: Acquiring the data from differ-
ent sources.
Preparing the data: Matching and cleaning mislabeled
data, excluding incomplete records and appropriate
annotation of data types, combining the data into one
master table.
Performing exploratory data analysis (EDA): Exploring
data by testing the relationships between different vari-
ables to discover possible relationships, patterns, rules
that could help identify the potential predictors. EDA
does not require a prior hypothesis in contrast to for-
malistic scientific methodology that tests a previously
known theory (Velleman & Hoaglin 2012).
Building the predictive model: Predicting studentsout-
come and identifying at-risk students using appropriate
predictive models. We used regression models, as they
are among the most common predictive models used in
education research at large (Peng et al. 2002), and in
analytics research (Ramos & Yudko 2008; Macfadyen &
Dawson 2010;Ga
sevi
c et al. 2016; Howard et al. 2016),
available in most statistical packages, and can be eval-
uated in several ways (Peng et al. 2002; Bewick et al.
2005). Two types of regression models according to the
type of outcome to be predicted:
Automatic linear modeling (ALM) for grade prediction,
ALM uses a group of predicting factors to predict a
single scaled outcome. It offers improvements over
traditional methods in two main areas. First, auto-
matic variable selection, which is useful when the
number of variables is high (Filippou et al. 2015). The
second is automatic data preparation, data prepar-
ation is a popular concept in data science that
includes re-classifying continuous variables with less
than 5 unique values as ordinal and re-classifying
ordinal values with more than 10 unique values as
continuous variables. It normalizes outliers or extreme
values (predictors that lie beyond 3 SDs), so that they
do not to exert an exaggerated influence on the
model. And finally, it does a supervised merge of
similar predictors (Yang 2013).
Binary logistic regression (BLR) for prediction of at-risk
students: BLR is a powerful model for the prediction
of dichotomous outcomes like pass/fail or at-risk/safe.
It overcomes some of the restrictive assumptions of
linear regressions like linearity, normality and equal
variances. The test has a large array of tests to evalu-
ate its performance (the 2 log likelihood, Cox &
Snell R
2
, the Omnibus Tests of Model Coefficients,
758 M. SAQR ET AL.
Hosmer and Lemeshow goodness of fit) (Peng et al.
2002; Bewick et al. 2005).
Evaluation of predictive accuracy: Receiver operating
characteristic (ROC) plots the sensitivity (true positive
rate) of each model versus 1-specificityor (false
positive rate). The area under the curve (AUC) is a
quantification of model accuracy, where 0.5 means a
worthless model and 1.0 represents a perfect model
(Bewick et al. 2005;G
onen 2006).
Data collection (acquisition and recording)
This study was preceded by a pilot study to determine
feasibility, and to identify engagement parameters and the
tracking variables (Alghasham et al. 2013). This study was
performed on the students of the course Man and envi-
ronmentduring 20132014 at Qassim University, Kingdom
of Saudi Arabia. This is the second course in the medical
program, and the first to teach the medical subject after
an introductory course. The intention of studying students
in the first year was to be able to capture the full spec-
trum of freshly admitted students before some underach-
ievers may drop out. The study included 133 students
enrolled in a blended course where they were free to use
the LMS at their will. There was no incentive or punish-
ment of using the LMS apart from studentsself-perceived
benefit.
At Qassim College of Medicine, the Moodle LMS is used
as the main platform for learning management, Moodle
produces robust logs of studentsactivities; however, the
available reporting tools are deficient, and Moodle does
not have built-in analytic tools (Falakmasir & Habibi 2010).
Therefore, we used MySQL database queries (SQL) and five
add-on tools.
First, Attendance Register module was used to report
total time spent by a student in a course (Moodle plugins
directory: Attendance Register). Second, Configurable
Reports was used to run SQL queries to generate custom
reports about studentsactivities like number of course
views, forum posts or reads, and course edits (Moodle plu-
gins directory: Configurable Reports). Third, Analytics
Graphs was used to calculate total unique days of course
access, total number of course views, and total number of
course views) (Moodle plugins directory: Analytics Graphs).
Fourth, Mailchimp e-mail tracking was used to track
studentsresponse to e-mails, and the frequency of open-
ing course related e-mails (About Open Tracking
jMailChimp.com: KB Article). Finally, NodeXL, was used to
calculate betweenness centrality (Smith et al. 2009).
Collected data
Collected data were divided into six categories and detailed
in Table 1.
Engagement sub-scores
Based on the findings of the pilot study (Alghasham et al.
2013), five engagement indicators were identified. They
reflect regularity of using the LMS and balance for the fact
that some students would be highly active over a short
time and then go into periods of inactivity (Richards 2011;
Shea et al. 2013; Cruz-Benito et al. 2015; Panzarasa et al.
2016). The indicators were calculated as follows:
By login: a student was considered engaged in a certain
week when he/she logs in 3 days or more in that week,
that student is then given a score of one. The login
engagement sub-indicator is the sum of scores of the 6
weeks. By course views: a score of one is given to the stu-
dent when he/she views the course materials more than a
1Z-score of mean course views. The views engagement
sub-indicator is the sum of scores of the 6 weeks. By forum
posts: a student is given the score of 1 when he/she posts
two or more posts in a certain week. The posting engage-
ment sub-indicator is the sum of the 6 weeks.
By time: a student is given the score of 1 when he
spends more than a 1Z-score of course average time of
all students. The time engagement sub-indicator is the sum
of the 6 weeks. By formative assessment: a score of 1 is
given to a student if he/she tried an assessment regardless
of the score. The formative engagement sub-indicator is
the sum of the 6 weeks.
At-risk students
There are several standard setting methods that might be
used to set the criteria for not passing a course or being a
borderline student (Tekian & Norcini 2015). The choice of
the method relies largely on the course and purpose of the
standard setting. For this study, we followed the procedure
described by Norcini (2003): a panel of expert judges were
formed to set a cut point that separates students who
barely pass the course (was at-risk) from students who
clearly pass (Macfadyen & Dawson 2010). The panel esti-
mated that 5% over the passing mark would define this cut
point; accordingly, students were classified into two main
categories:
Potentially safe (coded as Safe): Have a final score of
65% or more.
Potentially at-risk (coded as At-risk): Have a final score
below 65%.
Research ethics
The study was approved by the Medical Research Ethics
Committee of Qassim College of Medicine. All users of
Qassim College of Medicine LMS sign an online privacy pol-
icy that detail possible use of data for research and user
protection guarantees (Qassim College of Medicine).
All data were anonymized, personal identifying information
were masked, no private data or personal information were
used in the analysis or published. College Privacy guidelines
and policies of dealing with studentsdata were strictly fol-
lowed (Qassim College of Medicine 2014). It is also import-
ant to mention here that using e-learning portal is neither
graded nor mandatory; only depending on student self-
interest, and that specific course did not contain any
online-graded assignments.
Participants
The study initially included 145 students (44 females and
101 males). However, 12 were excluded due to incomplete
MEDICAL TEACHER 759
data and delayed enrollment in the LMS; thus, the final
number was 133 (43 females and 90 males) over the period
of 6 weeks.
Results
The study was performed using an EDA approach, testing
all possible parameters to try to identify the significant
ones (Macfadyen & Dawson 2010; Velleman & Hoaglin
2012). Studies in similar environments have been rare and
the previous studies have not been able to find a general
set of metrics that can be used as predictors of student
achievement (Macfadyen & Dawson 2010; Agudo-Peregrina
et al. 2014; Tempelaar et al. 2015; Wise & Shaffer 2015;
Rienties et al. 2016). A previous study in a medical educa-
tion course would have been helpful (Ga
sevi
c et al. 2016),
but unfortunately to the best of our knowledge, we could
not identify a study with same scope of metrics and com-
parable design as this actual project.
The study identified 64 possible tracking variables in six
main categories, we report here the most important and
significant indicators at mid-course, second half, and end of
course. First, correlation coefficient was calculated to iden-
tify the possible indicators, followed by the prediction of
student grade, then we try to predict the at-risk students at
the end of course and whether this is possible at mid-
course or not.
Correlations
In Table 2, the findings related to correlations are displayed
and the most interesting findings were the following.
Content creation/interaction: There was a positive and
significant correlation between the studentsfinal grade
and interaction/content creation variables. The most
important were total edits or created content r(131) ¼0.31,
p<0.01, number of edits in the first half of course
r(131) ¼0.3, p<0.01, total posts initiated by student
r(131) ¼0.29, p<0.01 and total posts and replies
(131) ¼0.29, p<0.01.
Views/hits: The correlation between views and final
grade was weak in most parameters, except for the number
of resources accessed which showed moderate correlation
r(131) ¼0.32, p<0.01, followed by the total hits on resour-
ces r(131) ¼0.25, p<0.01.
Login frequency: The frequency of logins had the stron-
gest correlation with final grade r(131) ¼0.47, p<0.01. It
was moderately correlated with the number of logins in
the first half of course r(131) ¼0.31, p<0.01 and the num-
ber of logins in the second half of course r(131) ¼0.36,
p<0.01.
Time: The total time showed a weak correlation with
final grade r(131) ¼0.22, p¼0.01 and the other parameters
of time showed similar low correlation.
Formative assessment: The formative assessment parame-
ters were the most consistent and most of them showed
moderate correlation with the final grade. The number of
the exams attempted was the highest r(131) ¼0.46,
p<0.01, followed by the grade r(131) ¼0.43, p<0.01.
Communications: There was no significant correlation
between following course communications and final grade.
Engagement sub-indicators showed more consistent
and higher correlation coefficients with final grades than
simple counts of activities; especially, for the parameters
that measured studentsconsistency of using the LMS
resources (see Table 3). The highest was total
unique days of access r(131) ¼0.46, p<0.01, followed by
the formative assessment sub-indicator r(131) ¼0.45 login
sub-indicator r(131) ¼0.41, p<0.01, r¼0.41 and 0.46
(Table 3).
Table 2. Bivariate correlations between LMS tracking variables and final
grade.
Parameter
Correlation
(r)
Sig.
(two-tailed)
Interaction and content creation
Total posts initiated by a student 0.29 <0.01
Total posts and replies by a student 0.29 <0.01
Number of posts in the first half of course 0.27 <0.01
Number of posts in the second half of course 0.23 <0.01
Total edits of content created 0.31 <0.01
Number of edits in the first half of course 0.30 <0.01
Number of edits in the second half of course 0.22<0.01
Hits and views
Total times forums were read 0.25 <0.01
Total hits on recourses 0.28 <0.01
Total Hits on course information 0.210.02
Total course hits 0.26 <0.01
Total views before course started 0.037 0.672
Number of hits in the first half of course 0.24 <0.01
Number of hits in the second half of course 0.24 <0.01
Number of resources accessed 0.32 <0.01
Logins and course access
Total logins 0.47 <0.01
Number of logins in the first half of course 0.31 <0.01
Number of logins in the second half of course 0.36 <0.01
Time
Time spent online 0.220.01
Total time spent online first half of course 0.23 <0.01
Time spent in the second half of course 0.180.04
Formative assessment
Formative assessment grade 0.43 <0.01
Mid-course formative assessment grade 0.42 <0.01
Number attempted the formative quiz 0.46 <0.01
End of course formative assessment grade 0.25 <0.01
Communications
Response to communications 0.05 0.57
Total views of news forum 0.068 0.44
Correlation is significant at 0.05 level (2-tailed); Correlation is significant
at 0.01 level (2-tailed).
Table 1. Detailed description of collected data.
Parameter Collected data
Logins Weekly, mid-course, and total course logins
Logins before and after the end of the course
(pre and post-term)
Total number of days with course access
Views (hits) Daily number of views, weekly, mid-course
Total course views, number of unique resources
accessed and types of the accessed resources
Forums Weekly, at mid-course and the overall total of num-
ber created posts
Reads, replies and total number of edits made in
course (creation of new content)
Betweenness centrality was calculated by NodeXL to
get an idea of how influential the participation of a
student was in the forms, we calculated
Time Weekly, at mid-course and the overall total time
spent online using educational materials, time navi-
gating profile or viewing non-educational materials
was excluded
Formative
assessment
Grades of each formative assessment and participa-
tion in assessment regardless of the submission of
the answers
760 M. SAQR ET AL.
The prediction of student grade
Studentsonline behavior involves a number of factors and
therefore a simple univariate correlation cannot simply pre-
dict the final grade; for example, spending more time
online does not usually translate into higher achievement
unless that time is spent learning, participating in activities
and interacting (Macfadyen & Dawson 2010; Shea et al.
2013; Ballard & Butler 2016). Therefore, it would be errone-
ous to rely on just the previous correlations to draw con-
clusions about studentsachievement.
Since univariate correlation and simple linear regression
have limitations (inability to identify and handle outliers,
prone to type I/II errors) (Yang 2013) and there was no
standard tracking variables that could be used as a guide
for this study (Macfadyen & Dawson 2010). We sought a
predictive model that can combine all potential tracking
variables of studentsonline activity. We found ALM a
relatively new method introduced in SPSS 19 to be the
most suitable for the analysis of this study (Yang 2013). The
accuracy (adjusted R
2
) of ALM for predicting the final grade
was 63.5%. Figure 1 presents a scatterplot of the relation-
ship between the actual and predicted grade.
Figure 2 shows how much each factor contributed to
the prediction model. Total predictor importance is 1.0, and
the most important and statistically significant predictors
were login engagement sub-indicator (0.16), number of
views of course information (0.11), formative assessment
sub-indicator (0.10), and posting engagement sub-indicator
(0.10).
Figure 1. A scatterplot summarizing the relationship between actual and predicted grade.
Figure 2. Visualization of predictor importance according to automatic linear modeling.
Table 3. Bivariate correlations between final grade and engagement sub-
indicators.
Parameters of engagement
Pearsons
correlation
Sig.
(two-tailed)
Login sub-indicator 0.41 <0.01
Formative assessment sub-indicator 0.45 <0.01
Posting sub-indicator 0.31 <0.01
Time sub-indicator 0.32 <0.01
Course views sub-indicator 0.28 <0.01
Total unique days with access 0.46 <0.01
Correlation is significant at 0.01 level (2-tailed).
MEDICAL TEACHER 761
Predicting at-risk students
Using results of ALM to classify students according to safety
level, the model identified 18 students as at-risk (14 true
positive, 4 false positives and missed 12), the sensitivity for
picking at-risk students was 53.85% (CI: 33.37%73.41%),
and the specificity was 96.26% (CI: 90.70%98.97%)). A v
2
test of independence was performed to examine the rela-
tionship between actual and predicted at-risk students.
ALM was more likely to correctly identify at-risk students in
53.8% of cases, v
2
(1, N¼133) ¼44.9, p<0.01; detailed
results are cross tabulated in Table 4.
To further illustrate the efficiency of ALM in classifying
students as at-risk or safe, in Figure 3, every circle repre-
sents a student, and while circles to the left side were at-
risk students, the green-colored circles are students cor-
rectly predicted to be at-risk, and on the right, side the
green circles are false positive.
Binary logistic regression
BLR successfully classified 21 out of 26 as at-risk students
and misclassified two students. The sensitivity of BLR was
80.8% (CI: 60.7%93.5%) and the specificity was 96.3% (CI:
93.4%99.8%). A v
2
test of independence was performed
to examine the relationship between actual and predicted
at-risk students. BLR was more likely to correctly identify
at-risk students in 80.8% of cases, v
2
(1, N¼133) ¼91,
p<0.01.
The 2 log likelihood was 43.01, Cox & Snell R
2
was
0.49, and Nagelkerke R
2
was 0.77. These findings are
strong indicators that a large portion of the variation of
the final grade can be explained by the selected
predictors. The Omnibus Tests of Model Coefficients was
statistically significant (v
2
¼88.35, df ¼39, p<0.01) and
Hosmer and Lemeshow goodness of fit was non-significant
(v
2
¼13.95 df ¼8, p¼0.08). Both results are further indica-
tions of the adequacy of fitness of the binary logistic
model (Table 5).
To what extent can tracking data at mid-course
predict at-risk students?
Using BLR to test the possibility of early predicting at-risk
students, we were able to detect 42.3% of at-risk students
correctly (CI: 23.35%63.08%) (Table 6;Figure 4).
The 2 log likelihood was 89.05, Cox & Snell R
2
was
0.27, and Nagelkerke R
2
was 0.43 indicating that a signifi-
cant part of the variation of the final grade could be
explained by the mid-course predictors. The Omnibus Tests
of Model Coefficients was significant (v
2
¼42.37, df ¼19,
p¼0.002) and Hosmer and Lemeshow goodness of fit was
non-significant (v
2
¼5.29 df ¼8, p¼0.73) indicating that
the logistic model adequately fits the data.
Table 4. Cross-tabulation of predicted and at-risk students using ALM.
Predicted
Actual
Percentage correctAt-risk Safe
At-risk 14 4 53.8%
Safe 12 103 96.3%
Total 26 107 133
Bold numbers denote correctly identified students at-risk.
Figure 3. Scatterplot of predicted against actual grade, green circles to left of the red dotted line represent correctly identified at-risk students (true positives)
using ALM.
Table 5. Cross-tabulation of predicted and at-risk students using the binary
logistic model with the cutoff of 0.50.
Predicted
Actual
Percentage correctAt-risk Safe
At-risk 21 2 80.8%
Safe 5 105 98.1%
Total 26 107 133
Bold numbers denote correctly identified.
Table 6. Cross-tabulation of predicted and at-risk students using mid-course
data.
Predicted
Actual
Percentage correctAt-risk Safe
At-risk 11 4 42.3%
Safe 15 103 96.3%
Total 26 107 133
Bold numbers denote correctly identified.
762 M. SAQR ET AL.
Comparing predictive models
BLR using at the end-of-course data was the most sensitive
and had excellent AUC value of 0.9, followed by linear
regression, and BLR at mid-course (Figure 5). Table 7
presents a detailed comparison between the three methods
and ROC area. Table 8 contains all the students who were
correctly or in-correctly identified by each model along
with the engagement indicators.
Figure 4. Scatterplot of predicted against actual grade at mid-term, blue circles to left of the vertical line represent correctly identified at-risk students (true
positives).
Figure 5. Receiver operating characteristic (ROC) comparing the predictive models used in the study, it shows that all three models have values more than 0.5
and that binary logistic regression was the most sensitive.
Table 7. Comparison between different predictive models AUC and sensitivities in prediction of at-risk students.
Model Sensitivity AUC p95% Confidence interval
Binary logistic regression 0.81 0.90 0.00 0.80.99
Automatic linear model 0.54 0.75 0.00 0.630.88
Mid-course Binary logistic regression 0.42 0.69 0.00 0.560.82
MEDICAL TEACHER 763
Discussion
In this study, we investigated the variables that best correlate
with studentsperformance and can be used to identify stu-
dents who might be at-risk of under-achievement for pos-
sible timely intervention. We have expanded over the
previous studies (Macfadyen and Dawson 2010; Wolff et al.
2014; Tempelaar et al. 2015;Ga
sevi
c et al. 2016) and included
multidimensional data about access, hits, time, forums, com-
munications, and social network parameters, as well as for-
mative assessments across different points in time.
We have calculated parameters (engagement sub-indica-
tors) that reflect consistency of using the LMS resources
and self-motivation. The concept was borrowed from mar-
keting and brand loyalty that is long recognized by market-
ers (Ballard & Butler 2016; Panzarasa et al. 2016). The
engagement sub-indicators also normalize the extremes of
use and temporary surges by some students, that adds
much to their overall counts and do not reflect a consistent
practice throughout the course. The indicators also serve as
an indirect measure of self-regulation in the course, since
e-learning is offered in a blended scenario and other activ-
ities are going beyond the LMS (Shea et al. 2013;
Cruz-Benito et al. 2015).
Our results indicated that the engagement indicators
showed consistent and significantly higher correlations with
the studentsperformance across all categories of measure-
ment. In contrast to the simple generic metrics, which
showed inconsistent and relatively weaker correlations with
studentsperformance. Parameters such as time and hits
(the most generic metrics) were the weakest, and
parameters that reflected motivation and disposition such
as taking the optional formative assessments, frequency of
access, and content creation were the best indicators of
studentsperformance.
The shortcomings of using generic parameters in ana-
lytics research have been previously recognized (Macfadyen
& Dawson 2010) and were further emphasized by recent
findings in large-scale studies (Conde & Hern
andez-Garc
ıa
2015; Rienties et al. 2016). They have produced conflicting
and inconsistent results from one study to the other, and
from course to course (Ramos & Yudko 2008; Tempelaar
et al. 2015;Ga
sevi
c et al. 2016), especially in blended scen-
arios (Agudo-Peregrina et al. 2014). They offer limited
insights to the understanding of the complex learning envi-
ronments or to the development of theory for LA (Conde &
Hern
andez-Garc
ıa2015;Ga
sevi et al. 2015). Reliance on
generic parameters has been further criticized for having a
detrimental role that has hampered the advance of LA as a
field (Conde & Hern
andez-Garc
ıa2015).
No study has established a standard set of variables that
fits all online courses, and studies comparing multiple
courses have actually indicated the opposite: variables and
indicators differ between course learning designs (Wolff et al.
2013; Agudo-Peregrina et al. 2014;Ga
sevi
c et al. 2016;
Rienties et al. 2016). That highlights the need to find new
sets of indicators that are contextually relevant to the course
design and the learning environment (Pistilli & Arnold 2010;
Ga
sevi
c et al. 2016). In this study, we proposed a set of indi-
cators (engagement sub-indicators) that reflect engagement
and self-regulation (Richards 2011; Shea et al. 2013; Cruz-
Table 8. All the students who were correctly or in-correctly (bold) identified by each model along with the engagement indicators.
Regularity
Actual
Identified by regression models
Serial Course views Formative assessment Login Posting Time ALM BLR BLR-mid-course
S1 0 0 6 2 3 Safe At-risk Safe Safe
S2 3 0 6 5 6 Safe At-risk Safe Safe
S3 3 2 6 5 6 Safe At-risk Safe Safe
S4 2 2 3 5 5 Safe At-risk Safe Safe
S5 0 0 6 3 5 Safe Safe At-risk Safe
S6 3 5 6 6 6 Safe Safe Safe At-risk
S7 0 3 6 0 2 Safe Safe Safe At-risk
S8 3 0 6 3 6 Safe Safe Safe At-risk
S9 2 3 5 3 6 Safe Safe At-risk At-risk
S10 0 2 6 0 6 At-risk At-risk Safe Safe
S11 2 0 5 2 6 At-risk At-risk At-risk Safe
S12 2 0 5 2 3 At-risk At-risk At-risk Safe
S13 2 0 5 5 6 At-risk At-risk At-risk Safe
S14 0 0 5 0 2 At-risk At-risk At-risk Safe
S15 2 0 6 5 6 At-risk At-risk At-risk Safe
S16 2 3 6 2 6 At-risk At-risk At-risk Safe
S17 0 3 3 0 2 At-risk At-risk At-risk Safe
S18 0 0 6 2 3 At-risk At-risk At-risk At-risk
S19 0 3 5 5 6 At-risk At-risk At-risk At-risk
S20 0 0 5 2 2 At-risk At-risk At-risk At-risk
S21 0 0 3 0 3 At-risk At-risk At-risk At-risk
S22 0 0 2 0 2 At-risk At-risk At-risk At-risk
S23 3 2 6 5 6 At-risk At-risk At-risk At-risk
S24 3 3 6 3 6 At-risk Safe Safe Safe
S25 2 2 6 5 3 At-risk Safe Safe Safe
S26 0 0 6 3 6 At-risk Safe Safe Safe
S27 5 0 6 5 6 At-risk Safe At-risk Safe
S28 3 2 6 5 6 At-risk Safe At-risk Safe
S29 0 5 6 0 2 At-risk Safe At-risk Safe
S30 5 0 6 5 6 At-risk Safe At-risk Safe
S31 2 2 6 3 6 At-risk Safe Safe At-risk
S32 3 2 6 2 6 At-risk Safe At-risk At-risk
S33 0 2 5 0 6 At-risk Safe At-risk At-risk
S34 5 2 6 5 6 At-risk Safe At-risk At-risk
S35 2 2 5 5 6 At-risk Safe At-risk At-risk
764 M. SAQR ET AL.
Benito et al. 2015). They are also independent of the
extremes of clicking behavior, and in line with the course
design and previous findings of a pilot study in a similar
course (Alghasham et al. 2013; Lockyer et al. 2013).
Using variables identified in the first step of the study,
we were able to predict the final grade by means of auto-
matic linear regression model with 63.5% accuracy, and
identify 53.9% of at-risk students at the end of the course.
The prediction improved when a binary logistic model was
used, which was able to accurately classify 80.8% of the at-
risk students. Using data recorded up till the mid-course,
prediction accuracy was 42.3%; in the three predictive mod-
els the specificity was above 96%. The most important pre-
dictors were factors reflecting engagement of the students
and the consistency of using the online resources rather
than the number of clicks.
In this study, we opted for using only tracking variables
collected from studentsuse of the LMS and the calculated
engagement parameters. While adding other variables to
the predictive model (variables such as previous perform-
ance or sociodemographic data) might have added to
the power of prediction (Agudo-Peregrina et al. 2014;
Tempelaar et al. 2015; Rienties et al. 2016), these variables
are definitely not modifiable and cannot be used to design
an intervention for underachieving students (Tempelaar
et al. 2015), and would simply defeat the purpose of ana-
lytics as being actionable (Conde & Hern
andez-Garc
ıa2015;
Ga
sevi
c et al. 2016).
Engagement has been shown to be an important factor
for the adoption of learning technology (Cruz-Benito et al.
2015; Ballard & Butler 2016). Aspects of engagement, like
involvement in the learning process, time spent on a task
and compliance have been shown to positively correlate
with effective learning and positive outcome (Shea et al.
2013; Cruz-Benito et al. 2015; Tempelaar et al. 2015; Ballard
& Butler 2016), and there is a large body of research con-
firming that efforts at increasing student engagement
would help students who need support (Cruz-Benito et al.
2015; Tempelaar et al. 2015; Ballard & Butler 2016). Since
engagement has different dimensions or aspects; therefore,
there are different ways to measure engagement (Cruz-
Benito et al. 2015; Ballard & Butler 2016). In this study, we
have explored the potential of LA to measure some of
these aspects such as compliance, involvement, and quality
effort in purposeful activities. The strength of this study is
that it proposes a technique to spot disengaged students
who can be helped. In contrast to teachersintuition, LA is
automatic, effortless, samples a large number of indicators,
and offer a quantifiable risk index (Macfadyen & Dawson
2010; Brown 2011; Siemens 2013; Ellaway et al. 2014;
Ga
sevi et al. 2015; Tempelaar et al. 2015).
Comparing students to their peers in the same course
has proven to be the most effective way to build an accur-
ate predictive model (Pistilli & Arnold 2010; Wolff et al.
2013,2014;Ga
sevi
c et al. 2016) and most predictive models
were applied to individual courses (Macfadyen & Dawson
2010; Pistilli & Arnold 2010; Alghasham et al. 2013;
Tempelaar et al. 2015; Howard et al. 2016), mainly, because
each course is structurally different, uses distinct LMS fea-
ture and incur a different load on learners (Wolff et al.
2013;Ga
sevi
c et al. 2016). Corroborating evidence came
from studies investigating the use of a single predictive
model across different courses, which found significant
variations throughout different courses (Finnegan et al.
2008;Ga
sevi
c et al. 2016). Finnegan et al. (2008) could not
find a single predictor that was shared among all investi-
gated courses and Ga
sevi
c et al. (2016) found that the
same predictors vary, even within the same discipline and
advised against using same predictive model (one size fits
all) for multipe courses.
This study is a step in a long road that will define the
field of LA, we hope that our work will open the door for
others to explore the possible potential of LA, build on and
critique our approach. We recommend medical schools to
adopt analytics capable LMSs, train staff to follow students
through LA dashboards, discover relevant metrics, and pre-
diction models tailored to their educational context.
This study is not without limitations, being exploratory
in nature like most LA studies is our most important limita-
tion, although we have tried to link our findings to theories
of engagement, there is still a long road ahead to replicate
these findings and build generalizable approaches. Another
limitation comes from the methodology, we used different
and diverse tools to collect data from various sources, since
Moodle LMS does not have a built-in analytics dashboard
that automates the data collection, it might need special
skills to replicate these results. However, there are emerg-
ing tools and dashboards that have started to address the
problem and offer these capabilities to everyone.
Conclusions
This research study was set out to identify quantitative
markers that correlate with studentsperformance and can
be used to identify potential risk of a student failing or
dropping a course. We collected data from studentsuse of
the LMS about access, hits, time, forums, communications,
social network parameters, as well as formative assessments
across different points in time. We also calculated engage-
ment indicators that would reflect self-motivation and con-
sistency of interest in using the LMS resources.
The parameters of engagement showed significant posi-
tive correlations with studentsperformance, especially the
parameters that reflected motivation and self-regulation
such as trying formative assessments, frequency of logging,
and creation of new content. We were able to predict the
final grade with 63.5% accuracy, and identify 53.9% of at-
risk students at the end of the course. Using a binary logis-
tic model improved prediction to 80.8%. Using data
recorded up till the mid-course, prediction accuracy was
42.3%, and the most important predictors were factors
reflecting engagement of the students and the consistency
of using the online resources.
This study demonstrated that a significant number of at-
risk medical students can be early identified, and may
benefit from positive intervention that addresses modifiable
factors such as engagement. The study findings represent
an additional method to foresee studentsperformance
before formal final examinations and might be a tool for
early alert to underachievers.
Disclosure statement
The authors report no conflicts of interest. The authors alone are
responsible for the content and writing of this article.
MEDICAL TEACHER 765
Notes on contributors
Mohammed Saqr is assistant professor of medicine with interest in
learning enhanced technology, learning analytics, and simulation of
medicine, he is currently the head of e-learning unit, Qassim
University, College of Medicine.
Matti Tedre is a professor at the University of Eastern Finland, a docent
at Stockholm University, Sweden, and the author of Science of
Computing: Shaping a Discipline (Taylor & Francis/CRC Press, 2014).
Uno Fors DDS, PhD, is professor of IT and learning as well as head of
Department of Computer and Systems Sciences at Stockholm
University, Sweden. Fors research focuses on Technology enhanced
learning and especially on virtual cases for learning within the health-
care area. Fors has published more than 150 papers in scientific jour-
nals and conferences.
Glossary
Learning analytics: Measurement, collection, analysis, and
reporting of data about learners and their contexts, for pur-
poses of understanding and optimizing learning and the envi-
ronments in which it occurs.
Siemens G. Learning analytics: The Emergence of a Discipline.
American Behavioral Scientist. 2013;57:13801400.
Social network analysis: Social network analysis is the study
of structure, it deals with the relational structure and patterns
of relationships among social entities, which might be people,
groups, or organizations.
Hawe P, Webster C, Shiell A. A glossary of terms for navigating
the field of social network analysis. J Epidemiol Community
Health [Internet]. 2004 Dec [cited 2017 Feb 1];58(12):9715.
Available from: http://www.ncbi.nlm.nih.gov/pubmed/15547054
Outlier: An outlier is an observation which deviates so much
from the other observations and might exert an exaggerated
effect on the results (deviate more than 3 times the SD from
the mean).
Hawkins D. 1980. Identification of Outliers: Chapman and Hall.
Betweenness centrality: Is a measure of the influence a node
has over the spread of information through the network.
Hawe P, Webster C, Shiell A. A glossary of terms for navigating
the field of social network analysis. J Epidemiol Community
Health [Internet]. 2004 Dec [cited 2017 Feb 1];58(12):9715.
Available from: http://www.ncbi.nlm.nih.gov/pubmed/15547054
References
About Open TrackingjMailChimp.com: KB Article. Available from: http://
kb.mailchimp.com/reports/about-open-tracking
Agudo-Peregrina AF, Iglesias-Pradas S, Conde-Gonzalez MA,
Hernandez-Garcia A. 2014. Can we predict success from log data in
VLEs? Classification of interactions for learning analytics and their
relation with performance in VLE-supported F2F and online learn-
ing. Comput Human Behav. 31:542550.
Alghasham A, Saqr M, Kamal H. 2013. Using learning analytics to evalu-
ate the efficacy of blended learning in pbl based medical course.
AMEE 2013 Conference at Prague Congress Centre; 2013 Aug
2428; Prague, Czech Republic.
Arnold KE, Pistilli MD. 2012. Course signals at purdue. Proceedings of
the 2nd International Conference on Learning Analytics and
Knowledge LAK; 2012 29 April2 May; Vancouver, British
Columbia, Canada.
Ballard J, Butler PI. 2016. Learner enhanced technology. J Appl Rese
HE. 8:1843.
Bewick V, Cheek L, Ball J. 2005. Statistics review 14: logistic regression.
Crit Care. 9:112118.
Brown M. 2011. Learning analytics: the coming third wave. EDUCAUSE
Learning Initiative Brief . [cited 2016 Jan 24]. Available from: https://
net.educause.edu/ir/library/pdf/elib1101.pdf
Conde M
A, Hern
andez-Garc
ıa
A. 2015. Learning analytics for educa-
tional decision making. Comput Human Behav. 47:13.
Cruz-Benito J, Ther
on R, Garc
ıa-Pe~
nalvo FJ, Pizarro Lucas E. 2015.
Discovering usage behaviors and engagement in an Educational
Virtual World. Comput Human Behav. 47:1825.
Dahlstrom E, Brooks DC, Bichsel J. 2014. The current ecosystem of
learning management systems in higher education: student, faculty,
and IT perspectives. 27.
Dean J. 2014. Big data, data mining, and machine learning: value cre-
ation for business leaders and practitioners. Hoboken (NJ): John
Wiley & Sons; p. 5570.
Doherty I, Sharma N, Harbutt D. 2014. Contemporary and future
eLearning trends in medical education. Med Teach. 37:13.
Ellaway R, Masters K. 2008. AMEE Guide 32: e-Learning in medical edu-
cation Part 1: learning, teaching and assessment. Med Teach.
30:455473.
Ellaway RH, Pusic MV, Galbraith RM, Cameron T. 2014. Developing the
role of big data and analytics in health professional education. Med
Teach. 36:216222.
Falakmasir MH, Habibi J. 2010. Using educational data mining methods
to study the impact of virtual classroom in e-learning. The 3rd
International Conference on Educational Data Mining; Pittsburgh,
PA, USA.
Ferguson R. 2012. Learning analytics: drivers, developments and chal-
lenges. IJTEL. 4:304.
Filippou J, Cheong C, Cheong F. 2015. Designing persuasive systems to
influence learning: modelling the impact of study habits on aca-
demic performance. Pacific Asia Conference on Information Systems
(PACIS); Singapore.
Finnegan C, Morris LV, Lee K. 2008. Differences by course discipline on
student behavior, persistence, and achievement in online courses of
undergraduate general education. J Coll Stud Retent Res Theory
Prac. 10:3954.
Gandomi A, Haider M. 2014. Beyond the hype: big data concepts,
methods, and analytics. Int J Inform Manage. 35:137144.
Ga
sevi D, Dawson S, Siemens G. 2015. Lets not forget: learning ana-
lytics are about learning course signals: lessons learned. Tech
Trends 59:6471.
Ga
sevi
c D, Dawson S, Rogers T, Gasevic D. 2016. Learning analytics
should not promote one size fits all: the effects of instructional con-
ditions in predicting academic success. Internet High Educ.
28:6884.
G
onen M. 2006. Receiver operating characteristic (ROC) curves. SAS
Users Group International (SUGI). 31:210231.
Howard E, Meehan M, Parnell A. 2016. Developing accurate early warn-
ing systems via data analytics [Journal]. Available from: http://arxiv.
org/abs/1612.05735
Liu Q, Peng W, Zhang F, Hu R, Li Y, Yan W. 2016. The effectiveness of
blended learning in health professions: systematic review and meta-
analysis. J Med Internet Res. 18:e2.
Lockyer L, Heathcote E, Dawson S. 2013. Informing pedagogical action:
aligning learning analytics with learning design. Am Behav Scient.
57:14391459.
Macfadyen LP, Dawson S. 2010. Mining LMS data to develop an early
warning systemfor educators: a proof of concept. Comput Educ.
54:588599.
Macfadyen LP, Dawson S. 2012. Numbers are not enough. Why e-learn-
ing analytics failed to inform an institutional strategic plan. Educat
Technol Soc. 15:149163.
Moodle plugins directory: Analytics Graphs [Internet]. [cited 2014 Feb 2].
Available from: https://moodle.org/plugins/block_analytics_graphs
Moodle plugins directory: Attendance Register [Internet]. [cited
2014 Feb 2]. Available from: https://moodle.org/plugins/mod_
attendanceregister
Moodle plugins directory: Configurable Reports [Internet]. [cited 2014
Feb 2]. Available from: https://moodle.org/plugins/block_configur-
able_reports
Norcini JJ. 2003. Setting standards on educational tests. Med Educ.
37:464469.
ONeill LD, Wallstedt B, Eika B, Hartvigsen J. 2011. Factors associated
with dropout in medical education: a literature review. Med Educ.
45:440454.
766 M. SAQR ET AL.
Panzarasa P, Kujawski B, Hammond EJ, Roberts CM. 2016. Temporal
patterns and dynamics of e-learning usage in medical education.
Educ Tech Res Dev. 64:1335.
Papamitsiou ZK, Economides AA. 2014. Learning analytics and educa-
tional data mining in practice: a systematic literature review of
empirical evidence. Educat Technol Soc. 17:4964.
Peng C-yJ, Lee KL, Ingersoll GM. 2002. An introduction to logistic
regression analysis and reporting. J Educat Res. 2:314.
Pistilli MD, Arnold KE. 2010. In practice: purdue signals: mining real-
time academic data to enhance student success. About Campus.
15:2224.
Qassim College of Medicine. 2014. Qassim College of Medicine Privacy
Policy and User Agreement. 3rd ed. [Internet]. [cited 2014 June 2].
Available from: http://qumed.org/code.htm
Ramos C, Yudko E. 2008. Hits(not Discussion Posts) predict student
success in online courses: a double cross-validation study. Comput
Educ. 50:11741182.
Richards G. 2011. Measuring engagement: learning analytics in online
learning. Electr Kazan 2011; 2011 Apr 19; Kazan, Russian Federation.
Rienties B, Boroowa A, Cross S, Kubiak C, Mayles K, Murphy S. 2016.
Analytics4Action evaluation framework: a review of evidence-based
learning analytics interventions at the Open University UK. J Interact
Media Educ. 2016:11.
Shea P, Hayes S, Smith SU, Vickers J, Bidjerano T, Gozza-Cohen M, Jian
S-B, Pickett A, Wilde J, Tseng C-H. 2013. Online learner self-regula-
tion: learning presence viewed through quantitative content-and
social network analysis. IRRODL. 14:427461.
Siemens G. 2013. Learning analytics: the emergence of a discipline. Am
Behav Scientist. 57:13801400.
Smith MA, Shneiderman B, Milic-Frayling N, Mendes Rodrigues E,
Barash V, Dunne C, Capone T, Perer A, Gleave E. Analyzing
(social media) networks with NodeXL. Proceedings of the fourth
international conference on Communities and technologies; 2009:
ACM.
Stegers-Jager KM, Themmen APN, Cohen-Schotanus J, Steyerberg
EW. 2015. Predicting performance: relative importance of
studentsbackground and past performance. Med Educ.
49:933945.
Tekian A, Norcini J. 2015. Overcome the 60% passing score and
improve the quality of assessment. GMS Zeitschrift f
ur Medizinische
Ausbildung. 32:Doc43.
Tempelaar DT, Rienties B, Giesbers B. 2015. In search for the most
informative data for feedback generation: learning analytics in a
data-rich context. Comput Human Behav. 47:157167.
Vaitsis C, Nilsson G, Zary N. 2014. Big data in medical informatics:
improving education through visual analytics. Stud Health Technol
Inform. 205:11631167.
Velleman PF, Hoaglin DC. 2012. Exploratory data analysis. In: Cooper H,
Camic PM, Long DL, Panter AT, Rindskopf D, Sher KJ, editors. APA
handbook of research methods in psychology, Vol. 3: data analysis
and research publication. Washington (DC): American Psychological
Association. p. 5170.
Wise AF, Shaffer DW. 2015. Why theory matters more than ever in the
age of big data. JLA. 2:513.
Wolff A, Zdrahal Z, Herrmannova D, Kuzilek J, Hlosta M. 2014.
Developing predictive models for early detection of at-risk students
on distance learning modules; 2014 Mar 2428; The 4th
International Conference on Learning Analytics and Knowledge
(LAK14); Indianapolis, Indiana, USA.
Wolff A, Zdrahal Z, Nikolov A, Pantucek M. 2013. Improving retention:
predicting at-risk students by analysing clicking behaviour in a vir-
tual learning environment. Proceedings of the Third International
Conference on Learning Analytics and Knowledge; Leuven, Belgium.
New York: ACM Press.
Yang H. 2013. The case for being automatic: introducing the automatic
linear modeling (LINEAR) procedure in SPSS statistics. Multiple
Linear Regression Viewpoints. 39:2737.
MEDICAL TEACHER 767
... In this step, we analysed the articles based on the quality of the dataset, in terms of the features used, the sample size, population, and collection methods. As can be seen in Figure 4, majority of researchers have used available data sources and records found in public dataset or through university/school repositories, precisely a total of seventy-three (73) studies indicate this ( [10], [13], [56], [58]- [66], [20], [69], [71]- [78], [81], [27], [82], [83], [85], [87]- [93], [45], [94]- [97], [100]- [104], [106], [49], [107]- [110], [113]- [117], [119], [50], [121], [123]- [126], [128], [129], [131], [132], [134], [51], [135], [136], [138], [54], [55]). While these sources are easily accessible and provide expected accuracy of the prediction models, some researchers opted for a mixed-mode of data collection where both available sources and questionnaires data are utilized, precisely seven (7) studies indicated this ( [49], [52], [79], [80], [84], [86], [120]). ...
... As previously mentioned in section B, figure 4, most researchers often opt for extracting records available from the university. In another way, records that demonstrate students' online behaviour can be extracted using different Moodle Plugins from the directory (such as Attendance Register, Configurable Reports, and Analytics Graphs) [83]. Additionally, email can be another source for data collection which can be done using tracking tools such as Mailchimp e-mail tracking platforms [83]. ...
... In another way, records that demonstrate students' online behaviour can be extracted using different Moodle Plugins from the directory (such as Attendance Register, Configurable Reports, and Analytics Graphs) [83]. Additionally, email can be another source for data collection which can be done using tracking tools such as Mailchimp e-mail tracking platforms [83]. Furthermore, researchers further extended the data collection methods through surveys and questionnaires, usually, gathered with the help of survey tools such as LimeSurvey [52], Google Forms [52], [115] and SurveyMonkey [86]. ...
Article
Full-text available
Student retention is an essential measurement metric in education, indicated by retention rates, which are accumulated as students re-enroll from one academic year to the next. High retention rates can be obtained if institutions aim to provide appropriate support and teaching methods among the various practices to prevent students from deferring their studies. To address this pressing challenge faced by educational institutions, the underlying factors and the methodological aspects of building robust predictive models are reviewed and scrutinized. Educational Data Mining (EDM) and Learning Analytics (LA) have been widely adopted for knowledge discovery from educational data sources, improving the teaching practice, and identifying at-risk students. Various predictive techniques are applied in LA, such as Machine Learning (ML), Statistical Analysis, and Deep Learning (DL). To gain an in-depth review of these techniques, academic publications have been reviewed to highlight their potential to resolve Student Retention issues in education. Additionally, the paper presents a taxonomy of ML approaches and a comprehensive review of the success factors and the features that are not indicative of student performance in three different learning environments: Traditional Learning, Blended Learning, and Online Learning. The survey reveals that supervised ML and DL techniques are broadly applied in Student Retention. However, the application of ensemble and unsupervised learning clustering techniques supporting the heterogenous and homogenous groups of students is generally lacking. Moreover, static and traditional features are commonly used in student performance, ignoring vital factors such as educators-related, cognitive, and personal data. Furthermore, the paper highlights open challenges for future research directions.
... Out of 29 LMS usage variables examined, 14 were found significant, and among them threecounts of messages read and posted, counts of Wiki edits, and counts of content creation contributionsstood out by their effect size. Saqr, Fors, and Tedre (2017) examined several indicators of regularity of study, derived from LMS log data in a blended learning setting. Regularity indicators based on LMS logins and weekly formative assessment activities showed the highest correlation with final grades. ...
... We introduced this type of indicator as a mean of capturing students' 'presence' in the online component of a course, as well as to neutralize the effect of the students' tendency to exhibit nonuniform, bursty temporal patterns (i.e. sequences of almost instantaneous events) in interaction with online learning resources (Saqr, Fors, and Tedre 2017). ...
... Similarly, the active days and regularity indicators showed comparable predictive power; such indicators were consistently and positively correlated with student performance in all variables, indicating their reliability in reflecting students' engagement with online activities and consequently, academic success. There is growing evidence that these indicators are at least on par withand sometimes better thanoften used frequency-based indicators (Jovanovic et al. 2019;Jovanović et al. 2021;Saqr, Fors, and Tedre 2017). Our findings replicate previous findings and suggest future replicability of such findings in similar contexts. ...
Article
Full-text available
Predictors of student academic success do not always replicate well across different learning designs, subject areas, or educational institutions. This suggests that characteristics of a particular discipline and learning design have to be carefully considered when creating predictive models in order to scale up learning analytics. This study aimed to examine if and to what extent frequently used predictors of study success are portable across a homogenous set of courses. The research was conducted in an integrated blended problem-based curriculum with trace data (n = 2,385 students) from 50 different course offerings across four academic years. We applied the statistical method of single paper meta-analysis to combine correlations of several indicators with students' success. Total activity and the forum indicators exhibited the highest prediction intervals, where the former represented proxies of the overall engagement with online tasks, and the latter with online collaborative learning activities. Indicators of lecture reading (frequency of lecture view) showed statistically insignificant prediction intervals and, therefore, are less likely to be portable across course offerings. The findings show moderate amounts of variability both within iterations of the same course and across courses. The results suggest that the use of the meta-analytic statistical method for the examination of study success indicators across courses with similar learning design and subject area can offer valuable quantitative means for the identification of predictors that reasonably well replicate and consequently can be reliably portable in the future.
... According to a recent survey [23], EDM studies have mainly incorporated issues based on classification [24] or clustering [25] in the setting of online e-portals and platforms. Predictive analytics [26] has also been performed, and the main focus was not on the hypothesis, but rather on the data characteristics, such as predicting the outcome of an assessment performed in an online setting [27], proposing and developing a platform capable of predicting the outcome of a test in a timely and efficient manner. Similarly, in another study [28], an online system was developed for distance-based learning, and the objective was to predict the performance for a particular test or exam. ...
Article
Full-text available
Recent technological advancements in e-learning platforms have made it easy to store and manage students’ related data, such as personal details, initial grade, intermediate grades, final grades, and many other parameters. These data can be efficiently processed and analyzed by intelligent techniques and algorithms to generate useful insights into the students’ performance, such as to identify the factors impacting the progress of successful students or the performance of the students who are struggling in their courses and are at risk of failing. Such a framework is scarce in the current literature. This study proposes an interpretable framework to generate useful insights from the data produced by e-learning platforms using machine learning algorithms. The proposed framework incorporates predictive models, as well as regression and classification models to analyze multiple factors of student performance. Classification models are used to systematize normal and at-risk students based on their academic performance, with high precision and accuracy. Regression analysis is performed to determine the inherent linear and nonlinear relationships between the academic outcomes of the students acting as the target or independent variables and the performance indicative features acting as dependent variables. For further analysis, a predictive modeling problem is considered, where the performance of the students is anticipated based on their commitment to a specific course, their performance for the whole course, and their final grades. The efficiency of the proposed framework is also optimized by reliably tuning the algorithmic parameters. Furthermore, the performance is accelerated by empowering the system with a GPU-based infrastructure. Results reveal that the proposed interpretable framework is highly accurate and precise and can identify factors that play a vital role in the students’ success or failure.
... The early studies of student performance prediction are mainly based on traditional machine learning methods, such as regression analysis [12,13], decision trees [14,15], Naive Bayes [16], etc. These traditional machine learning algorithms have good interpretability and simple implementation and have achieved good results in the field of student performance prediction. ...
Article
Full-text available
Over the past few years, the growing popularity of online education has enabled there to be a large amount of students’ learning behavior data stored, which brings great opportunities and challenges to the field of educational data mining. Students’ learning performance can be predicted, based on students’ learning behavior data, so as to identify at-risk students who need timely help to complete their studies and improve students’ learning performance and online teaching quality. In order to make full use of these learning behavior data, a new prediction method was designed based on existing research. This method constructs a hybrid deep learning model, which can simultaneously obtain the temporal behavior information and the overall behavior information from the learning behavior data, so that it can more accurately predict the high-risk students. When compared with existing deep learning methods, the experimental results show that the proposed method offers better predicting performance.
... We consider these two metrics because students' online engagement with learning management systems and students' study performance, have been reported as the most commonly used source of data to identify students for intervention [32]. Correspondingly, a wide range of LA techniques use these data to identify underengaged and under-performed students, and provide evidence on the eectiveness of intervention towards those students [6,10,27]. ...
Conference Paper
Full-text available
Increased enrolments in higher education, and the shift to online learning that has been escalated by the recent COVID pandemic, have made it challenging for instructors to assist their students with their learning needs. Contributing to the growing literature on instructor-facing systems, this paper reports on the development of a learning analytics (LA) technique called Student InspectionFacilitator (SIF) that provides an explainable interpretation of students learning behaviour to support instructors with the identification of students in need of attention. Unlike many previous predictive systems that automatically label students, our approach provides explainable recommendations to guide data exploration while still reserving judgement about interpreting student learning to instructors. The insights derived from applying SIF in an introductory Information Systems course with 407 enrolled students suggest that SIF can be utilised independent of the context and can provide a meaningful interpretation of students’ learning behaviour towards facilitating proactive support of students.
... While students use these systems, the log records produced have become ever more accessible. (Macfadyen & Dawson, 2010;Kotsiantis et al., 2013;Saqr et al., 2017). Universities now should improve the capacity of using these data to predict academic success and ensure student progress (Bernacki et al., 2020). ...
Article
Full-text available
Educational data mining has become an effective tool for exploring the hidden relationships in educational data and predicting students' academic achievements. This study proposes a new model based on machine learning algorithms to predict the final exam grades of undergraduate students, taking their midterm exam grades as the source data. The performances of the random forests, nearest neighbour, support vector machines, logistic regression, Naïve Bayes, and k-nearest neighbour algorithms, which are among the machine learning algorithms, were calculated and compared to predict the final exam grades of the students. The dataset consisted of the academic achievement grades of 1854 students who took the Turkish Language-I course in a state University in Turkey during the fall semester of 2019–2020. The results show that the proposed model achieved a classification accuracy of 70–75%. The predictions were made using only three types of parameters; midterm exam grades, Department data and Faculty data. Such data-driven studies are very important in terms of establishing a learning analysis framework in higher education and contributing to the decision-making processes. Finally, this study presents a contribution to the early prediction of students at high risk of failure and determines the most effective machine learning methods.
... Critics suggest focusing on quality, not the quantity of online learning behavior (You, 2016): As active participation is crucial for success, indicators that do not distinguish between active and passive engagement are problematic (Ransdell & Gaillard-Kenney, 2009). Overall, these contradictory assumptions on the usefulness of broad log data indicators go along with inconsistent findings on the association between those indicators of general online activity and learning outcomes: some studies reported no association (Broadbent, 2016), negative correlations (e.g., Ransdell & Gaillard-Kenney, 2009;Strang, 2016), or positive correlations (e.g., Liu & Feng, 2011;McCuaig & Baldwin, 2012;Saqr et al., 2017). Other studies that examined various online courses simultaneously obtained mixed results across different courses (e.g., Conijn et al., 2017;Gašević et al., 2016), indicating that online courses might be too heterogeneous to draw a general conclusion about the link between general online activity and learning outcomes. ...
Article
Full-text available
Analyzing log data from digital learning environments provides information about online learning. However, it remains unclear how this information can be transferred to psychologically meaningful variables or how it is linked to learning outcomes. The present study summarizes findings on correlations between general online activity and learning outcomes in university settings. The course format, instructions to engage in online discussions, requirements, operationalization of general online activity, and publication year are considered moderators. A multi-source search provided 41 studies ( N = 28,986) reporting 69 independent samples and 104 effect sizes. The three-level random-effects meta-analysis identified a pooled effect of r = .25 p = .003, 95% CI [.09, .41], indicating that students who are more active online have better grades. Despite high heterogeneity, Q(103) = 3,960.04, p < .001, moderator analyses showed no statistically significant effect. We discuss further potential influencing factors in online courses and highlight the potential of learning analytics.
Article
Due to the COVID-19 pandemic during spring semester 2020, teachers and students were forced to engage in online instruction. However, there is little evidence on the feasibility of online physiology teaching. This study demonstrated a 3-week preliminary online physiology course based on Rain Classroom assisted by the mobile application WeChat. Eighty-seven nursing undergraduate students attended an online physiology course during the spring semester of the 2019-2020 academic year from March 9 to March 29. We determined the effects of the online physiology learning based on in-class tests, pre-class preparation and review rates for the course materials. We also measured the students' perceptions and attitudes about online learning with a questionnaire survey. Post-test scores from the first week to the third week in online physiology course (7.22 ± 1.83, 7.68 ± 2.09, and 6.21 ± 2.92, respectively) exceeded the pre-test scores (5.32 ± 2.14, 6.26 ± 2.49, and 3.72 ± 2.22, respectively), and this finding was statistically significant (all P < 0.001). Moreover, the pre-test scores were significant positive predictors of final grade (all P < 0.01). In addition, the percentage of pre-class preparation increased in three weeks, from 43.68% to 57.47% to 68.97%. From the first week to the third week, the review rate increased from 86.21% to 91.95%, however the second week was the lowest of all (72.41%). Finally, students' perceptions about their online physiology learning experiences were favorable. In conclusion, online physiology instruction based on Rain Classroom assisted by WeChat was an effective strategy during the COVID-19 pandemic.
Article
Having records of the massive use of digital tools allows observing additional aspects of the learning process, from the Learning Analytics approach. This study, as a controlled case, analyzes the uses and describes the types of students, according to access to digital resources and temporality, which differ according to the course, gender or presence. It is carried out in cetheris paribus conditions, for different courses, first and fourth, with the same teacher, subject, teaching methodology and period. The results show that first-year students have a more temporally polarized behavior and a greater digital gap with respect to the average, while fourth-year students are more committed to the program, with women being more digitally active, pointing to a positive relationship between grades and digital activity, more consolidated in fourth grade. The cluster analysis confirms the differences in the grouping of students in the different courses, conditional on digital use and performance.
Article
Learning outcomes can be predicted with machine learning algorithms that assess students’ online behavior data. However, there have been few generalized predictive models for a large number of blended courses in different disciplines and in different cohorts. In this study, we examined learning outcomes in terms of learning data in all of the blended courses offered at a Chinese university and proposed a new classification method of blended courses, in which students were primarily clustered on the basis of their online learning behaviors in blended courses using the expectation–maximization algorithm. Then, the blended courses were classified on the basis of the cluster of students who were present in the course and had the highest proportion. The advantage of this method is that the criteria used for classification of the blended courses are clearly defined on the basis of students' online behavior data, so it can easily be used by machine learning systems to algorithmically classify blended courses based on log data collected from a learning management system. Drawing on the classification of the blended courses, we also proposed and validated a general model using the random forest algorithm to predict learning outcomes based on students’ online behaviors in blended courses with different disciplines and different cohorts. The findings of this study indicated that after blended courses were classified on the basis of students’ online behavior, prediction accuracy in each category increased. The overall accuracies for Course I (380 courses out of 661 after screening), L (14 courses out of 661 after screening), A (237 courses out of 661 after screening), V (8 courses out of 661 after screening), and H (22 courses out of 661 after screening) were 38.2%, 48.4%, 42.3%, 42.4%, and 74.7%, respectively. According to these results, it was found that a prerequisite for the accurate prediction of students' learning outcomes in a blended course was that most students should be highly engaged in a variety of online learning activities rather than being focused on only one type of activity, such as only watching online videos or submitting online assignments. The prediction model achieved accuracies of 80.6%, 85.3%, 63%, 54.8%, and 14.3% for grades A, B, C, D, and F in Course H, respectively. The results demonstrated the potential of the proposed model for accurately predicting learning outcomes in blended courses. Finally, we found that there was no single online learning behavior that had a dominant effect on the prediction of students' final grades.
Article
Full-text available
Background: Blended learning, defined as the combination of traditional face-to-face learning and asynchronous or synchronous e-learning, has grown rapidly and is now widely used in education. Concerns about the effectiveness of blended learning have led to an increasing number of studies on this topic. However, there has yet to be a quantitative synthesis evaluating the effectiveness of blended learning on knowledge acquisition in health professions. Objective: We aimed to assess the effectiveness of blended learning for health professional learners compared with no intervention and with nonblended learning. We also aimed to explore factors that could explain differences in learning effects across study designs, participants, country socioeconomic status, intervention durations, randomization, and quality score for each of these questions. Methods: We conducted a search of citations in Medline, CINAHL, Science Direct, Ovid Embase, Web of Science, CENTRAL, and ERIC through September 2014. Studies in any language that compared blended learning with no intervention or nonblended learning among health professional learners and assessed knowledge acquisition were included. Two reviewers independently evaluated study quality and abstracted information including characteristics of learners and intervention (study design, exercises, interactivity, peer discussion, and outcome assessment). Results: We identified 56 eligible articles. Heterogeneity across studies was large (I(2) ≥93.3) in all analyses. For studies comparing knowledge gained from blended learning versus no intervention, the pooled effect size was 1.40 (95% CI 1.04-1.77; P<.001; n=20 interventions) with no significant publication bias, and exclusion of any single study did not change the overall result. For studies comparing blended learning with nonblended learning (pure e-learning or pure traditional face-to-face learning), the pooled effect size was 0.81 (95% CI 0.57-1.05; P<.001; n=56 interventions), and exclusion of any single study did not change the overall result. Although significant publication bias was found, the trim and fill method showed that the effect size changed to 0.26 (95% CI -0.01 to 0.54) after adjustment. In the subgroup analyses, pre-posttest study design, presence of exercises, and objective outcome assessment yielded larger effect sizes. Conclusions: Blended learning appears to have a consistent positive effect in comparison with no intervention, and to be more effective than or at least as effective as nonblended instruction for knowledge acquisition in health professions. Due to the large heterogeneity, the conclusion should be treated with caution.
Article
Full-text available
Not all students who fail or drop out would have done so if they had been offered help at the right time. This is particularly true on distance learning modules where there is no direct tutor/student contact, but where it has been shown that making contact at the right time can improve a student's chances. This paper explores the latest work conducted at the Open University, one of Europe's largest distance learning institutions, to identify when is the optimum time to make student interventions and to develop models to identify the at-risk students in this time frame. This work in progress is taking real-time data and feeding it back to module teams as the module is running. Module teams will be indicating which of the predicted at-risk students have received an intervention, and the nature of the intervention.
Article
Full-text available
It is an exhilarating and important time for conducting research on learning, with unprecedented quantities of data available. There is a danger, however, in thinking that with enough data, the numbers speak for themselves. In fact, with larger amounts of data, theory plays an ever-more critical role in analysis. In this introduction to the special section on learning analytics and learning theory, we describe some critical problems in the analysis of large-scale data that occur when theory is not involved. These questions revolve around what variables a researcher should attend to and how to interpret a multitude of micro-results and make them actionable. We conclude our comments with a discussion of how the collection of empirical papers included in the special section, and the commentaries that were invited on them, speak to these challenges, and in doing so represent important steps towards theory-informed and theory-contributing learning analytics work. Our ultimate goal is to provoke a critical dialogue in the field about the ways in which learning analytics research draws on and contributes to theory.
Article
Full-text available
This study examined the extent to which instructional conditions influence the prediction of academic success in nine undergraduate courses offered in a blended learning model (n = 4134). The study illustrates the differences in predictive power and significant predictors between course-specific models and generalized predictive models. The results suggest that it is imperative for learning analytics research to account for the diverse ways technology is adopted and applied in course-specific contexts. The differences in technology use, especially those related to whether and how learners use the learning management system, require consideration before the log-data can be merged to create a generalized model for predicting academic success. A lack of attention to instructional conditions can lead to an over or under estimation of the effects of LMS features on students' academic success. These findings have broader implications for institutions seeking generalized and portable models for identifying students at risk of academic failure.
Article
Full-text available
Despite the increasing popularity of e-learning systems across a variety of educational programmes, there is relatively little understanding of how students and trainees distribute their learning efforts over time. This study aimed to analyse the usage patterns of an e-learning resource designed to support specialty training. Data were collected from e-learning Anaesthesia, a web-based training programme offered by the Royal College of Anaesthetists in partnership with e-Learning for Healthcare. We constructed the time series of 45,020 records of knowledge and self-assessment sessions accessed by 2491 trainees between August 2008 and February 2010. Analysis of the time series suggested that e-learning usage was characterised by concentrations of rapidly occurring sessions within short time frames of intense activity, separated by disproportionally long periods of reduced activity. Non-uniform temporal fluctuations of usage were pronounced especially for self-assessment sessions, the timing of which was highly correlated with examination dates. While on average trainees’ involvement in knowledge sessions was larger than in self-assessment sessions, for both sessions average hourly activity and length remained stable between 9:00 am and 10:00 pm during weekdays. Average daily activity decayed as the weekend approached, but average session length did not vary significantly across the week. Combined with previous research on distributed practice, learning time distribution and test-enhanced learning, our study has implications for the improvement of long-term retention through the redistribution of knowledge sessions uniformly over time and the sustainment of frequent information retrieval and repeated testing. Findings on hourly and daily periodicities also suggest how new learning materials could be broken down into suitable components that fit learners’ time constraints. © 2015 Association for Educational Communications and Technology
Chapter
Predictive modeling is the process of using historical data to anticipate what will likely happen in the future. There are best practices and guidelines that can help make this a productive and repeatable process regardless of industry. The pattern is to sample (optionally), explore, modify, model, and finally assess (sEMMA). In predictive models there are three distinct types related to the type of prediction variable: binary classification, multilevel classification, and interval prediction. The final step in building predictive models is the assessment. Assessment objectively measures the quality of a model compared to its challengers. There are a number of different assessment criteria to determine the “best” model, and the preferred assessment criteria can vary by industry, company, and individual modeler.
Article
In this study we develop an early warning system for a large first year STEM undergraduate course and discuss the data analytics decisions at each stage. Numerous studies have provided evidence in favour of adopting early warning systems as a means of identifying at-risk students in online courses. Many of these early warning systems are in practical use and rely on data from students' engagement with Virtual Learning Environments (VLEs), also referred to as Learning Management Systems. Our study examines how to develop an early warning system, and investigates the optimal time in a course to apply such a system. We present findings from a statistics university course which has a large proportion of resources on the VLE Blackboard and weekly continuous assessment. We identify weeks 5-6 of the course as the critical time in the semester to implement an early warning system. Through detailed (fine-grained) variables, mixed mode clustering and our final prediction method of BART (Bayesian Additive Regressive Trees) we are able to predict students' final grade by week 6 based on mean absolute error (MAE) to within 2.5 percentage points. Subsequently, we argue for the inclusion of cluster membership as explanatory variables.
Article
Purpose – The purpose of this paper is to propose a conceptual model of engagement, appropriated from social media marketing, as a sense-making framework to understand engagement as a measurable process through the development of engagement profiles. To explore its potential application to education the paper follows previous work with Personalised Learning strategies to place emphasis on the promotion of the learner voice – their ability to influence decisions affecting them and their community. Design/methodology/approach – This paper will position engagement as a sociocultural process and adopt an Activity Theory based methodology demonstrated through a desk analysis of VLE data from a further education college. Findings – The analysis suggests that the approach can yield insights that may be elusive in traditional measures reinforcing the overall conceptual proposal for a multi-method approach to profiling learner engagement. Research limitations/implications – The paper has focused on presentation and exploration of the conceptual approach, which has limited the scope to broaden the discussion of the desk analysis and wider findings that this approach reveals. Practical implications – It is intended that the approach offers a generalizable model that can be adopted by institutions planning to measure engagement or develop learner activity profiles. Several areas of immediate potential are identified throughout the paper. Originality/value – This paper contributes a multi-method approach to engagement as argued for in recent engagement literature. This should offer institutions a way to realise value from emerging ideas within related domains of Learning Design and Learning Analytics.