Conference PaperPDF Available

Early Warning Systems for More Effective Student Counseling in Higher Education - Evidence from a Dutch Field Experiment

Authors:
Simone Plak at Vrije Universiteit Amsterdam
Simone Plak
Ilja Cornelisz at Vrije Universiteit Amsterdam
Ilja Cornelisz
Martijn Meeter at Vrije Universiteit Amsterdam
Martijn Meeter
Chris Van Klaveren at Vrije Universiteit Amsterdam
Chris Van Klaveren
Download full-text PDF
Read full-text
Download citation

Abstract

Early Warning Systems (EWS) accommodate student counselors by identifying at-risk students and allow them to intervene timely and prevent student dropout. This paper evaluates an EWS that shares student-specific risk information with student counselors, which was implemented at a large Dutch university. A randomized field experiment was conducted to estimate the effect of EWS-assisted counseling on first-year student dropout and academic performance. The results indicate that the EWS accurately predicts at-risk students. Yet, EWS-assisted counseling did not reduce dropout, nor improved academic performance. Solving the underlying problem of poor academic performance might require additional actionable feedback and recommended counseling practices.
Content uploaded by Simone Plak
Author content
All content in this area was uploaded by Simone Plak on Dec 05, 2019
Content may be subject to copyright.
Early Warning Systems for More Effective Student Counseling in
Higher Education – Evidence from a Dutch Field Experiment
Simone Plak, Ilja Cornelisz, Martijn Meeter, and Chris van Klaveren
Amsterdam Center for Learning Analytics, Department of Educational Sciences, Vrije Universiteit
Amsterdam, The Netherlands
Background
The absolute number of dropouts in higher education increased considerably in recent years (see
for example the increase in entry and completion rates from 2001 to 2012 in: OECD 2003 & 2014).
This has led to increased scrutiny of dropout in higher education and movements to hold universities
accountable for graduation rates (Bettinger & Baker, 2014).
One way to reduce dropout is to improve the effectiveness of student counseling, since, for
instance, proactive individual student coaching was found to improve student retention and graduation
rates (Bettinger & Baker, 2014). However, student counselors often have caseloads that can span
many hundreds of students (Hughes & Scott-Clayton, 2011), making individualized student counseling
difficult, if not impossible.
A promising low-cost intervention that can unburden student counselors by early detection of
students at risk of dropping out is an early warning system (EWS). Essentially, an EWS uses machine
learning techniques to optimally predict dropout at an early stage and communicates these predictions
through reporting tools, like dashboards. Currently, information student counselors can act upon (e.g.
student grades) is frequently accessed at a stage in the academic year where it might be too late to
rectify any problems. Providing student counselors with an EWS that flags at-risk students allows
them to timely intervene, which might result in dropout reduction.
Several dashboards have been developed over the past years to support learning or teaching
(Verbert, Duval, Klerkx, Govaerts, & Santos, 2013). However, not many of them are evaluated,
especially not in terms of student dropout. One exception is the EWS Course Signals, which improved
retention rates when it was used in at least one course (Arnold & Pistilli, 2012).
Research Objective
This study evaluated whether the implementation of an EWS at a university to assist student
counselors reduced dropout or increased obtained course credits. For this purpose, a randomized field
experiment was conducted.
Setting
The experiment was conducted in 2016 at the Vrije Universiteit Amsterdam in the Netherlands,
which is one of the two large publicly-funded universities in Amsterdam. During the academic year,
student counselors could proactively invite low-performing students for individual meetings based on
the information displayed by the student analytics monitor (the EWS). At the end of the academic
year, students were asked to fill in a follow-up questionnaire online and counselors were interviewed.
Simone Plak is the corresponding author (s.plak@vu.nl).
Participants
In this study, 758 first-year students (57% female) enrolled in 12 study programs participated, of
whom 125 (16.5%) filled in the follow-up questionnaire. In total, 34 student counselors participated.
Intervention
The intervention group received EWS-assisted counseling throughout the whole academic year,
while the control group received counseling as usual. A student analytics monitor was built to inform
counselors on their students’ probability of dropping out. Counselors received a monitor of the
intervention group students every week and were free to use this monitor in their counseling as
they saw fit. The monitor showed for each student their probability of dropping out and additional
information on their motivation and performance (see Figure 1). The counselors received a training
on the content of the monitor. After each study term the dropout prediction was updated based on
new data (obtained study results).
This probability of dropping out is based on a dropout prediction model developed on data of
the first-year student cohorts of 2013 and 2014. The dropout prediction model was trained using 5-fold
cross validation on 85% of the 2014 cohort data, and tested on the 15% hold out set and on the 2013
cohort data. Several models were tested and the best predictive model, the Generalized Additive
Model, was selected based on its lowest mean absolute error (MAE); the smallest mean difference
between predicted probability to drop out and actual dropout. At each term of the academic year,
the model includes the variables that are most predictive of dropout.
Figure 1. Example of the student analytics monitor.
2
Research Design
To obtain an unbiased estimate of the effect of EWS-assisted counseling, the participating
students were randomly assigned to the intervention or control group as follows. For each study
program, all students were ordered descending on the exact moment of consent. Per study program,
either a 1 or a 0 was randomly sampled with equal probability. If the drawn number was a 1, the first
student was assigned to the intervention group, the second student to the control group, the next to
the intervention group, and so on; if the sampled number was a 0, the first student was assigned to
the control group, the second student to the intervention group, and so on.
Thereafter, all students were individually informed within one week by e-mail about their as-
signment. Counselors were informed on which students were assigned to the intervention group, but
not on students assigned to the control group.
Data Collection and Analysis
We use register data to monitor study progress (e.g. obtained credits, and retention). Addi-
tionally, we use student characteristics and study program controls to get a more precise estimate of
the intervention effect, and we use survey data to examine students’ experiences with counselors and
whether they acted on the counselors advise.
To test the effect of EWS-assisted counseling on both dropout and obtained credits we will
perform OLS regressions and cluster standard errors by participating study programs. We will step
wise add student and study characteristics to the models, to increase precision of the intervention
effect estimate.
Results
Table 1 shows that EWS-assisted counseling did not have an effect on dropout, and Table 2
demonstrates no effect on total obtained credits at the end of the academic year. These findings
suggest that EWS-assisted counseling has no effect on student retention and educational attainment.
The follow-up questionnaire demonstrated no differences between the intervention and control
group. For instance, no differences were found in number of meetings with a counselor. Interviews
showed that student counselors did not pro-actively invite students at risk of dropping out, because
they either had not enough time or mentioned that they expected students not to show up.
(1) (2) (3)
EWS-assisted Counseling 0.041 (0.033) 0.047 (0.035) 0.048 (0.036)
Constant 0.224*** (0.019) 0.322*** (0.044) 0.249*** (0.035)
Student Controls X X
Study Program Control X
R20.002 0.145 0.160
N758 758 758
Table 1
Three OLS regressions - when (1) no controls are included, (2) student characteristics are included, (3)
student and study characteristics are included in the model - to determine the effect of the treatment
(EWS-assisted counseling) on dropout. */**/*** denote significance at a 10/5/1 percent confidence
level (two-sided). Standard errors are given in parentheses.
3
(1) (2) (3)
EWS-assisted Counseling -1.970 (1.401) -2.420* (1.471) -2.431 (1.495)
Constant 44.668*** (1.337) 40.452*** (2.349) 45.383*** (1.553)
Student Controls X X
Study Program Control X
R20.002 0.228 0.274
N758 758 758
Table 2
Three OLS regressions - when (1) no controls are included, (2) student characteristics are included, (3)
student and study characteristics are included in the model - to determine the effect of the treatment
(EWS-assisted counseling) on total obtained credits. */**/*** denote significance at a 10/5/1 percent
confidence level (two-sided). Standard errors are given in parentheses.
Conclusions
The findings suggest that providing student counselors timely with information about students’
risk of dropping out together with students’ background characteristics does not reduce dropout or
increase obtained credits. Early identification of at-risk students is useful as it can support student
counselors, but risk identification does not imply that the underlying problem that causes poor per-
formance is identified. Moreover, the results show that the number of interactions does not become
higher in the presence of an EWS.
References
Arnold, K. E., & Pistilli, M. D. (2012). Course signals at purdue: Using learning analytics to increase student
success. In Proceedings of the 2nd international conference on learning analytics and knowledge (pp.
267–270).
Bettinger, E. P., & Baker, R. B. (2014). The effects of student coaching: An evaluation of a randomized
experiment in student advising. Educational Evaluation and Policy Analysis,36 , 3–19.
Hughes, K. L., & Scott-Clayton, J. (2011). Assessing developmental assessment in community colleges. Com-
munity College Review,39 , 327–351.
OECD. (2003). Education at a glance 2003: OECD indicators. OECD Publishing, Paris. (Retrieved from:
https://www.oecd-ilibrary.org/education/education-at-a-glance-2014_eag-2014-en)
OECD. (2014). Education at a glance 2014: OECD indicators. OECD Publishing, Paris. (Retrieved from:
https://www.oecd-ilibrary.org/education/education-at-a-glance-2014_eag-2014-en)
Verbert, K., Duval, E., Klerkx, J., Govaerts, S., & Santos, J. L. (2013). Learning analytics dashboard applica-
tions. American Behavioral Scientist,57 , 1500–1509.
4
... The most widespread rigorously evaluated interventions can be characterized as remediation, matching activities before enrolling in vocational or higher education, and financial aid. The empirical literature on the effectiveness of these efforts is largely disappointing (Lyche 2010;Plak et al. 2019). Remediation programs demonstrate no or mixed-effects (Boatman and Long 2018;Calcagno et al. 2008;Scott-Clayton and Rodriguez 2012;Valentine et al. 2011) at enormous costs (Scott-Clayton, Crosta, andBelfield 2014), and matching initiatives tend to not affect student dropout or performance (van Klaveren et al. 2019). ...
... The empirical literature on the effectiveness of these efforts is largely disappointing (Lyche 2010;Plak et al. 2019). Remediation programs demonstrate no or mixed-effects (Boatman and Long 2018;Calcagno et al. 2008;Scott-Clayton and Rodriguez 2012;Valentine et al. 2011) at enormous costs (Scott-Clayton, Crosta, andBelfield 2014), and matching initiatives tend to not affect student dropout or performance (van Klaveren et al. 2019). Evaluations of financial aid programs show that financial aid increases degree attainment and improves grades when combined with other support programs (Angrist, Lang, and Oreopoulos 2009;Page and Scott-Clayton 2016). ...
... This study thus proposes a new method that makes the sensitivity/precision trade-off insightful over time and offers a systematic way to actually act on this intertemporally. This study follows Sansone (2019) and Plak et al. (2019) in that machine learning techniques are used to identify students at risk of dropping out, but applies a broader palette of machine learning algorithms, uses these algorithms to target students at risk, and focuses on vocational education students instead of college students. The use of machine learning algorithms is important because the objective is to predict dropout well for later students entering vocational education (i.e. ...
Identifying false positives when targeting students at risk of dropping out
Article
Full-text available
  • Apr 2022
  • Educ Econ
Inefficient targeting of students at risk of dropping out might explain why dropout-reducing efforts often have no or mixed effects. In this study, we present a new method which uses a series of machine learning algorithms to efficiently identify students at risk and makes the sensitivity/precision trade-off inherent in targeting students for dropout prevention explicit. Data of a Dutch vocational education institute is used to show how out-of-sample machine learning predictions can be used to formulate invitation rules in a way that targets students at risk more effectively, thereby facilitating early detection for effective dropout prevention.
View
... Colleges and universities use predictive analytics for various purposes, ranging from identifying students who might default on their loans to targeting alumni who are likely to give generously to the institution (Ekowo & Palmer, 2016). The most common use of predictive analytics, however, is to identify students at risk of failing courses or dropping out (Alamuddin et al., 2019;Milliron et al., 2014;Plak et al., 2019), and to direct various student success strategies (e.g., intrusive advising, additional financial aid) to these students. Numerous contextual factors have motivated institutions to turn toward predictive analytics. ...
... A final question is whether predictive analytics are actually resulting in more effective targeting of and support for at-risk students in higher education. While few studies to date have examined the effects of predictive analytics on college academic performance, persistence, and degree attainment, the three experimental studies of which we are aware find limited evidence of positive effects for at-risk students (Alamuddin et al., 2019;Milliron et al., 2014;Plak et al., 2019). More research is needed to understand the role of predictive analytics in improving institutional performance. ...
Bringing Transparency to Predictive Analytics: A Systematic Comparison of Predictive Modeling Methods in Higher Education
Article
Full-text available
  • Aug 2021
Colleges have increasingly turned to predictive analytics to target at-risk students for additional support. Most of the predictive analytic applications in higher education are proprietary, with private companies offering little transparency about their underlying models. We address this lack of transparency by systematically comparing two important dimensions: (1) different approaches to sample and variable construction and how these affect model accuracy and (2) how the selection of predictive modeling approaches, ranging from methods many institutional researchers would be familiar with to more complex machine learning methods, affects model performance and the stability of predicted scores. The relative ranking of students’ predicted probability of completing college varies substantially across modeling approaches. While we observe substantial gains in performance from models trained on a sample structured to represent the typical enrollment spells of students and with a robust set of predictors, we observe similar performance between the simplest and the most complex models.
View
... Due to the complexity of the topic, there exist several intertwining research streams; some being very domain-specific such as identifying students' performance using log data collected from a programming environment [19,21,63], down to the level of utilizing keystrokes students take while programming [14,32,36], while other research streams may be domain-generic such as using past grades, completed courses or consistency of work to predict future performance [2,31,59]. Yet, even with the ability of identify students in need of assistance, it might be that the interventions themselves do not work [51], or only work for a subset of the students [23]. ...
View
... Existing research in Learning Analytics (LA) and EDM focused on developing highly accurate predictive models that can estimate student learning outcomes related to assignment scores, course grades, and drop-out probability [3,4]. These course-based predictive models provide early warning to student counselors or instructors associated with a specific course [5,6]. ...
Student-centric Model of Login Patterns: A Case Study with Learning Management Systems
Conference Paper
Full-text available
  • Jul 2021
With the increasing adoption of Learning Management Systems (LMS) in colleges and universities, research in exploring the interaction data captured by these systems is promising in developing a better learning environment and improving teaching practice. Most of these research efforts focused on course-level variables to predict student performance in specific courses. However, these research findings for individual courses are limited to develop beneficial pedagogical interventions at the student level because students often have multiple courses simultaneously. This paper argues that student-centric models will provide systematic insights into students' learning behavior to develop effective teaching practice. This study analyzed 1651 undergraduate student's data collected in Fall 2019 from computer science and information systems departments at a US university that actively uses Blackboard as an LMS. The experimental results demonstrated the prediction performance of student-centric models and explained the influence of various predictors related to login volumes, login regularity, login chronotypes, and demographics on predictive models. Our findings show that student prior performance and normalized student login volume across courses significantly impact student performance models. We also observe that regularity in student logins has a significant influence on low performing students and students from minority races. Based on these findings, the implications were discussed to develop potential teaching practices for these students.
View
... Interestingly, this was already the case after the first quarter. This is in line with previous studies focussing on college students, where first grades in the program tend to be much better predictors than any other variable that students bring to college (Montmarquette et al. 2001;Plak et al. 2019). This suggests that the first weeks of college are particularly decisive when it comes to subsequent dropout and suggests programs intended to stop dropout may have to start early after matriculation. ...
Content expectations and dropout in Dutch vocational education
Article
Full-text available
  • Oct 2020
Unrealistic expectations with regard to one’s study program has been linked to negative consequences for future academic success. Previous studies that have shown this are often retrospective, however, and focus on performance-related expectations (e.g. expected grades), while unrealistic expectations about the required effort and the content of the program (content-related expectations) may be more relevant for explaining dropout in tertiary education. This study prospectively investigates whether the content-related expectations of 208 Dutch Sport Academy students elicited before the start of their vocational program are associated with subsequent dropout and academic performance. Our results show that dropped-out students did not differ in expected grades (even though they did differ in prevocational GPA). Moreover, their content-related expectations at the start of the program did not differ from successful students, nor were they any less realistic. Still, when retrospectively asked, 50% of the students answered that the concerning program did not fit. This suggests that retrospective reports of inadequate expectations may not reflect deficient expectations before starting the program. Instead, tertiary educational programs may defy expectations in both successful students and later unsuccessful students, with surprises being pleasant for successful students and unpleasant for unsuccessful ones.
View
... The decision tree is used to partition data recursively into two groups such as to maximize data purity. This recursive data-splitting process can be explained using Fig 2, under the simplifying assumption that the observed input vector X consists of only two background characteristics, x 1 and x 2 (this example is taken from Plak et al. [15]). The red dots in the figure represent {x 1 , x 2 } observations for missing outcomes, while the green dots represent non-missing outcome observations. ...
Addressing missing data in randomized clinical trials: A causal inference perspective
Article
Full-text available
Background The importance of randomization in clinical trials has long been acknowledged for avoiding selection bias. Yet, bias concerns re-emerge with selective attrition. This study takes a causal inference perspective in addressing distinct scenarios of missing outcome data (MCAR, MAR and MNAR). Methods This study adopts a causal inference perspective in providing an overview of empirical strategies to estimate the average treatment effect, improve precision of the estimator, and to test whether the underlying identifying assumptions hold. We propose to use Random Forest Lee Bounds (RFLB) to address selective attrition and to obtain more precise average treatment effect intervals. Results When assuming MCAR or MAR, the often untenable identifying assumptions with respect to causal inference can hardly be verified empirically. Instead, missing outcome data in clinical trials should be considered as potentially non-random unobserved events (i.e. MNAR). Using simulated attrition data, we show how average treatment effect intervals can be tightened considerably using RFLB, by exploiting both continuous and discrete attrition predictor variables. Conclusions Bounding approaches should be used to acknowledge selective attrition in randomized clinical trials in acknowledging the resulting uncertainty with respect to causal inference. As such, Random Forest Lee Bounds estimates are more informative than point estimates obtained assuming MCAR or MAR.
View
... Student performance follow-up [41] -The system is unable to provide the reasons of low performance and dropout reports. ...
View
... Finally, Plak et al. implemented an EWS at Vrije Universiteit Amsterdam in order to support student counselors [37]. This study involved 758 first-year students from 12 different study programs, as well as 34 counselors. ...
Systematic Literature Review of Predictive Analysis Tools in Higher Education
Article
Full-text available
  • Dec 2019
The topic of predictive algorithms is often regarded among the most relevant fields of study within the data analytics discipline. They have applications in multiple contexts, education being an important one of them. Focusing on higher education scenarios, most notably universities, predictive analysis techniques are present in studies that estimate academic outcomes using different kinds of student-related data. Furthermore, predictive algorithms are the basis of tools such as early warning systems (EWS): applications able to foresee future risks, such as the likelihood of students failing or dropping out of a course, and alert of such risks so that corrective measures can be taken. The purpose of this literature review is to provide an overview of the current state of research activity regarding predictive analytics in higher education, highlighting the most relevant instances of predictors and EWS that have been used in practice. The PRISMA guidelines for systematic literature reviews were followed in this study. The document search process yielded 1382 results, out of which 26 applications were selected as relevant examples of predictors and EWS, each of them defined by the contexts where they were applied and the data that they used. However, one common shortcoming is that they are usually applied in limited scenarios, such as a single course, evidencing that building a predictive application able to work well under different teaching and learning methodologies is an arduous task.
View
Machine Learning in Learning Analytics Dashboards: A Systematic Literature Review
Conference Paper
  • Jan 2025
This systematic literature review explores the use of Machine Learning algorithms in Learning Analytics Dashboards, highlighting the shift from traditional approaches to predictive, data-driven methods in higher education. A total of30 relevant papers were identified from major online research databases. The review focuses on three key areas: the goals of Machine Learning implementation in Learning Analytics Dashboards, the types of algorithms used, and the evaluation metrics applied. The supervised approach dominates, with tasks such as Performance Prediction and Identification of At-Risk Students. Common algorithms include Decision Trees, Support Vector Machines, Naive Bayes, and Ensemble Learning techniques, with Accuracy, Precision, and Recall as the main evaluation metrics. Top-performing models were identified, revealing the need for algorithm adaptation to specific educational contexts. Additionally, small sample sizes were a common challenge, stressing the need for larger datasets to improve prediction reliability. The review highlights a gap in the use of unsupervised Machine Learning approach, suggesting opportunities for future research in areas like student clustering and personalized feedback.
View
Adaptive multi‐agent smart academic advising framework
Article
Full-text available
  • Apr 2021
Abstract Academic advising is a crucial process in higher education and usually requires better understanding of student capabilities and curriculum structure to achieve its intended goals. Here, the authors propose a framework of integrated environment based on multi‐agents to automate the full process of academic advising. The proposed framework consists of six agents namely, student agent, instructor agent, administrator agent, performance agent, schedule agent, and smart advisor agent. These agents are interacting together with the help of smart advisor agent, which manages the communication between them and provides smart advice based on machine learning techniques. In addition, the analysis of the proposed framework along with the deployment map is discussed by the authors. Moreover, a case study is presented in terms of a sample part of adaptive multi‐agent smart academic advising framework to demonstrate the workflow of the proposed approach.
View
Assessing Developmental Assessment in Community Colleges
Article
Full-text available
  • Oct 2011
For many students entering community colleges, the first stop on campus is to an assessment center. More than half of these students will be placed into developmental education as a result of their scores on reading, writing, and mathematics entry assessments, yet there is little evidence that this improves student outcomes. We examine alternative perspectives on the role of assessment and how it is best implemented, review the validity of the most common assessments, and discuss emerging directions in assessment policy and practice. We conclude with implications for policy and research.
View
Education at a Glance 2013 OECD Indicators: OECD Indicators
Book
  • Sep 2013
Education at a Glance 2013: Highlights summarises the OECD’s flagship compendium of education statistics, Education at a Glance. It provides easily accessible data on key topics in education today, including: • Education levels and student numbers: How far have adults studied, and how does early childhood education affect student performance later on? • Higher education and work: How many young people graduate from tertiary education, and how easily do they enter the world of work? • Economic and social benefits of education: How does education affect people’s job prospects, and what is its impact on incomes? • Paying for education: What share of public spending goes on education, and what is the role of private spending? • The school environment: How many hours do teachers work, and how does class size vary? Each indicator is presented on a two-page spread. The left-hand page explains the significance of the indicator, discusses the main findings, examines key trends and provides readers with a roadmap for finding out more in the OECD education databases and in other OECD education publications. The right-hand page contains clearly presented charts and tables, accompanied by dynamic hyperlinks (StatLinks) that direct readers to the corresponding data in Excel™ format.
View
The Effects of Student Coaching: An Evaluation of a Randomized Experiment in Student Advising
Article
  • Feb 2013
College graduation rates often lag behind college attendance rates. One theory as to why students do not complete college is that they lack key information about how to be successful or fail to act on the information that they have. We present evidence from a randomized experiment which tests the effectiveness of individualized student coaching. Over the course of two separate school years, InsideTrack, a student coaching service, provided coaching to students attending public, private, and proprietary universities. Most of the participating students were nontraditional college students enrolled in degree programs. The participating universities and InsideTrack randomly assigned students to be coached. The coach contacted students regularly to develop a clear vision of their goals, to guide them in connecting their daily activities to their long-term goals, and to support them in building skills, including time management, self-advocacy, and study skills. Students who were randomly assigned to a coach were more likely to persist during the treatment period and were more likely to be attending the university 1 year after the coaching had ended. Coaching also proved a more cost-effective method of achieving retention and completion gains when compared with previously studied interventions such as increased financial aid.
View
Learning Analytics Dashboard Applications
Article
  • Sep 2013
This article introduces learning analytics dashboards that visualize learning traces for learners and teachers. We present a conceptual framework that helps to analyze learning analytics applications for these kinds of users. We then present our own work in this area and compare with 15 related dashboard applications for learning. Most evaluations evaluate only part of our conceptual framework and do not assess whether dashboards contribute to behavior change or new understanding, probably also because such assessment requires longitudinal studies.
View
Course signals at purdue: Using learning analytics to increase student success
  • Jan 2012
  • 267-270
  • K E Arnold
  • M D Pistilli
Arnold, K. E., & Pistilli, M. D. (2012). Course signals at purdue: Using learning analytics to increase student success. In Proceedings of the 2nd international conference on learning analytics and knowledge (pp. 267-270).