No full-text available
To read the full-text of this research,
you can request a copy directly from the authors.
Modelling students’ performance in MOOCs: a multivariate approach
Abstract
Massive Open Online Courses, universally labelled as MOOCs, become more and more relevant in the era of digitalization of higher education. The availability of free education resources without access restrictions for a plenty of potential users has changed the learning market in a way unthinkable only few decades ago. This form of web-based education allows to track all the actions of the students, thus providing an information base to understand how students' behaviour can influence their performance. The paper proposes a structural equation model in the framework of the component-based approach to measure which are the main factors affecting students' performance (Partial Least Squares Path Modelling). The novelty of the approach is the simultaneous analysis of more than one factor that exerts an impact on the performance. The analysis is carried out on the log data of a course available on the edX MOOCs platform named FedericaX.
... In this framework, studying and predicting student performance is a challenge. Performance is a complex issue that depends on motivation, attitude, and engagement (Carannante et al. 2020;de Barba et al. 2016;Moore and Wang 2021;Phan et al. 2016;among others). For MOOCs, massive by definition, this complexity is greater, as it results from great sociodemographic and cultural heterogeneity among students. ...
... The aim of the paper is to investigate the impact of sociodemographics on engagement, learning, and performance, focusing on group effects, i.e., on the study of possible differences according to the personal characteristics of students. We accordingly extend the study by Carannante et al. (2020), who used composite-based path modelling (CB-PM) (Esposito Vinzi et al. 2010;Hair et al. 2016;Wold 1985) to measure the main factors affecting MOOC student performance. Starting from a conceptualization of performance and its main drivers (engagement and learning), as discussed in Carannante et al. (2020), we analysed differences in the structural model according to student profile (gender, age, and country of origin) and course instructional design. ...
... We accordingly extend the study by Carannante et al. (2020), who used composite-based path modelling (CB-PM) (Esposito Vinzi et al. 2010;Hair et al. 2016;Wold 1985) to measure the main factors affecting MOOC student performance. Starting from a conceptualization of performance and its main drivers (engagement and learning), as discussed in Carannante et al. (2020), we analysed differences in the structural model according to student profile (gender, age, and country of origin) and course instructional design. Our ultimate aim was to analyse heterogeneity in student performance from the perspective of learning and engagement behaviour while controlling for sociodemographic profiles. ...
Massive open online courses (MOOCs) are potentially participated in by very many students from different parts of the world, which means that learning analytics is especially challenging. In this framework, predicting students’ performance is a key issue, but the high level of heterogeneity affects understanding and measurement of the causal links between performance and its drivers, including motivation, attitude to learning, and engagement, with different models recommended for the formulation of appropriate policies. Using data for the FedericaX EdX MOOC platform (Federica WebLearning Centre at the University of Naples Federico II), we exploit a consolidated composite-based path model to relate performance with engagement and learning. The model addresses heterogeneity by analysing gender, age, country of origin, and course design differences as they affect performance. Results reveal subgroups of students requiring different learning strategies to enhance final performance. Our main findings were that differences in performance depended mainly on learning for male students taking instructor-paced courses, and on engagement for older students (> 32 years) taking self-paced courses.
... The practical implications and the relevance of the approach is shown on real data in Sect. 5 through an empirical analysis on MOOC students' performance, one of the major challenges in learning analytics (Siemens and Long 2011). The empirical application allows to evaluate if and how the effect of learning and engagment, the two main drivers of student's performance (Carannante et al. 2020;de Barba et al. 2016; Moore and Wang 2021), changes according to the way the courses are offered, namely distinguishing self-paced courses and instructor-paced courses (Fianu et al. 2018;Goopio and Cheung 2020). Finally, a discussion on the main results, and the conclusions with some further research developments to be explored are included in Sect. ...
... The aim is to model students' performance in a particular type of course, the Massive Open Online Courses, also known as MOOCs. The data and model have recently been published by Carannante et al. (2020). MOOCs are an increasingly common type of course in education, especially in higher education. ...
... The considered model uses performance as response variable, and learning and engagment as explicative variable. This in line with the model proposed in Carannante et al. (2020). Performance was measured as the proportion of correct answers to a set of questions. ...
The paper aims to introduce a multigroup approach to assess group effects in quantile regression. The procedure estimates the same regression model at different quantiles, and for different groups of observations. Such groups are defined by the levels of one or more stratification variables. The proposed approach exploits a computational procedure to test group effects. In particular, a bootstrap parametric test and a permutation test are compared through artificial data taking into account different sample sizes, and comparing their performance in detecting low, medium, and high differences among coefficients pertaining different groups. An empirical analysis on MOOC students’ performance is used to show the proposal in action. The effect of the two main drivers impacting on performance, learning and engagement, is explored at different conditional quantiles, and comparing self-paced courses with instructor-paced courses, offered on the EdX platform.
... Interaction of students with advanced technologies creates an essential experience supporting a modern demand for critical skills in the digital age. (Carannante et al., 2021). ...
This systematic literature review evaluates the transformative effects of Massive Open Online Courses (MOOCs) in Sri Lanka’s higher education and library services. It explores how MOOCs democratize education by broadening access to previously costly and limited educational infrastructure and integrating them into library practices for enhanced access and professional development. The review synthesizes various sources to assess MOOC utilization, challenges like low completion rates, inconsistent quality, and digital divides. Strategic improvements are proposed to align MOOCs with educational and socio-economic goals in Sri Lanka. The findings highlight MOOCs’ current uses, barriers, and potential as supplements to traditional education, aiming to optimize learning outcomes and support educational transformation in both academic and library environments.
... Three focus on substantive questions (democracy and autocracy, world politics, and comparative political systems), while two are primarily methodological (concept building and comparative research design and methods). We scrutinise the five IPSAMOOCs in the "Introduction to Political Science" program. 2 Instead of being tied to a single university, an international university centre (Federica Web Learning) runs these MOOCs in collaboration with the International Political Science Association (IPSA) (Carannante et al., 2021(Carannante et al., , pp. 2375. Our study introduced connectivist elements to the IPSAMOOC experience of a diverse learner group. ...
Massive Open Online Courses (MOOCs), praised for their global scale and open-access elements, are commonly associated with several challenges. These include unequal access, limited interactivity, or insufficient learner background and skills. This article aims to understand how introducing online and offline connectivist elements influences MOOC learners’ engagement and motivation and self-perceived benefits from undertaking MOOCs. To do so, we present descriptive statistics and analyse results from focus groups of regionally and disciplinarily diverse learners of the Federica Web Learning – International Political Science Association (IPSA) MOOCs, who were introduced to connectivist elements in the IPSAMOOCs. We find that limited connectivist elements added to the IPSAMOOCs did not notably affect learner engagement and motivation. However, the IPSAMOOCs have considerable potential to impact engagement and motivation, if combined with offline activities. The findings contribute to studying the prospects of MOOCs as a potential avenue for accessible, global digital Political Science education, advancing the appreciation of democracy. Highlights:
• Explores how (social science) MOOCs help advance democratic consciousness.
• Presents online and offline connectivist elements in MOOC design.
• Offline connectivist elements in MOOC design amplify online communities.
• Connectivist MOOC design choices may foster learner engagement and progress.
• (Social science) MOOCs benefit from more emphasis on online community-building.
... New approaches to teaching have emerged, offering new possibilities for advancements and information delivery (Chen, Chen, and Lin 2020). As an example, MOOCs (Massive Open Online Courses), a model for delivering learning content online to any person who wants to take a course with no limit on attendance, have succeeded in delivering effective teaching to students and reducing the load of both students and teachers (Carannante, Davino, and Vistocco 2021;Loeckx 2016; Thomas and Nedeva 2018). In addition, coupled with gamification and resource digitalisation, AI can help create personalised learning experiences (VanLehn 2011). ...
... Worldwide, a large number of higher education institutions (HEIs) have used information and communication technology (ICT) as a crucial educational process [1][2][3][4]. ICT-enhanced teaching utilizes various technologies like LMS, video collaboration, MOOCs, mobile devices, and virtual labs for flexible, distance, and blended learning [1,[5][6][7]. ICT-related progression is seen in the field of schooling to a significant automated assembly for changing and advancement in preparing, subsequently making ICT intriguing to ...
The purpose of the research was to investigate university teachers' perceptions of information and communication technology (ICT) to enhance teaching and construct new knowledge. The objective was to determine how teachers at the University of the Gambia perceived the effectiveness of the TPACK (Technology, Pedagogy, and Content Knowledge) model in teaching through ICT and how different attributes of teachers affect their level of TPACK knowledge. This study used a questionnaire, adapted from the TPACK model, which was administered to 88 faculty selected purposively from the University of the Gambia to collect data on their level of awareness of ICT-based teaching. Descriptive analysis was first run to check the scores of all components of TPACK, then a correlational analysis was carried out to measure the relationship between the TPACK constructs. The data collected from these analyses were used to construct a structural equation model that showed the relationship between TPACK constructs as perceived by the faculty of the University of the Gambia. The results of this study showed that the individual construct (i.e., technology, pedagogy, and content) and the paired constructs (i.e. Technology and Pedagogy, Technology and Content, Pedagogy and Content knowledge) of TPACK are positively correlated with teachers’ perceived TPACK, as well as the findings are consistent with the earlier claim that ICT-based instruction may not progress as quickly in education even when it is available and teachers are not knowledgeable about it. The results contribute to earlier research that aims to develop strategies for implementing ICT-based instruction in HEIs and offer advice to educators on how to create policies including TPACK components in their teaching and learning.
... Advanced ICT-enhanced education platform, such as Learning Management Systems (LMS), video collaboration tools (Microsoft Teams, WebEx, and Zoom), offers diverse learning experiences that are used to organize course content, provide learning opportunities to diverse categories of learners, and provide flexible course delivery mechanisms for long distance and blended learning (Ain, Kaur, & Waheed, 2016;Bahri, Idris, Muis, Arifuddin, & Fikri, 2021). These technologies are the reasons for introducing Massive Open Online Courses (MOOCs), Mobile learning for students, Virtual labs that allow simulations of a physical experiment, serious games that engage and retain learners' attention, personalized blended learning using learning analytics, mobile devices for engaging students (Callaghan, Savin-Baden, McShane, & Eguiluz, 2017;Carannante, Davino, & Vistocco, 2020;Khan, Abdou, Kettunen, & Gregory, 2019). Thus, ICT continues to provide new and emerging opportunities that simplify ways of representing and delivering teaching and learning experiences in HEIs around the globe. ...
This study investigates the knowledge that engineering teachers should possess in order to effectively implement technology-enhanced instruction in their teaching practice using Technological Pedagogical Content Knowledge (TPACK) framework, although its (TPACK) use in HEIs is inadequate. The objectives of this investigation are to investigate what TPACK construct is using in engineering education (Eng. Ed) and to study how different attributes of a teacher affect their level of TPACK knowledge. In order to accumulate engineer teachers' knowledge, a descriptive self-assessment tool designed in a Google form was administered via email to 220 teachers from two different universities of Bangladesh located in the business district of Dhaka. Descriptive analysis, Pearson's correlation coefficient (r), Exploratory Factor Analysis, Cronbach Alpha test, ANOVA, and Levene test were carried out to analyse the collected data. The outcomes of this investigation confirmed the practicality of the framework and discovered significant differences regarding technological knowledge (TK), conventional knowledge (PK/PCK) in field of study of the teacher and a significant difference in technology-enhanced instructions in regard to age group of the teacher. The results support the previous argument that only availability of technology and teachers' technology knowledge in Eng. Ed may not accelerate technology-enhanced teaching. The findings add knowledge to prior research whose objective is to find ways of incorporating technology-enhanced instructions in HEIs and thus, provide recommendation to Eng. Ed towards formulating policies on incorporating TPACK components in their teaching.
... A number of studies have identified the significance of student satisfaction to use of the MOOCs system as a sustainability education approach. As a measure, the findings of this study confirm previous findings [27,42,62,75,76]. According to [77], the majority of e-tutoring users felt that online sources for learning English offer greater convenience and are more effective than no internet resources. ...
The aim of the present study was to narrow the gap in the literature on the adoption of massive open online courses (MOOCs) and the role of task-technology fit (TTF), which influences student satisfaction, academic performance, and the long-term viability (sustainability) of MOOCs in higher education. While researchers have examined MOOC acceptance in a variety of contexts, the role of TTF as a mediating variable in evaluating education sustainability has not been explored using the technology acceptance model (TAM). As a result, the aim of this study was to create a new paradigm by combining two theories: TTF and TAM. Therefore, this study surveyed 277 university students from public universities using the structural equation modeling (SEM) approach to learn about their perceptions toward MOOCs as a method of achieving higher education sustainability. According to the findings, perceived ease of use had a positive impact on perceived enjoyment, perceived usefulness, and social influence, which in turn had a positive impact on task-technology fit and MOOCs use as a method of sustainability in higher education. Task-technology fit also had a positive impact on MOOC use as a method of sustainability. Finally, the role of task-technology fit and MOOCs in educational sustainability had a positive effect on students satisfaction and academic performance. As a result, the use of MOOCs in learning processes should be encouraged in higher education institutions to ensure their long-term viability (sustainability).
This study investigates the effects of student engagement and rapidity of completing exams on student performance before and during the first wave of COVID-19 in March 2020, examining the effect of the shift from face-to-face to online teaching and exams in a Master’s in Business Administration degree at a university in Italy. Prior literature mainly finds that student marks benefit from student engagement, but it has been unclear how COVID-19 affected this link. We find that COVID-19 reduced this benefit in the short term. Prior literature also finds that student performance benefits from passing the exam at the earliest opportunity but the effect of COVID-19-related changes on this remains unclear. We find that the link between higher exam marks and rapidity of completing exams was strengthened by COVID-19. The research contributes to the debate on costs and benefits of COVID-19 on accounting education quality. It confirms that there are disadvantages, in terms of the lower efficacy of student engagement, and advantages, in terms of higher marks from more rapid academic progress.
Using the two-stage Heckman procedure, a preliminary analysis is proposed on the job satisfaction of immigrants in Campania (Italy). Data come from a sample survey carried out in 2013 based on the “centre sampling technique”. Citizenship and gender are prominent features, as well as the type and characteristics of work activity; over-education turns out to be of little relevance to satisfaction. The levels of social and cultural integration are significant factors.
This chapter analyses the deployment of active and blended approaches to course design and teaching across one of the largest writing programmes in the British higher education sector. The programme, which teaches and assesses 12,000 students per academic year, adopts an entirely active and blended approach to all elements of its design. This chapter presents the pedagogical underpinnings to the programme, a case study of how the programme has been deployed, and discussion of some key challenges. The chapter establishes that active and blended learning approaches were central to the success of the programme, and that without such pedagogical designs the programme would have failed to meet its essential requirements.
p> The main purpose of this study is to investigate whether emotional, cognitive and behavioral engagements, represents three conceptually and empirically distinct psychological constructs when studied within the same domain. This paper reports part of the findings from a major study entitled “Predictors of Self-Handicapping Behavior among Muslim Students”. Testing for factorial equivalence of scores from a measuring instrument was carried-out through structural equation modeling by using AMOS version 16. Results of Confirmatory Factor Analysis of responses from 790 undergraduates prove that the SEM three factor model of University Student Engagement (USE) is empirically fit and reliable, which also supports the argument that emotion, behavior and cognition are the student engagement manifestations of an interrelated constellation of academic student engagement. </p
Exploring theoretically plausible alternative models for explaining the phenomenon under study is a crucial step in advancing scientific knowledge. This paper advocates model selection in Information Systems (IS) studies that use Partial Least Squares path modeling (PLS) and suggests the use of model selection criteria derived from Information Theory for this purpose. These criteria allow researchers to compare alternative models and select a parsimonious yet well-fitting model. However, as our review of prior IS research practice shows, their use—while common in the econometrics field and in factor-based SEM—has not found its way into studies using PLS. Using a Monte Carlo study, we compare the performance of several model selection criteria in selecting the best model from a set of competing models under different model set-ups and various conditions of sample size, effect size, and loading patterns. Our results suggest that appropriate model selection cannot be achieved by relying on the PLS criteria (i.e., R2, Adjusted R2, GoF, and Q2), as is the current practice in academic research. Instead, model selection criteria, in particular the Bayesian information criterion (BIC) and the Geweke-Meese criterion (GM), should be used due to their high model selection accuracy and ease of use. To support researchers in the adoption of these criteria, we introduce a five-step procedure that delineates the roles of model selection and statistical inference, and discuss misconceptions that may arise in their use.
Predicting student performance is a major tool in learning analytics. This study aims to identify how different measures of massive open online course (MOOC) data can be used to identify points of improvement in MOOCs. In the context of MOOCs, student performance is often defined as course completion. However, students could have other learning objectives than MOOC completion. Therefore, we define student performance as obtaining personal learning objective(s). This study examines a subsample of students in a graduate-level blended MOOC who shared on-campus course completion as a learning objective. Aggregated activity frequencies, specific course item frequencies, and order of activities were analysed to predict student performance using correlations, multiple regressions, and process mining. All aggregated MOOC activity frequencies related positively to on-campus exam grade. However, this relation is less clear when controlling for past performance. In total, 65% of the specific course items showed significant correlations with final exam grade. Students who passed the course spread their learning over more days compared with students who failed. Little difference was found in the order of activities within the MOOC between students who passed and who failed. The results are combined with course evaluations to identify points of improvement within the MOOC. © 2018 The Authors. Journal of Computer Assisted Learning Published by John Wiley & Sons, Ltd.
The field of learning analytics needs to adopt a more rigorous approach for predictive model evaluation that matches the complex practice of model-building. In this work, we present a procedure to statistically test hypotheses about model performance which goes beyond the state-of-the-practice in the community to analyze both algorithms and feature extraction methods from raw data. We apply this method to a series of algorithms and feature sets derived from a large sample of Massive Open Online Courses (MOOCs). While a complete comparison of all potential modeling approaches is beyond the scope of this paper, we show that this approach reveals a large gap in dropout prediction performance between forum-, assignment-, and clickstream-based feature extraction methods, where the latter is significantly better than the former two, which are in turn indistinguishable from one another. This work has methodological implications for evaluating predictive or AI-based models of student success, and practical implications for the design and targeting of at-risk student models and interventions.
Purpose
The purpose of this paper is to present a systematic review of the mounting research work on learning analytics.
Design/methodology/approach
This study collects and summarizes information on the use of learning analytics. It identifies how learning analytics has been used in the higher education sector, and the expected benefits for higher education institutions. Empirical research and case studies on learning analytics were collected, and the details of the studies were categorized, including their objectives, approaches, and major outcomes.
Findings
The results show the benefits of learning analytics, which help institutions to utilize available data effectively in decision making. Learning analytics can facilitate evaluation of the effectiveness of pedagogies and instructional designs for improvement, and help to monitor closely students’ learning and persistence, predict students’ performance, detect undesirable learning behaviours and emotional states, and identify students at risk, for taking prompt follow-up action and providing proper assistance to students. It can also provide students with insightful data about their learning characteristics and patterns, which can make their learning experiences more personal and engaging, and promote their reflection and improvement.
Originality/value
Despite being increasingly adopted in higher education, the existing literature on learning analytics has focussed mainly on conventional face-to-face institutions, and has yet to adequately address the context of open and distance education. The findings of this study enable educational organizations and academics, especially those in open and distance institutions, to keep abreast of this emerging field and have a foundation for further exploration of this area.
Partial least squares structural equation modeling (PLS-SEM) has become a popular method for estimating (complex) path models with latent variables and their relationships. Building on an introduction of the fundamentals of measure- ment and structural theory, this chapter explains how to specify and estimate path models using PLS-SEM. Complementing the introduction of the PLS-SEM method and the description of how to evaluate analysis results, the chapter also offers an overview of complementary analytical techniques. An application of the PLS-SEM method to a well-known corporate reputation model using the SmartPLS 3 software illustrates the concepts.
Open online distance learning in higher education has quickly gained popularity, expanded, and evolved, with Massive Open Online Courses (MOOCs) as the most recent development. New web technologies allow for scalable ways to deliver video lecture content, implement social forums and track student progress in MOOCs. However, we remain limited in our ability to assess complex and open‑ended student assignments. In this paper, we present a study on various forms of assessment and their relationship with the final exam score. In general, the reliability of both the self‑assessments and the peer assessments was high. Based on low correlations with final exam grades as well as with other assessment forms, we conclude that self‑assessments might not be a valid way to assess students performance in MOOCs. Yet the weekly quizzes and peer assessment significantly explained differences in students final exam scores, with one of the weekly quizzes as the strongest explanatory variable. We suggest that both self‑assessment and peer assessment would better be used as assessment for learning instead of assessment of learning. Future research on MOOCs implies a reconceptualization of education variables, including the role of assessment of students achievements.
Massive Open Online Courses represent a fertile ground for examining student behavior. However, due to their openness MOOC attract a diverse body of students, for the most part, unknown to the course instructors. However, a certain number of students enroll in the same course multiple times, and there are records of their previous learning activities which might provide some useful information to course organizers before the start of the course. In this study, we examined how student behavior changes between subsequent course offerings. We identified profiles of returning students and also interesting changes in their behavior between two enrollments to the same course. Results and their implications are further discussed.
Students who are enrolled in MOOCs tend to have different motivational patterns than fee-paying college students. A majority of MOOC students demonstrate characteristics akin more to "tourists" than formal learners. As a consequence, MOOC students’ completion rate is usually very low. The current study examines the relations among student motivation, engagement, and retention using structural equation modeling and data from a Penn State University MOOC. Three distinct types of motivation are examined: intrinsic motivation, extrinsic motivation, and social motivation. Two main hypotheses are tested: (a) motivation predicts student course engagement; and (b) student engagement predicts their retention in the course. The results show that motivation is significantly predictive of student course engagement. Furthermore, engagement is a strong predictor of retention. The findings suggest that promoting student motivation and monitoring individual students’ online activities might improve course retention
Within the evolution of technology in education, Learning Analytics has reserved its position as a robust technological field that promises to empower instructors and learners in different educational fields. The 2014 horizon report (Johnson et al., 2014), expects it to be adopted by educational institutions in the near future. However, the processes and phases as well as constraints are still not deeply debated. In this research study, the authors talk about the essence, objectives and methodologies of Learning Analytics and propose a first prototype life cycle that describes its entire process. Furthermore, the authors raise substantial questions related to challenges such as security, policy and ethics issues that limit the beneficial appliances of Learning Analytics processes.
The most popular measures of multidimensional constructs typically fail to meet standards of good measurement: goodness of fit, measurement invariance, lack of differential item functioning, and well-differentiated factors that are not so highly correlated as to detract from their discriminant validity. Part of the problem, the authors argue, is undue reliance on overly restrictive independent cluster models of confirmatory factor analysis (ICM-CFA) in which each item loads on one, and only one, factor. Here the authors demonstrate exploratory structural equation modeling (ESEM), an integration of the best aspects of CFA and traditional exploratory factor analyses (EFA). On the basis of responses to the 11-factor Motivation and Engagement Scale (n = 7,420, Mage = 14.22), we demonstrate that ESEM fits the data much better and results in substantially more differentiated (less correlated) factors than corresponding CFA models. Guided by a 13-model taxonomy of ESEM full-measurement (mean structure) invariance, the authors then demonstrate invariance of factor loadings, item intercepts, item uniquenesses, and factor variancescovariances, across gender and over time. ESEM has broad applicability to other areas of research that cannot be appropriately addressed with either traditional EFA or CFA and should become a standard tool for use in psychometric tests of psychological assessment instruments.
Engagement is one of the most widely misused and overgeneralized constructs found in the educational, learning, instructional, and psychological sciences. The articles in this special issue represent a wide range of traditions and highlight several key conceptual, theoretical, methodological, and analytical issues related to defining and measuring engagement. All the approaches exemplified by the contributors show different ways of conceptualizing and measuring engagement and demonstrate the strengths and weaknesses of each method to significantly augment our current understanding of engagement. Despite the numerous issues raised by the authors of this special issue and in my commentary, I argue that focusing on process data will lead to advances in models, theory, methods, analytical techniques, and ultimately instructional recommendations for learning contexts that effectively engage students.
In the last five years, the world has seen a remarkable level of interest in Massive Open Online Courses, or MOOCs. A consistent message from universities participating in MOOC delivery is their eagerness to understand students' online learning processes. This paper reports on an exploratory investigation of students' learning processes in two MOOCs which have different curriculum and assessment designs. When viewed through the lens of common MOOC learning analytics, the high level of initial student interest and, ultimately, the high level of attrition, makes these two courses appear very similar to each other, and to MOOCs in general. With the goal of developing a greater understanding of students' patterns of learning behavior in these courses, we investigated alternative learning analytic approaches and visual representations of the output of these analyses. Using these approaches we were able to meaningfully classify student types and visualize patterns of student engagement which were previously unclear. The findings from this research contribute to the educational community's understanding of students' engagement and performance in MOOCs, and also provide the broader learning analytics community with suggestions of new ways to approach learning analytic data analysis and visualization.
As MOOCs grow in popularity, the relatively low completion rates of learners has been a central criticism. This focus on completion rates, however, reflects a monolithic view of disengagement that does not allow MOOC designers to target interventions or develop adaptive course features for particular subpopulations of learners. To address this, we present a simple, scalable, and informative classification method that identifies a small number of longitudinal engagement trajectories in MOOCs. Learners are classified based on their patterns of interaction with video lectures and assessments, the primary features of most MOOCs to date.
In an analysis of three computer science MOOCs, the classifier consistently identifies four prototypical trajectories of engagement. The most notable of these is the learners who stay engaged through the course without taking assessments. These trajectories are also a useful framework for the comparison of learner engagement between different course structures or instructional approaches. We compare learners in each trajectory and course across demographics, forum participation, video access, and reports of overall experience. These results inform a discussion of future interventions, research, and design directions for MOOCs. Potential improvements to the classification mechanism are also discussed, including the introduction of more fine-grained analytics.
Nell’era di Internet, delle tecnologie mobili e dell’istruzione aperta, la necessità di interventi per migliorare l’efficienza e la qualità dell’istruzione superiore è diventata pressante. I big data e il Learning Analytics possono contribuire a condurre questi interventi, e a ridisegnare il futuro dell’istruzione superiore. Basare le decisioni su dati e sulle evidenze empiriche sembra incredibilmente ovvio. Tuttavia, l’istruzione superiore, un campo che raccoglie una quantità enorme di dati sui propri “clienti”, è stata tradizionalmente inefficiente nell’utilizzo dei dati, spesso operando con notevole ritardo nell’analizzarli, pur essendo questi immediatamente disponibili.
In questo articolo, viene evidenziato il valore delle tecniche di analisi dei dati per l’istruzione superiore, e presentato un modello di sviluppo per i dati legati all’apprendimento. Ovviamente, l’apprendimento è un fenomeno complesso, e la sua descrizione attraverso strumenti di analisi non è semplice; pertanto, l’articolo presenta anche le principali problematiche etiche e pedagogiche connesse all’utilizzo delle tecniche di analisi dei dati in ambito educativo. Cionondimeno, il Learning Analytics può penetrare la nebbia di incertezza che avvolge il futuro dell’istruzione superiore, e rendere più evidente come allocare le risorse, come sviluppare vantaggi competitivi e, soprattutto, come migliorare la qualità e il valore dell’esperienza di apprendimento.
In studying online learning, researchers should examine three critical interactions: instructor-student, student-student, and student-content. Student-content interaction may include a wide variety of pedagogical tools (e.g., streaming media, PowerPoint, and hyperlinking). Other factors that can affect the perceived quality of online learning include distance education advantages (e.g., work and family flexibility) and antecedent personal characteristics (e.g., experience and gender). The study indicated that instructor-student interaction is most important, twice that of student-student interaction; that some student-content interaction is significantly related to perceived learning; that antecedent variables are not significant; and that distance education advantages/flexibility, although significant, are less important than other interactions.
This paper discusses a recent development in partial least squares (PLS)
path modeling, namely goodness-of-fit indices. In order to illustrate the behavior of
the goodness-of-fit index (GoF) and the relative goodness-of-fit index (GoF_rel), we
estimate PLS path models with simulated data, and contrast their values with fit indices
commonly used in covariance-based structural equation modeling. The simulation
shows that the GoF and the GoF_rel are not suitable for model validation. However, the
GoF can be useful to assess how well a PLS path model can explain different sets of
data.
A large proportion of information systems research is concerned with developing and testing models pertaining to complex cognition, behaviors, and outcomes of individuals, teams, organizations, and other social systems that are involved in the development, implementation, and utilization of information technology. Given the complexity of these social and behavioral phenomena, heterogeneity is likely to exist in the samples used in IS studies. While researchers now routinely address observed heterogeneity by introducing moderators, a priori groupings, and contextual factors in their research models, they have not examined how unobserved heterogeneity may affect their findings. We describe why unobserved heterogeneity threatens different types of validity and use simulations to demonstrate that unobserved heterogeneity biases parameter estimates, thereby leading to Type I and Type II errors. We also review different methods that can be used to uncover unobserved heterogeneity in structural equation models. While methods to uncover unobserved heterogeneity in covariance-based structural equation models (CB-SEM) are relatively advanced, the methods for partial least squares (PLS) path models are limited and have relied on an extension of mixture regression-finite mixture partial least squares (FIMIX-PLS) and distance measure-based methods-that have mismatches with some characteristics of PLS path modeling. We propose a new method-prediction-oriented segmentation (PLS-POS)-to overcome the limitations of FIMIX-PLS and other distance measure-based methods and conduct extensive simulations to evaluate the ability of PLS-POS and FIMIX-PLS to discover unobserved heterogeneity in both structural and measurement models. Our results show that both PLS-POS and FIMIX-PLS perform
To date, multi-group comparison of Partial Least Square (PLS) models where differences in path estimates for different sampled
populations have been relatively naive. Often, researchers simply examine and discuss the difference in magnitude of specific
model path estimates from two or more data sets. When evaluating the significance of path differences, a t-test based on the pooled standard errors obtained via a resampling procedure such as bootstrapping from each data set is
made. Yet problems can occur if the assumption of normal population or similar sample size is made. This paper provides an
introduction to an alternative distribution free approach based on an approximate randomization test – where a subset of all
possible data permutations between sample groups is made. The performance of this permutation procedure is tested on both
simulated data and a study exploring the differences of factors that impact outsourcing between the countries of US and Germany.
Furthermore, as an initial examination of the consistency of this new procedure, the outsourcing results are compared with
those obtained from using covariance based SEM (AMOS 7).
Massive Open Online Courses (MOOCs) provide a great platform to study individual and group differences of learners in perceptions, motivations, and behaviors under self-directed learning context. This study examined the relationships, in particular, influential relationships, among MOOC learners’ demographics, their self-regulated learning (SRL) strategy usage, perceived learning, and satisfaction. Participants were 4503 learners from 17 Coursera courses who responded to an online survey in 2018. Structural equation modeling showed that participants’ age, gender, highest degree, and the number of online courses previously taken significantly predicted both goal setting and environment structuring usage. Previous experience with the course topics only predicted goal setting, not environment structuring. Gender, goal setting and environment structuring strategy usage predicted participants’ perceived affective learning. Highest degree, the number of online courses previously took, goal setting, environment structuring strategy usage and perceived affective learning predicted participants’ satisfaction with the course. Participants identified themselves with a Latin America culture had better environment structuring strategy usage than any other cultural group and higher perceived affective learning than the other cultural groups except for Other. The results provided implications for researchers studying self-directed learning environments, differences in learning of learners with diverse backgrounds, and SRL behaviors, as well as for educators dealing with increasing SRL strategy usage, improving online learners’ satisfaction and teaching cross-culturally.
Recently, interest in how this data can be used to improve teaching and learning has also seen unprecedented growth and the emergence of the field of learning analytics. In other fields, analytics tools already enable the statistical evaluation of rich data sources and the identification of patterns within the data. These patterns are then used to better predict future events and make informed decisions aimed at improving outcomes (Educause, 2010). This paper reviews the literature related to this emerging field and seeks to define learning analytics, its processes, and its potential to advance teaching and learning in online education.
Partial least squares path modeling (PLS-PM) has become popular in various disciplines to model structural relationships among latent variables measured by manifest variables. To fully benefit from the predictive capabilities of PLS-PM, researchers must understand the efficacy of predictive metrics used. In this research, we compare the performance of standard PLS-PM criteria and model selection criteria derived from Information Theory, in terms of selecting the best predictive model among a cohort of competing models. We use Monte Carlo simulation to study this question under various sample sizes, effect sizes, item loadings, and model setups. Specifically, we explore whether, and when, the in-sample measures such as the model selection criteria can substitute for out-of-sample criteria that require a holdout sample. Such a substitution is advantageous when creating a holdout causes considerable loss of statistical and predictive power due to an overall small sample. We find that when the researcher does not have the luxury of a holdout sample, and the goal is selecting correctly specified models with low prediction error, the in-sample model selection criteria, in particular the Bayesian Information Criterion (BIC) and Geweke-Meese Criterion (GM), are useful substitutes for out-of-sample criteria. When a holdout sample is available, the best performing out-of-sample criteria include the root mean squared error (RMSE) and mean absolute deviation (MAD). Finally, we recommend against using standard the PLS-PM criteria (R2, Adjusted R2, and Q2), and specifically the out-of-sample mean absolute percentage error (MAPE) for prediction-oriented model selection purposes. Finally, we illustrate the model selection criteria’s practical utility using a well-known corporate reputation model.
We present a systematic literature review of the emerging field of visual learning analytics. We review existing work in this field from two perspectives: First, we analyze existing approaches, audiences, purposes, contexts, and data sources—both individually and in relation to one another—that designers and researchers have used to visualize educational data. Second, we examine how established literature in the fields of information visualization and education has been used to inform the design of visual learning analytics tools and to discuss research findings. We characterize the reviewed literature based on three dimensions: (a) connection with visualization background; (b) connection with educational theory; and (c) sophistication of visualization(s). The results from this systematic review suggest that: (1) little work has been done to bring visual learning analytics tools into classroom settings; (2) few studies consider background information from the students, such as demographics or prior performance; (3) traditional statistical visualization techniques, such as bar plots and scatter plots, are still the most commonly used in learning analytics contexts, while more advanced or novel techniques are rarely used; (4) while some studies employ sophisticated visualizations, and some engage deeply with educational theories, there is a lack of studies that both employ sophisticated visualizations and engage deeply with educational theories. Finally, we present a brief research agenda for the field of visual learning analytics based on the findings of our literature review.
Before the emergence of computer‐based educational systems (CBES) whose aims of providing teaching and learning experiences to hundreds even thousands of users, an explosion of information (e.g., students' log data) demands sophisticated methods to gather, analyze, and interpret learners' traces to regulate and enhance education. Thus, learning analytics (LA) arises as a knowledge discovery paradigm that provides valuable findings and facilitates stakeholders to understand the learning process and its implications. Therefore, a landscape of the LA nature, its underlying factors, and applications achieved is outlined in this paper according to a suggested LA Taxonomy that classifies the LA duty from a functional perspective. The aim is to provide an idea of the LA toil, its research lines, and trends to inspire the development of novel approaches for improving teaching and learning practices. Furthermore, the scope of this review covers recently published papers in prestigious journals and conferences, where the works dated from 2016 are summarized and those corresponding to 2014–2015 are cited according to the proposed LA taxonomy. A glimpse is sketched of LA, where underlying elements frame the field foundations to ground the approaches. Moreover, LA strengths, weaknesses, challenges, and risks are highlighted to advice how the LA arena could be enhanced and empowered. In addition, this review offers an insight of the recent LA labor, as well as motivates readers to enrich the LA achievements. This work promotes the LA practice giving an account of the job being achieved and reported in literature, as well as a reflection of the state‐of‐the‐art and an acumens vision to inspire future labor.
This article is categorized under: Application Areas > Education and Learning
Application Areas > Science and Technology
Fundamental Concepts of Data and Knowledge > Human Centricity and User Interaction
In this article, we review applications of covariance-based structural equation modeling (SEM) in the Journal of Advertising (JA) starting with the first issue in 1972. We identify 111 articles from the earliest application of SEM in 1983 through 2015, and discuss important methodological issues related to the following aspects: confirmatory factor analysis (CFA), causal modeling, multiple group analysis, reporting, and guidelines for interpretation of results. Moreover, we summarize some issues related to varying terminology associated with different SEM methods. Findings indicate that the use of SEM in the JA contributes greatly to conceptual, empirical, and methodological advances in advertising research. The assessment contributes to the literature by offering advertising researchers a summary guide to best practices and a reminder of the basics that distinguish the powerful and unique approach involving structural analysis of covariances.
Massive open online courses (MOOCs) are a new form of educational provision occupying a space between formal online courses and informal learning. Adopting measures used with formal online courses to assess the outcomes of MOOCs is often not informative because the context is very different. The particular affordances of MOOCs shaping learning environments comprise scale (in terms of numbers of students) and diversity (in terms of the types of students). As learning designers, we focus on understanding the particular tools and pedagogical affordances of the MOOC platform to support learner engagement. Drawing on research into learner engagement conducted in the broader field of online learning, we consider how learner engagement in a MOOC might be designed for by looking at three pedagogical aspects: teacher presence, social learning, and peer learning. © 2016 Open and Distance Learning Association of Australia, Inc.
It is shown that a strongly consistent estimation procedure for the order of an autoregression can be based on the law of the iterated logarithm for the partial autocorrelations. As compared to other strongly consistent procedures this procedure will underestimate the order to a lesser degree.
Modeling has often failed to meet expectations, mostly because of the difficulty of comprehending relationships within phenomena and expressing them in mathematical models. Reality is frequently too complex to be reflected in a single model. This is often the case of marketing research, where variables relating to socioeconomics or psychographics constitute potential sources of heterogeneity. In such cases, the assumption of ‘one model fits all’ is unrealistic and may lead to inaccurate decisions. Thus, heterogeneity is a major issue in modeling. Once a model has been fitted to a complete data set that fulfills all validation criteria, it is difficult to establish whether it is valid for the whole population or it is merely an average artifact from several sub-populations. The purpose of this paper is to present the Pathmox approach to deal with heterogeneity in partial least squares path modeling. The idea behind Pathmox is to build a tree of path models that have look-alike structure as a binary decision tree, with different models for each of its nodes. The split criterion consists of an F statistic comparing two structural models. In order to ensure the suitability of the split criterion, a simulation study was conducted. Finally, we have applied Pathmox to a survey that measured Satisfaction among Spanish mobile phone operators. Results suggest that the Pathmox approach performs adequately in detecting partial least squares path modeling heterogeneity. Copyright © 2016 John Wiley & Sons, Ltd.
Over the last 5 years, massive open online courses (MOOCs) have increasingly provided learning opportunities across the world in a variety of domains. As with many emerging educational technologies, why and how people come to MOOCs needs to be better understood and importantly what factors contribute to learners' MOOC performance. It is known that online learning environments require greater levels of self-regulation, and that high levels of motivation are crucial to activate these skills. However, motivation is a complex construct and research on how it functions in MOOCs is still in its early stages. Research presented in this article investigated how motivation and participation influence students' performance in a MOOC, more specifically those students who persist to the end of the MOOC. Findings indicated that the strongest predictor of performance was participation, followed by motivation. Motivation influenced and was influenced by students' participation during the course. Moreover, situational interest played a crucial role in mediating the impact of general intrinsic motivation and participation on performance. The results are discussed in relation to how educators and designers of MOOCs can use knowledge emerging from motivational assessments and participation measures gleaned from learning analytics to tailor the design and delivery of courses.
This study compared students' academic procrastination tendency with the (1) frequency and nature of rationalizations used to justify procrastination, (2) self-regulation, and (3) performance in a web-based study strategies course with frequent performance deadlines. 106 college students completed the 16-item Tuckman Procrastination Scale, a measure of tendency to procrastinate, the Frequency of Use Self-survey of Rationalizations for Procrastination, and a 9-item self-regulation scale. Students' subsequent course performance was measured by total points earned. A linear regression with Academic Procrastination as the criterion variable and Rationalization score and Course Points as the predictor variables suggested academic procrastinators support procrastinating by rationalizing, not self-regulating, and thus put themselves at a disadvantage, with respect to evaluation in highly structured courses with frequent enforced deadlines.
Partial Least Squares (PLS) based Structural Equation Modeling (SEM) has become increasingly popular in Management Information Systems (MIS) research to model complex relationships and to make valid inferences from the restricted sample to the larger population. Given the larger goal of creating generalizable theories in MIS research, we argue that the lack of model selection criteria in PLS that penalize model complexity might be causing researchers to select unnecessarily complex but highly fitting models that may not generalize to other samples. We introduce several Information Theoretic (IT) model selection criteria in the PLS context that penalize model complexity but reward high fit, and therefore guide researchers to select a parsimonious and generalizable model. In this Monte Carlo study, we compare their performance to the currently existing PLS indices, in selecting the best model among a set of competing models under various conditions of sample size, effect size and data distribution. Based on our simulation results, we strictly advise against the use of R2 and GoF based measures in PLS for model selection. Instead, we demonstrate that the IT criteria have much superior model selection rates than the currently existing PLS indices. Therefore, we recommend a core set of IT criteria that researchers should regularly employ when selecting models among a competing set of models using PLS based SEM.
This study sought to identify significant behavioral indicators of learning using learning management system (LMS) data regarding online course achievement. Because self-regulated learning is critical to success in online learning, measures reflecting self-regulated learning were included to examine the relationship between LMS data measures and course achievement. Data were collected from 530 college students who took an online course. The results demonstrated that students' regular study, late submissions of assignments, number of sessions (the frequency of course logins), and proof of reading the course information packets significantly predicted their course achievement. These findings verify the importance of self-regulated learning and reveal the advantages of using measures related to meaningful learning behaviors rather than simple frequency measures. Furthermore, the measures collected in the middle of the course significantly predicted course achievement, and the findings support the potential for early prediction using learning performance data. Several implications of these findings are discussed.
Patterns in student accesses of online materials and their effects upon student performance in a blended course are examined. Our blended course is an introductory business and economic statistics course where lectures are only available online while the traditional class period is used for complementary learning activities. Timing, volumes, intensity, and consistency of the student accesses of the online lectures are considered. Using bivariate and multivariate analyses, measures of timing and consistency are shown to be related to student performance but volumes and intensity of accesses are not.
In massive open online courses (MOOCs), low barriers to registration attract large numbers of students with diverse interests and backgrounds, and student use of course content is asynchronous and unconstrained. The authors argue that MOOC data are not only plentiful and different in kind but require reconceptualization—new educational variables or different interpretations of existing variables. The authors illustrate this by demonstrating the inadequacy or insufficiency of conventional interpretations of four variables for quantitative analysis and reporting: enrollment, participation, curriculum, and achievement. Drawing from 230 million clicks from 154,763 registrants for a prototypical MOOC offering in 2012, the authors present new approaches to describing and understanding user behavior in this emerging educational context.
The concept of school engagement has attracted increasing attention as representing a possible antidote to declining academic motivation and achievement. Engagement is presumed to be malleable, responsive to contextual features, and amenable to environmental change. Researchers describe behavioral, emotional, and cognitive engagement and recommend studying engagement as a multifaceted construct. This article reviews definitions, measures, precursors, and outcomes of engagement; discusses limitations in the existing research; and suggests improvements. The authors conclude that, although much has been learned, the potential contribution of the concept of school engagement to research on student experience has yet to be realized. They call for richer characterizations of how students behave, feel, and think—research that could aid in the development of finely tuned interventions
Racial and ethnic achievement gaps narrowed substantially in the 1970s and 1980s. As some of the gaps widened in the 1990s, there were some setbacks in the progress the nation made toward racial and ethnic equity. This article offers a look below the surface at Black-White and Hispanic-White achievement gap trends over the past 30 years. The literature review and data analysis identify the key factors that seem to have contributed to bifurcated patterns in achievement gaps. The conventional measures of socioeconomic and family conditions, youth culture and student behavior, and schooling conditions and practices might account for some of the achievement gap trends for a limited time period or for a particular racial and ethnic group. However, they do not fully capture the variations. This preliminary analysis of covariations in racial and ethnic gap patterns across several large data sets has implications for future research on the achievement of minority groups.
The present study examines whether the relationship between work engagement and job performance is moderated by the extent to which individuals are inclined to work hard, careful, and goal-oriented. On the basis of the literature, it was hypothesized that conscientiousness strengthens the relationship between work engagement and supervisor ratings of task and contextual performance as well as active learning. The hypotheses were tested on a sample of 144 employees from several occupations. Results of moderated structural equation modeling supported the hypotheses. Work engagement was positively related to task performance, contextual performance, and active learning, particularly for employees high in conscientiousness.
Using Akaike's information criterion, three examples of statistical data are reanalyzed and show reasonably definite conclusions. One is concerned with the multiple comparison problem for the means in normal populations. The second is concerned with the grouping of the categories in a contingency table. The third is concerned with the multiple comparison problem for the analysis of variance by the iogit model in contingency tables, Finite correction of Akaike's information criterionis also proposed.
This study focused on determining the relationships among student characteristics, such as gender, learning style, and varying prior computer experiences, and students’ linear and non-linear navigation of a hypermedia program. Additionally, the navigation patterns were analyzed at three different intervals to determine the relative temporal influence of these characteristics on linear and nonlinear navigation. It was found that authoring, programming, and gender (specifically female) were positively related to linear navigation during the early interval; that learning style (specifically field independent) and hypermedia experience were negatively related to linear navigation during the early interval; and that word processing experience, database experience, spreadsheet experience, learning style (specifically field independent), and hypermedia experience were positively related to nonlinear navigation during the early interval. At the middle interval, many of the characteristics were no longer distinguishing factors of linear or nonlinear behavior. Programming was still positively related to linear navigation; hypermedia experience, learning style (field independent), word processing experience, and database experience were negatively related to linear navigation. Hypermedia experience was the only factor having a relationship with nonlinear navigation; the relationship was positive. At the late interval, only years of computer experience and hypermedia were still distinguishing factors. Years of computer experience was positively related to linear navigation, and hypermedia was negatively related to linear navigation.
Structural equation models (SEMs) make it possible to estimate the causal relationships, defined according to a theoretical model, linking two or more latent complex concepts, each measured through a number of observable indicators, usually called manifest variables. Traditionally, the component-based estimation of SEMs by means of partial least squares (PLS path modelling, PLS-PM) assumes homogeneity over the observed set of units: all units are supposed to be well represented by a unique model estimated on the overall data set. In many cases, however, it is reasonable to expect classes made of units showing heterogeneous behaviours to exist.
Two different kinds of heterogeneity could be affecting the data: observed and unobserved heterogeneity. The first refers to the case of a priori existing classes, whereas in unobserved heterogeneity no information is available either on the number of classes or on their composition.
If a group structure for the statistical units is given, the aim of the analysis is to search for any differences in the behaviours of the a priori given classes. In PLS-PM this would mean studying the effect of directly observed moderating variables, i.e. estimating as many (local) models as there are classes.
Unobserved heterogeneity, instead, implies identifying classes of units (a priori unknown) having similar behaviours. Such heterogeneity is captured by an unobserved (latent) discrete moderating variable defining both the number of classes and the class membership.
A new method for unobserved heterogeneity detection in PLS-PM is proposed in this paper: response-based procedure for detecting unit segments in PLS-PM (REBUS-PLS). REBUS-PLS, according to PLS-PM features, does not require distributional hypotheses and may lead to local models that are different in terms of both structural and measurement models. An application of REBUS-PLS on real data will be shown. Copyright