Article

Using Process Mining to Examine the Sustainability of Instructional Support: How Stable Are the Effects of Metacognitive Prompting on Self-Regulatory Behavior?

June 2018
Computers in Human Behavior 96

DOI:10.1016/j.chb.2018.06.003

Authors:

Christoph Sonnenberg

BAUR Versand - A member of the Otto Group

Maria Bannert

Technische Universität München

The current study investigates the sustainability of metacognitive prompting on self-regulatory behavior using a Process Mining approach. Previous studies confirmed beneficial short-term effects of metacognitive prompts on the learning process and on learning outcomes. However, the question of how stable these effects are for similar tasks in the future so far remains unanswered. Also, the use of online trace methods and the emergence of new analytical approaches allow deeper insights into the sequential structure of learning behavior. Therefore, we examined long-term effects of instructional support on sub-processes of self-regulated learning using Process Mining. Think-aloud protocols from 69 university students were measured during two hypermedia learning sessions about Educational Psychology. Metacognitive prompts supported the experimental group (n = 35)only during the first session. Based on a process model generated by using the data of the first learning task, we analysed the sustainability of effects during the second learning session. Results showed significant differences between the experimental and control group regarding the frequency of metacognitive strategies, which remain stable over time. Additionally, the application of Process Mining indicated which sequences of learning events were transferred to the second session. Our findings demonstrate the benefits of evaluating instructional support using analysis techniques that take into account the sequential structure of learning processes. While the results provide initial evidence for sustainable long-term effects on self-regulatory behavior, they have to be replicated in future research.

Examining university teachers’ self-regulation in using a learning analytics dashboard for online collaboration

Article

Full-text available

Aug 2023
Educ Inform Tech

Learning analytics dashboards (LADs) are often used to display real-time data indicating student learning trajectories and outcomes. Successful use of LADs requires teachers to orient their dashboard reviews with clear goals, apply appropriate strategies to interpret visualized information on LADs and monitor and evaluate their interpretations to meet goals. This process is known as self-regulated learning (SRL). Critical as it is, little research investigates teachers’ SRL in LAD usage. The present study addressed the gap by examining teachers’ SRL and sought to understand how teachers’ SRL relates to their use of LADs. To this end, a case study was designed in which ten participants were invited to use a LAD for asynchronous online problem-based learning. Think-aloud techniques and process mining methods were applied. The findings show that teachers were cognitive regulation in the early stage of LAD usage and became more metacognitive regulated later. The comparison of SRL between the good and the weak regulators indicates that the good self-regulators enacted more monitoring and evaluation events. Thus their regulator pattern was more non-linear. The qualitative analysis of think-aloud protocols reveals that teachers with good SRL are more likely to use the LAD to diagnose issues in student learning and collaboration. The study highlights the importance of SRL for teachers’ success in using LAD for data-driven instructions. The study also reinforces the importance of fostering teachers’ SRL, which accounts for teachers’ professional success in the digital era.

Investigating behavior patterns of students during online self-directed learning through process mining

Article

Full-text available

May 2023
Educ Inform Tech

One of the recognized ways to enhance teaching and learning is having insights into the behavior patterns of students. Studies that explore behavior patterns in online self-directed learning (OSDL) are scant though. In addition, the focus is lacking on how high-achieving (HA) students’ behavior patterns affect the academic performance of low-achieving (LA) students. To fill these research gaps, this study investigates (1) how the behavior patterns in OSDL vary between HA and LA students and (2) how HA students’ behavior patterns affect LA students’ academic performance. We used three perspectives of learning achievement, engagement, and cognitive load to examine academic performance. By utilizing process mining, we reviewed the log data of 71 college students on the Moodle platform and designed a pretest–posttest test without a control group. Results show obvious variances in the behavior patterns between HA and LA students. In particular, HA students performed more OSDL behaviors; their behavior patterns were more in line with self-directed logic. By contrast, LA students exhibited unmethodical behavior patterns; they were unable to process course content in depth. An instructional intervention was created with HA students’ behavior patterns as basis. The engagement of LA students increased, and their cognitive load was reduced after the instructional intervention. However, their learning achievement did not increase substantially. The interview results were consistent with the quantitative data. These findings indicate that the behavior patterns of HA students can shed light on how to guide the OSDL of LA students. This study also provides fresh methodological perspectives for assessing OSDL.

Multimodal learning analytics for assessing teachers’ self-regulated learning in planning technology-integrated lessons in a computer-based environment

Article

Full-text available

May 2023
Educ Inform Tech

Teachers’ self-regulated learning (SRL) plays a crucial role in developing technological pedagogical content knowledge (TPACK), a complex professional skill. It is crucial to identify teachers’ SRL activities that may lead to favorable TPACK. Previous studies have focused on the analysis of individual data sources from self-reported surveys or log files, which are insufficient to capture all SRL activities in the TPACK context. While multimodal learning analytics (MMLA) has the potential to improve SRL measurement, it remains unknown how multimodal data collected from different sources can be combined to identify salient features of SRL activities and examine how TPACK outcomes can be predicted by SRL activities identified from multimodal data. This study combined multimodal data from computer logs and think-aloud data to analyze teachers’ SRL activities in designing a technology-integrated lesson. We identified the salient features of SRL from the combined data and explored how identified SRL activities might predict TPACK outcomes reflected in teacher-generated lesson plans. The results of random forest regression analysis show that three SRL activities from the logs and two from the think-aloud data formed the best combination that explained a significant proportion of variances in TPACK performance. The impact of MMLA in SRL measurement and the implication of this study are discussed.

Using metacognitive prompts to enhance self‐regulated learning and learning outcomes: A meta‐analysis of experimental studies in computer‐based learning environments

Article

Full-text available

Feb 2022
J COMPUT ASSIST LEAR

Lin Guo

Background It has been assumed that prompting students to plan, monitor and evaluate their learning process could stimulate strategy use and thereby improve learning outcomes. Objectives This study aimed to examine the effects of metacognitive prompts on students' self‐regulated learning (SRL) and learning outcomes in the context of computer‐based learning environments (CBLEs). Methods To achieve this, the current study took a meta‐analytic approach to critically evaluate evidence for the effectiveness of metacognitive prompts and identify potential moderators of the effects. Results and conclusions With random‐effects models, the results showed that metacognitive prompts significantly enhanced SRL activities (g = 0.50, 95% confidence interval [0.37, 0.63]) and learning outcomes (g = 0.40, 95% confidence interval [0.31, 0.49]) relative to the control conditions. Furthermore, moderator analyses revealed that the effects varied as a function of three prompts features: feedback, specificity and adaptability. Implications Developing task‐specific, individual‐adaptive prompts and feedback should be a design principle in CBLEs, such that the prompt effect could be retained, sustainably enhanced and transferred to novel situations.

A Review of Hypermedia in Problem-Based Learning

Conference Paper

Nov 2020

Roger Chambers

The pedagogical practice of problem-based learning (PBL) has early ties to medical education. PBL was designed to give pre-clinical medical students hands-on experience in solving a real-world problem. The pedagogical model was eventually adopted by varying non-medical disciplines based on research indicating substantial experiential learning benefits evidenced by PBL learners. Hypermedia is the combination of text, audio, images, animation, and hypertext to create a multimedia-rich, web-based learning environment. Hypermedia has been regarded as complementary to PBL because of its nonlinear characteristics and potential to present real problems in an enriched format. This paper explores the relationship between PBL and hypermedia and reviews implementation considerations. This paper also considers the future direction of the practice of PBL with hypermedia. Effective PBL design will always center around case problems that must be relevant, open-ended, and multifaceted to provide an authentic PBL learning experience. As the interest in the implementation of PBL implemented with hypermedia continues, more research in more non-medical disciplines is necessary to continue to add empirical information regarding the pedagogical model.

Examining temporal dynamics of self-regulated learning behaviors in STEM learning: A network approach

Article

Aug 2020
COMPUT EDUC

From a network perspective, self-regulated learning (SRL) can be conceptualized as networks of mutually interacting self-regulatory learning behaviors. Nevertheless, the research on how SRL behaviors dynamically interact over time in a network architecture is still in its infancy, especially in the context of STEM (sciences, technology, engineering, and math) learning. In the present paper, we used a multilevel vector autoregression (VAR) model to examine the temporal dynamics of SRL behaviors as 101 students designed green buildings in Energy3D, a simulation-based computer-aided design (CAD) environment. We examined how different performance groups (i.e., unsuccessful, success-oriented, and mastery-oriented groups) differed in SRL competency, actual SRL behaviors, and SRL networks. We found that the three groups had no significant difference in their perceived SRL competency; however, they differed in SRL behaviors of evaluation. Both the mastery-oriented and success-oriented groups performed more evaluation behaviors than the unsuccessful group. Moreover, the mastery-oriented group showed stronger interaction between SRL behaviors than the success-oriented group and the unsuccessful group. The SRL networks of the three groups shared some similarities, but they were different from each other in general. This study has significant theoretical and methodological implications for the advancement of research in SRL dynamics.

Analyzing temporal data for understanding the learning process induced by metacognitive prompts

Article

Full-text available

Apr 2021
LEARN INSTR

Self-regulated learning is a central skill for students and beyond formal education. Empirical investigations have found that metacognitive prompts can support self-regulated learning and increase learning outcomes. However, the self-regulated learning process itself remains opaque. Educational process mining techniques offer a developing approach to investigate learning processes more deeply. In this study, we investigated the self-regulated learning processes of supported and unsupported learners. Learners with (n = 28) and without (n = 29) metacognitive prompts studied in a hypermedia learning environment while thinking aloud. The think aloud protocols were coded for self-regulation events, constituting the data used for process mining. While the learners with prompts showed more metacognitive events during the learning process, the process models for both conditions were similar. The analysis reveals the insights that process models can provide. However, since process-mining methods are newly applied in educational research, more research is needed to understand their explanatory power.

Using Epistemic Networks to Analyze Self-regulated Learning in an Open-Ended Problem-Solving Environment

Chapter

Jan 2021

The micro-level analyses of how students’ self-regulated learning (SRL) behaviors unfold over time provides a valuable framework for understanding their learning processes as they interact with computer-based learning environments. In this paper, we use log trace data to investigate how students self-regulate their learning in the Betty’s Brain environment, where they engage in three categories of open-ended problem-solving actions: information seeking, solution construction and solution assessment. We use Epistemic Network Analysis (ENA) to provide us with an overall understanding of the co-occurrences between action types both within and between the three action categories. Comparisons of epistemic networks generated for two groups of students, those with low and high performance, provided us with insights into their self-regulated behaviors.

Using epistemic networks to analyze self-regulated learn- ing in an open-ended problem-solving environment

Conference Paper

Full-text available

Nov 2020

The micro-level analyses of how students' self-regulated learning (SRL) behaviors unfold over time provides a valuable framework for understanding their learning processes as they interact with computer-based learning environments. In this paper, we use log trace data to investigate how students self-regulate their learning in the Betty's Brain environment, where they engage in three categories of open-ended problem-solving actions: information seeking, solution construction and solution assessment. We use Epistemic Network Analysis (ENA) to provide us with an overall understanding of the co-occurrences between action types both within and between the three action categories. Comparisons of epistemic networks generated for two groups of students, those with low and high performance, provided us with insights into their self-regulated behaviors.

Analyzing and fostering students' self-regulated learning through the use of peripheral data in online learning environments

Thesis

Full-text available

Jan 2020

Markus Hörmann

Learning with digital media has become a substantial part of formal and informal educational processes and is gaining more and more importance. Technological progress has brought overwhelming opportunities for learners, but challenges them at the same time. Learners have to regulate their learning process to a much greater extent than in traditional learning situations in which teachers support them through external regulation. This means that learners must plan their learning process themselves, apply appropriate learning strategies, monitor, control and evaluate it. These requirements are taken into account in various models of self-regulated learning (SRL). Although the roots of research on SRL go back to the 1980s, the measurement and adequate support of SRL in technology-enhanced learning environments is still not solved in a satisfactory way. An important obstacle are the data sources used to operationalize SRL processes. In order to support SRL in adaptive learning systems and to validate theoretical models, instruments are needed which meet the classical quality criteria and also fulfil additional requirements. Suitable data channels must be measurable "online", i.e., they must be available in real time during learning for analyses or the individual adaptation of interventions. Researchers no longer only have an interest in the final results of questionnaires or tasks, but also need to examine process data from interactions between learners and learning environments in order to advance the development of theories and interventions. In addition, data sources should not be obtrusive so that the learning process is not interrupted or disturbed. Measurements of physiological data, for example, require learners to wear measuring devices. Moreover, measurements should not be reactive. This means that other variables such as learning outcomes should not be influenced by the measurement. Different data sources that are already used to study and support SRL processes, such as protocols on thinking aloud, screen recording, eye tracking, log files, video observations or physiological sensors, meet these criteria to varying degrees. One data channel that has received little attention in research on educational psychology, but is non-obtrusive, non-reactive, objective and available online, is the detailed, timely high-resolution data on observable interactions of learners in online learning environments. This data channel is introduced in this thesis as "peripheral data". It records both the content of learning environments as context, and related actions of learners triggered by mouse and keyboard, as well as the reactions of learning environments, such as structural or content changes. Although the above criteria for the use of the data are met, it is unclear whether this data can be interpreted reliably and validly with regard to relevant variables and behavior. Therefore, the aim of this dissertation is to examine this data channel from the perspective of SRL and thus further close the existing research gap. One development project and four research projects were carried out and documented in this thesis.

Prompting Metacognition During a Job Search: Evidence from a Randomized Controlled Trial with University Job Seekers

Article

Full-text available

Apr 2020

Searching for a job is a self‐directed activity that requires self‐regulation over significant periods of time; yet, in the scientific community, we know little about proximal support interventions that shape the self‐regulation processes during a job search. By using an experimental design, we tested a proximal support intervention in the form of metacognitive prompts and how this shaped metacognition and effort in 123 university job seekers over a period of 9 weeks. Job seekers who were exposed to the intervention reported higher levels of metacognition, effort, and a greater number of job interviews compared to job seekers in a control condition. Additionally, the present study revealed that the prompting intervention, compared to the active control condition, resulted in higher levels of metacognition for job seekers who were less committed to finding a job than for job seekers who were more committed to finding a job. In summary, the current study makes important contributions to the job search literature and practice by fine‐tuning our understanding of the interventions that may support metacognition during the job search process and the individuals for whom these interventions may be most effective.

Capturing self‐regulated learning processes in virtual reality: Causal sequencing of multimodal data

Article

Full-text available

Sep 2023

This study examines the embodied ways in which learners monitor their cognition while learning about exponential functions in an immersive virtual reality (VR) based game, Pandemic by Prisms of Reality. Traditionally, metacognitive monitoring has been assessed through behavioural traces and verbalised instances. When learning in VR, learners are fully immersed in the learning environment, actively manipulating it based on affordances designed to support learning, offering insights into the relationship between physical interaction and metacognition. The study collected multimodal data from 15 participants, including think‐aloud audio, bird's‐eye view video recordings and physiological data. Metacognitive monitoring was analysed through qualitative coding of the think‐aloud protocol, while movement was measured via optical flow analysis and cognitive load was assessed through heart rate variability analysis. The results revealed embodied metacognition by aligning the data to identify learners' physical states alongside their verbalised metacognition. The findings demonstrated a temporal interplay among cognitive load, metacognitive monitoring, and motion during VR‐based learning. Specifically, cognitive load, indicated by the low‐ and high‐frequency heart rate variability index, predicted instances of metacognitive monitoring, and monitoring predicted learners' motion while interacting with the VR environment. This study further provides future directions in understanding self‐regulated learning processes during VR learning by utilizing multimodal data to inform real‐time adaptive personalised support within these environments. Practitioner notes What is already known about this topic Immersive virtual reality (VR) environments have the potential to offer personalised support based on users' individual needs and characteristics. Self‐regulated learning (SRL) involves learners monitoring their progress and strategically regulating their learning when needed. Multimodal data captured during VR learning, such as birds‐eye‐view video, screen recordings, physiological changes and verbalisations, can provide insights into learners' SRL processes and support needs. What this paper adds Provides insights into the embodied aspects of learners' metacognitive monitoring during learning in an immersive VR environment. Demonstrates how SRL processes can be captured via the collection and analysis of multimodal data, including think‐aloud audio, bird's‐eye view video recordings and physiological data, to capture metacognitive monitoring and movement during VR‐based learning. Contributes to the understanding of the interplay between cognitive load, metacognitive monitoring, and motion in immersive VR learning. Implications for practice and/or policy Researchers and practitioners can use the causal relationships identified in this study to identify instances of SRL in an immersive VR setting. Educational technology developers can consider the integration of online measures, such as cognitive load and physiological arousal, into adaptive VR environments to enable real‐time personalised support for learners based on their self‐regulatory needs.

Analytics of self-regulated learning scaffolding: effects on learning processes

Article

Full-text available

Aug 2023

Self-regulated learning (SRL) is the ability to regulate cognitive, metacognitive, motivational, and emotional states while learning and is posited to be a strong predictor of academic success. It is therefore important to provide learners with effective instructions to promote more meaningful and effective SRL processes. One way to implement SRL instructions is through providing real-time SRL scaffolding while learners engage with a task. However, previous studies have tended to focus on fixed scaffolding rather than adaptive scaffolding that is tailored to student actions. Studies that have investigated adaptive scaffolding have not adequately distinguished between the effects of adaptive and fixed scaffolding compared to a control condition. Moreover, previous studies have tended to investigate the effects of scaffolding at the task level rather than shorter time segments—obscuring the impact of individual scaffolds on SRL processes. To address these gaps, we (a) collected trace data about student activities while working on a multi-source writing task and (b) analyzed these data using a cutting-edge learning analytic technique— ordered network analysis (ONA)—to model, visualize, and explain how learners' SRL processes changed in relation to the scaffolds. At the task level, our results suggest that learners who received adaptive scaffolding have significantly different patterns of SRL processes compared to the fixed scaffolding and control conditions. While not significantly different, our results at the task segment level suggest that adaptive scaffolding is associated with earlier engagement in SRL processes. At both the task level and task segment level, those who received adaptive scaffolding, compared to the other conditions, exhibited more task-guided learning processes such as referring to task instructions and rubrics in relation to their reading and writing. This study not only deepens our understanding of the effects of scaffolding at different levels of analysis but also demonstrates the use of a contemporary learning analytic technique for evaluating the effects of different kinds of scaffolding on learners' SRL processes.

i-Ntervene: applying an evidence-based learning analytics intervention to support computer programming instruction

Article

Full-text available

Jul 2023

Apart from good instructional design and delivery, effective intervention is another key to strengthen student academic performance. However, intervention has been recognized as a great challenge. Most instructors struggle to identify at-risk students, determine a proper intervention approach, trace and evaluate whether the intervention works. This process requires extensive effort and commitment, which is impractical especially for large classes with few instructors. This paper proposes a platform, namely i-Ntervene, that integrates Learning Management System (LMS) automatic code grader, and learning analytics features which can empower systematic learning intervention for large programming classes. The platform supports instructor-pace courses on both Virtual Learning Environment (VLE) and traditional classroom setting. The platform iteratively assesses student engagement levels through learning activity gaps. It also analyzes subject understanding from programming question practices to identify at-risk students and suggests aspects of intervention based on their lagging in these areas. Students’ post-intervention data are traced and evaluated quantitatively to determine effective intervention approaches. This evaluation method aligns with the evidence-based research design. The developed i-Ntervene prototype was tested on a Java programming course with 253 first-year university students during the Covid-19 pandemic in VLE. The result was satisfactory, as the instructors were able to perform and evaluate 12 interventions throughout a semester. For this experimental course, the platform revealed that the approach of sending extrinsic motivation emails had more impact in promoting learning behavior compared to other types of messages. It also showed that providing tutorial sessions was not an effective approach to improving students’ subject understanding in complex algorithmic topics. i-Ntervene allows instructors to flexibly trial potential interventions to discover the optimal approach for their course settings which should boost student’s learning outcomes in long term.

The Impact of Prompts and Feedback on the Performance during Multi-Session Self-Regulated Learning in the Hypermedia Environment

Article

Full-text available

Jul 2023

The hypermedia environment is among the most prevalent contemporary self-regulated learning (SRL) environments; however, methods for improving the effectiveness of students’ multi-session SRL in such environments remain under discussion. In this study, two experiments were conducted to explore whether and how prompts and feedback benefit performance during multi-session SRL in a hypermedia learning environment. A total of 76 senior students participated in Experiment 1, which used a mixed 2 (prompting condition: prompt, no prompt) × 2 (feedback condition: feedback, no feedback) × 2 (learning session: Session 1 and Session 2) design to explore the effects of prompting and feedback on the multi-session learning process in a hypermedia environment. The results indicated that, in learning Session 1, performance in the prompt condition was significantly better than in the unprompted condition, with or without feedback; in learning Session 2, participants in the prompt condition with feedback performed significantly better than those in the other three conditions. Students in the group with a prompt and feedback had the most accurate meta-comprehension absolute accuracy in both learning sessions. Experiment 2 recruited 94 secondary school students to further explore whether the combination of prompts and different types of feedback led to different learning outcomes according to the division of feedback timing. A mixed 2 (prompt condition: prompt, no prompt) × 3 (feedback condition: delayed feedback, immediate feedback, no feedback) × 2 (learning session: Session 1 and Session 2) design was used. The results indicated that, in learning Session 1, the prompt condition outperformed the unprompted condition with or without feedback; in learning Session 2, students with prompted delayed feedback outperformed the other five conditions. We also found that although there was no significant difference in meta-comprehension monitoring accuracy between delayed and immediate feedback, both groups performed significantly better than those in the no feedback condition. These results suggest that the combination of prompts and feedback in hypermedia environments facilitates student performance better than prompts or feedback alone; this improvement may be related to the correction of poor internal student feedback.

Theories of metacognition and pedagogy applied to AIED systems

Chapter

Apr 2023

Analysis of Differences in Self-Regulated Learning Behavior Patterns of Online Learners

Article

Full-text available

Dec 2022

Self-regulated learning is one of the important skills to achieve learning goals and is also the key factor to ensure the quality of online learning. With the rapid development of intelligent education and information technology, online learning behavior has become a new trend in the development of education modernization. Behavior data of online learning platforms are an important carrier to reflect the learners’ initiative to plan, monitor, and regulate their learning process. Self-regulated learning (SRL) is one of the important skills to achieve learning goals and is an essential means to ensure the quality of online learning. However, there are still great challenges in studying the types and sequential patterns of learners’ self-regulated learning behaviors in online environments. In addition, for higher education, the defects of the traditional education mode are increasingly prominent, and self-regulated learning (SRL) has become an inevitable trend. Based on Zimmerman’s self-regulation theory model, this paper first classifies learning groups using the hierarchical clustering method. Then, lag sequence analysis is used to explore the most significant differences in SRL behavior and its sequence patterns among different learning groups. Finally, the differences in academic achievement among different groups are discussed. The results are as follows: (1) The group with more average behavior frequency tends to solve online tasks actively, presenting a “cognitive oriented” sequential pattern, and this group has the best performance; (2) the group with more active behavior frequency tends to improve in the process of trial and error, showing a “reflective oriented” sequence pattern, and this group has better performance; (3) the group with the lowest behavior frequency tends to passively complete the learning task, showing a “negative regulated” sequence pattern, and this group has poor performance. From the aspects of stage and outcome of self-regulated learning, the behavior sequence and learning performance of online learning behavior mode are compared, and the learning path and learning performance of different learning modes are fully analyzed, which can provide reference for the improvement of online learning platform and teachers’ teaching intervention.

Time in various phases of self-regulation and problem-solving performance in an Intelligent Tutoring System

Article

Full-text available

Nov 2022
Educ Inform Tech

The present paper builds on the literature that emphasizes the importance of self-regulation for academic learning or self-regulated learning (SRL). SRL research has traditionally focused on count measures of SRL processing events, however, another important measure of SRL is being recognized: time-on-task. The current study captures the influence of time spent on learning performance. We study time-on-task—from the perspective of self-regulated learning—in the context of clinical reasoning in an intelligent tutoring system. Specifically, we examine the link between the time spent in the three phases of SRL (forethought, performance, and self-reflection) and confidence in diagnosis and diagnosis correctness. Our analyses revealed non-significant links between the time spent in the three phases of SRL and diagnosis correctness. On the other hand, significant associations were found between the time spent in the three phases of SRL and confidence in diagnosis (confidence in diagnosis was: positively associated with time spent in forethought phase; negatively associated with time spent in performance phase; and, positively associated with time spent in self-reflection phase). In addition, confidence in diagnosis was positively linked to diagnosis correctness. Considering learning time offers an alternative perspective on regulation of learning and problem-solving performance. We conclude by offering implications of our findings and recommendations for further research.

Understanding the Development of Self-Regulated Learning: An Intervention Study to Promote Self-Regulated Learning in Vocational Schools

Article

Full-text available

Sep 2022

Self-regulated learning (SRL) provides the foundation for building sustainable knowledge and is therefore important for schools, classrooms, and lifelong learning in general. Especially in vocational education and training, the concept of SRL remains fundamental as it relates to preparing future employees. However, further research is needed on how vocational students situationally regulate their learning process and the extent to which this may be related to a dispositional change in their SRL. In this study, we analyzed longitudinal questionnaire data from 159 students who attended either SRL-conducive or regular vocational classes. We refer to Perry and colleagues' (2018) framework of an SRL-conducive learning environment, which focuses on (meta)cognitive, motivational, and emotional aspects of learning. Using multilevel analysis, we found differences in the development of (meta)cognitive components of learning, whereas no clear differences could be identified for motivational and emotional components. The results support the assumption that process analyses can be used to draw a more differentiated picture of SRL in vocational schools. Moreover, indirect approaches to promoting SRL should be designed to include all SRL-relevant aspects.

Using HeuristicsMiner to Analyze Problem-Solving Processes: Exemplary Use Case of a Productive-Failure Study

Article

Full-text available

Aug 2022

This paper presents a fine-grained process analysis of 22 students in a classroom-based learning setting. The students engaged (and failed) in problem-solving attempts prior to instruction (i.e., the Productive-Failure approach). We used the HeuristicsMiner algorithm to analyze the data of a quasi-experimental study. The applied algorithm allowed us to investigate temporally structured think-aloud data, to outline productive and unproductive problem-solving strategies. Our analyses and findings demonstrated that HeuristicsMiner enables researchers to effectively mine problem-solving processes and sequences, even for smaller sample sizes, which cannot be done with traditional code-and-count strategies. The limitations of the algorithm, as well as further implications for educational research and practice, are also discussed.

Supporting self-regulated learning with learning analytics interventions – a systematic literature review

Article

Full-text available

Sep 2022
Educ Inform Tech

During the past years scholars have shown an increasing interest in supporting students' self-regulated learning (SRL). Learning analytics (LA) can be applied in various ways to identify a learner’s current state of self-regulation and support SRL processes. It is important to examine how LA has been used to identify the need for support in different phases of SRL cycle, which channels are used to mediate the intervention and how efficient and impactful the intervention is. This will help the learners to achieve the anticipated learning outcomes. The systematic literature review followed PRISMA 2020 statement to examine studies that applied LA interventions to enhance SRL. The search terms used for this research identified 753 papers in May 2021. Of these, 56 studies included the elements of LA, SRL, and intervention. The reviewed studies contained various LA interventions aimed at supporting SRL, but only 46% of them revealed a positive impact of an intervention on learning. Furthermore, only four studies reported positive effects for SRL and covered all three SRL phases (planning, performance, and reflection). Based on the findings of this literature review, the key recommendation is for all phases of SRL to be considered when planning interventions to support learning. In addition, more comparative research on this topic is needed to identify the most effective interventions and to provide further evidence on the effectiveness of interventions supporting SRL.

Lessons Learned and Future Directions of MetaTutor: Leveraging Multichannel Data to Scaffold Self-Regulated Learning With an Intelligent Tutoring System

Article

Full-text available

Jun 2022

Self-regulated learning (SRL) is critical for learning across tasks, domains, and contexts. Despite its importance, research shows that not all learners are equally skilled at accurately and dynamically monitoring and regulating their self-regulatory processes. Therefore, learning technologies, such as intelligent tutoring systems (ITSs), have been designed to measure and foster SRL. This paper presents an overview of over 10 years of research on SRL with MetaTutor, a hypermedia-based ITS designed to scaffold college students’ SRL while they learn about the human circulatory system. MetaTutor’s architecture and instructional features are designed based on models of SRL, empirical evidence on human and computerized tutoring principles of multimedia learning, Artificial Intelligence (AI) in educational systems for metacognition and SRL, and research on SRL from our team and that of other researchers. We present MetaTutor followed by a synthesis of key research findings on the effectiveness of various versions of the system (e.g., adaptive scaffolding vs. no scaffolding of self-regulatory behavior) on learning outcomes. First, we focus on findings from self-reports, learning outcomes, and multimodal data (e.g., log files, eye tracking, facial expressions of emotion, screen recordings) and their contributions to our understanding of SRL with an ITS. Second, we elaborate on the role of embedded pedagogical agents (PAs) as external regulators designed to scaffold learners’ cognitive and metacognitive SRL strategy use. Third, we highlight and elaborate on the contributions of multimodal data in measuring and understanding the role of cognitive, affective, metacognitive, and motivational (CAMM) processes. Additionally, we unpack some of the challenges these data pose for designing real-time instructional interventions that scaffold SRL. Fourth, we present existing theoretical, methodological, and analytical challenges and briefly discuss lessons learned and open challenges.

Reviewing Process Mining Applications and Techniques in Education

Article

Full-text available

Jan 2022

Process Mining (PM) emerged from business process management but has recently been applied to educational data and has been found to facilitate the understanding of the educational process. Educational Process Mining (EPM) bridges the gap between process analysis and data analysis, based on the techniques of model discovery, conformance checking and extension of existing process models. We present a systematic review of the recent and current status of research in the EPM domain, focusing on application domains, techniques, tools and models, to highlight the use of EPM in comprehending and improving educational processes.

Patterns of adaptive regulation in collaborative learning : a multimodal methodological approach

Thesis

Full-text available

Jun 2021

Márta Sobocinski

This dissertation explores adaptive regulation in collaborative learning by using novel technologies for data collection and analysis. Adaptation is a key feature of regulated learning, because through adaptation learners change their ways of thinking and learning when faced with challenges. However, because of methodological limitations, studying adaptation has been challenging. This dissertation focuses on exploring the differences between high- and low-challenge sessions in terms of phases of regulation, how metacognitive monitoring triggers adaptation, and how sequences of adaptive regulation and maladaptive behavior emerge throughout a learning session. To capture adaptation in group learning situations, two data sets were collected in authentic learning situations. The first data collection took place during a mathematics didactics course organized for teacher education students; the second data collection was conducted during an advanced physics course for high-school students. The data collected included log, video, and heart rate data. Process-oriented methods were used to combine qualitative analysis of the video data and group-level analysis of changes in heart rate values. The results indicate that in high-challenge sessions learners return to planning throughout the session, which can be interpreted as a sign of adaptation. The results also show that monitoring acts as a trigger for adaptation and report how group-level small-scale adaptation can be evidenced by considering the phase, target, and valence of shared monitoring events. Physiological state transitions defined from the heart rate data have potential to reveal information about whether the group is on track in the learning process. The findings provide insight into how adaptation happens in an authentic collaborative learning setting. Methodologically, the study provides an innovative solution for capturing adaptation using multimodal data and novel analytical methods. At the theoretical level, the dissertation contributes to the field with details of the relationship between metacognitive monitoring and small-scale adaptation. For pedagogical practice, this study signals a need for adaptive support during the learning process: collaborative groups take very different routes to success in terms of when and how they adapt their learning process.

Measuring Students' Self-Regulatory Phases in LMS with Behavior and Real-Time Self Report

Conference Paper

Full-text available

Apr 2021

Research has emphasized that self-regulated learning (SRL) is critically important for learning. However, students have different capabilities of regulating their learning processes and individual needs. To help students improve their SRL capabilities, we need to identify students’ current behaviors. Specifically, we applied instructional design to create visible and meaningful markers of student learning at different points in time in LMS logs. We adopted knowledge engineering to develop a framework of proximal indicators representing SRL phases and evaluated them in a quasi-experiment in two different learning activities. A comparison of two sources of collected students’ SRL data, self-reported and trace data, revealed a relatively high agreement between our classifications (weighted kappa, 𝜅 = .74 and 𝜅 = .68). However, our indicators did not always discriminate adjacent SRL phases, particularly for enactment and adapting phases, compared with students’ real-time self-reported behaviors. Our behavioral indicators also were comparably successful at classifying SRL phases for different self-regulatory engagement levels. This study demonstrated how the triangulation of various sources of students’ self-regulatory data could help to unravel the complex nature of metacognitive processes.

Han, F., & Ellis, R. A. (2020). Combining self-reported and observational measures to assess university student academic performance in blended course designs. Australasian Journal of Educational Technology, 36(6), 1-14. https://doi.org/10.14742/ajet.6369

Article

Full-text available

Dec 2020

This study combined the methods from student approaches to learning and learning analytics research by using both self-reported and observational measures to examine the student learning experience. It investigated the extent to which reported approaches and perceptions and observed online interactions are related to each other and how they contribute to variation in academic performance in a blended course design. Correlation analyses showed significant pairwise associations between approaches and frequency of the online interaction. A cluster analysis identified two groupings of students with different reported learning orientations. Based on the reported learning orientations, one-way ANOVAs showed that students with understanding orientation reported deep approaches to and positive perceptions of learning. The students with understanding orientation also interacted more frequently with the online learning tasks and had higher marks than those with reproducing orientation, who reported surface approaches and negative perceptions. Regression analyses found that adding the observational measures increased 36% of the variance in the academic performance in comparison with using self-reported measures alone (6%). The findings suggest using the combined methods to explain students’ academic performance in blended course designs not only triangulates the results but also strengthens the acuity of the analysis. © 2020. Articles published in the Australasian Journal of Educational Technology (AJET) are available under Creative Commons Attribution Non-Commercial No Derivatives Licence (CC BYNC- ND 4.0). Authors retain copyright in their work and grant AJET right of first publication under CC BY-NC-ND 4.0. All Rights Reserved.

Multimodal Data Fusion in Learning Analytics: A Systematic Review

Article

Full-text available

Nov 2020
SENSORS-BASEL

Multimodal learning analytics (MMLA), which has become increasingly popular, can help provide an accurate understanding of learning processes. However, it is still unclear how multimodal data is integrated into MMLA. By following the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines, this paper systematically surveys 346 articles on MMLA published during the past three years. For this purpose, we first present a conceptual model for reviewing these articles from three dimensions: data types, learning indicators, and data fusion. Based on this model, we then answer the following questions: 1. What types of data and learning indicators are used in MMLA, together with their relationships; and 2. What are the classifications of the data fusion methods in MMLA. Finally, we point out the key stages in data fusion and the future research direction in MMLA. Our main findings from this review are (a) The data in MMLA are classified into digital data, physical data, physiological data, psychometric data, and environment data; (b) The learning indicators are behavior, cognition, emotion, collaboration, and engagement; (c) The relationships between multimodal data and learning indicators are one-to-one, one-to-any, and many-to-one. The complex relationships between multimodal data and learning indicators are the key for data fusion; (d) The main data fusion methods in MMLA are many-to-one, many-to-many and multiple validations among multimodal data; and (e) Multimodal data fusion can be characterized by the multimodality of data, multi-dimension of indicators, and diversity of methods.

An Interactive Layers Model of Self- Regulated Learning and Cognitive Load

Article

Full-text available

Dec 2020
EDUC PSYCHOL REV

Models of self-regulated learning emphasize the active and intentional role of learners and, thereby, focus mainly on conscious processes in working memory and long-term memory. Cognitive load theory supports this view on learning. As a result, both fields of research ignore the potential role of unconscious processes for learning. In this review paper, we propose an interactive layers model on self-regulated learning and cognitive load that considers sensory memory, working memory, and long-term memory. The model distinguishes between (a) unconscious self-regulated learning initiated by so-called resonant states in sensory memory and (b) conscious self-regulated learning of scheme construction in working memory. In contrast with conscious self-regulation, unconscious self-regulation induces no cognitive load. The model describes conscious and unconscious self-regulation in three different layers: a content layer, a learning strategy layer, and a metacognitive layer. Interactions of the three layers reflect processes of monitoring and control. We first substantiate the model based on a narrative review. Afterwards, we illustrate how the model contributes to re-interpretation of inconsistent empirical findings reported in the existing literature.

What college students say, and what they do: Aligning self-regulated learning theory with behavioral logs

Conference Paper

Mar 2020

A central concern in learning analytics specifically and educational research more generally is the alignment of robust, coherent measures to well-developed conceptual and theoretical frameworks. Capturing and representing processes of learning remains an ongoing challenge in all areas of educational inquiry and presents substantive considerations on the nature of learning, knowledge, and assessment & measurement that have been continuously refined in various areas of education and pedagogical practice. Learning analytics as a still developing method of inquiry has yet to substantively navigate the alignment of measurement, capture, and representation of learning to theoretical frameworks despite being used to identify various practical concerns such as at risk students. This study seeks to address these concerns by comparing behavioral measurements from learning management systems to established measurements of components of learning as understood through self-regulated learning frameworks. Using several prominent and robustly supported self-reported survey measures designed to identify dimensions of self-regulated learning, as well as typical behavioral features extracted from a learning management system, we conducted descriptive and exploratory analyses on the relational structures of these data. With the exception of learners' self-reported time management strategies and level of motivation, the current results indicate that behavioral measures were not well correlated with survey measurements. Possibilities and recommendations for learning analytics as measurements for self-regulated learning are discussed.

Enhancing Classroom Learning Outcomes: The Power Of Immediate Feedback Strategy

Article

Full-text available

Oct 2023

The study investigates the effects of using immediate feedback as a teaching strategy. Additionally, the impacts of immediate feedback and gender on students' classroom learning outcomes were examined. The study was designed as a quasi-experimental, pretest-posttest, experimental, and control group. The sample consisted of 225 junior secondary school level 2 (JSS 2) students. Students were chosen from two intact coeducational classes and split into experimental and control groups. All relevant data was collected using a study tool called the Science, Technology, English Language & Mathematics Achievement Test-Questionnaire (STEMAT-Q), which was developed, validated, and used. Data collected were analysed using the mean, standard deviation, Student's t-test, and analysis of covariance (ANCOVA). The study's findings demonstrate that an immediate feedback technique significantly affects the learning outcomes of students. However, for treated male and female students, the interaction effects of the immediate feedback technique and gender on classroom learning outcomes were not significant. Immediate feedback is particularly successful at addressing student confusion, correcting errors, identifying learning gaps, bridging gender differences in student learning outcomes, and inspiring students to learn well. Based on the above findings, the researcher recommends the provision of immediate feedback for students during the learning process or class discussion or activities to enhance their learning skills and help them retain key concepts, ideas, and principles.

Measuring and Validating Assumptions About Self-Regulated Learning with Multimodal Data

Chapter

Jun 2023

Individuals who engage in self-regulated learning (SRL) tend to perform better in complex learning tasks. However, learners’ ability to self-regulate can vary. To understand and support learners’ SRL, collecting information about their engagement in specific learning processes in the context of learning tasks is necessary. However, SRL is sufficiently complex that it is not directly observable. Capturing the SRL processes that occur during learning, as students interact with elements of tasks hosted on virtual learning technologies (e.g., learning management systems; LMS), is possible because learners’ actions generate observable events that these technologies log. However, discerning how these events reflect SRL processes poses several major theoretical, methodical, and analytical challenges. To address these challenges, we present two projects to illustrate how researchers validated inferences about SRL processes. We demonstrate how observational indicators drawn from multiple channels of event data must be (a) collected from the technologies’ log files and the record of learners’ self-reports of their learning process, (b) instrumented to describe learner, event, and context, and (c) integrated and temporally aligned. Afterward, we how researchers can hypothesize about the SRL processes digital events reflect and test inferences using secondary channels of explanatory data provided by learners during the tasks.KeywordsSelf-regulated learningMultimodal dataMeasurement validation

Regulating Response Speed Promotes Associative Learning

Article

May 2023

Casadevante et al. (Curr Psychol 42: 4272–4285, 2023) used an objective test and found that regulation of response speed was related to better performance in a category learning task. The present study aims at analysing whether the relation between regulation of response speed and learning exists in an associative learning task. We developed ad hoc the Treasure Forest, an objective test consisting of a computerized associative learning task. We conducted a first study with 86 university students to assess the relation between spontaneous response speed and learning. Results showed that participants who acted slowly learned more than their mates who acted faster (t (83) = 8.898, p < .001, η2 = .672). Moreover, some students who began the task acting too fast to learn decreased their response speed by the second half of the task and simultaneously their learning index improved (t (11) = 2.325, p < .05, d = .721). Hence, self-regulating the response speed was linked to associative learning. We conducted a second study to analyse the influence of an external speed regulation on learning. The intervention group (N = 99) was prevented from clicking more than one click per second while the control group (N = 85) acted without restrictions. The intervention group achieved a higher learning index than the control group, who acted faster (t (160) = 4.828, p < .001, η2 =.117). Hence, regulating response speed promoted associative learning. We concluded that regulating response speed promoted associative learning, and we hypothesized that training self-regulation of response speed may improve learning and academic performance. Besides, we highlight the utility of employing objective test for analysing self-regulation.

Investigating learning processes through analysis of navigation behavior using log files

Article

Full-text available

Apr 2023
J Comput High Educ

The empirical study investigates what log files and process mining can contribute to promoting successful learning. We want to show how monitoring and evaluation of learning processes can be implemented in the educational life by analyzing log files and navigation behavior. Thus, we questioned to what extent log file analyses and process mining can predict learning outcomes. This work aims to provide support for learners and instructors regarding efficient learning with computer-based learning environments (CBLEs). We evaluated log file and questionnaire data from students (N = 58) who used a CBLE for two weeks. Results show a significant learning increase after studying with the CBLE with a very high effect size (p < .001, g = 1.71). A cluster analysis revealed two groups with significantly different learning outcomes accompanied by different navigation patterns. The time spent on learning-relevant pages and the interactivity with a CBLE are meaningful indicators for Recall and Transfer performance. Our results show that navigation behaviors indicate both beneficial and detrimental learning processes. Moreover, we could demonstrate that navigation behaviors impact the learning outcome. We present an easy-to-use approach for learners as well as instructors to promote successful learning by tracking the duration spent in a CBLE and the interactivity.

How Learning Process Data Can Inform Regulation in Collaborative Learning Practice

Chapter

Mar 2023

We have been working to understand when, how, and what makes regulation in collaborative learning functional aiming to understand the process of collaboration so that we could better inform learners and teachers in practice. Multimodal learning data collection and learning process analytics have guided our work. Seamless and accurate integration of multimodal learning data for measuring regulation in collaborative learning can be seen as a future direction of multimodal learning analytics (MMLA). Collaborative groups can be considered complex systems. The cognitive, emotional, motivational, and behavioural states of the group and its’ members are related to each other and in constant flux. Therefore, research has shown that regulation is a crucial process for making the maladaptive process of collaborative learning more adaptive. Regulation in collaborative learning involves groups taking metacognitive control of the task together through negotiated, iterative fine-tuning of internal and external conditions as needed. Metacognitive monitoring is always an internal mental process, but it can be externalized via visible interactions with other group members in collaborative situations. When aiming to support the adaptive collaborative learning process, we highlight two aspects: metacognitive awareness and participation in cognitive and socio-emotional interaction. In this chapter we present our recent empirical progress in these two aspects and discuss how learning process data and MMLA can be used to unravel regulation in collaborative learning and practical implications for collaborative learning.KeywordsSelf-regulated learningSSRLMetacognitionCollaborative learningMultimodal methods

The impact of scripts on blended and online socially shared regulation of learning: A role-playing game theory perspective

Article

Dec 2022

Self-regulated learning theory is central to computer supported collaborative learning (CSCL) and depends on learner autonomy to create socially shared learning, and yet function within the restraints and goals of a specific class pedagogy. By integrating the rich theoretical CSCL literature with an inductively derived theory of role-playing game practice, we develop an insightful foundation for designing, implementing, and measuring the effectiveness of low-cost scripts. This takes the form of a prompt (mere exposure prompt) that nudges learners toward a pedagogical goal while maintaining freedom of learner creativity and minimizing instructor intrusion. We assert learner engagement can be viewed through the lens of role-playing’s emphasis on aligning players’ creative agendas with game design to create a shared imagined space. Through behavioral trace data and social network analysis, we measure behaviors that differ between test/control groups, receiving the prompt, and comparing a fully online versus blended course delivery over a semester of group-based simulated business negations following role-playing game design principles. Fully online test group members accurately recall the prompt’s messages while exhibiting behaviors congruent with the pedagogical script. Learners in the blended mode recall the prompt, but their behavior is unchanged. This suggests socially shared regulation of learning in the classroom context conforms to established classroom norms, overlooking the script prompt. Learners in the fully online mode, in contrast, initiate fewer social interactions, but search out opportunities across many players, thereby demonstrating the effect of the script prompt message.

Analyzing the research grant process in Iran's National Elites Foundation: An approach based on process mining and machine learning

Article

Full-text available

Jun 2022

Analyzing the event logs extracted from the process-aware information systems provide critical insights into improving the organizational processes. This case study reports the essential findings and lessons from a process mining project run in analyzing the postdoctoral research grant process in Iran's National Elites Foundation (INEF). Different deductions are reached by exploring the process participants' activities in the INEF web portal containing (i) the organizational inefficiencies exposed through the process mining techniques, where the most time-consuming activities are detected and suggested to the domain experts, and (ii) the decision tree technique applied in determining how the successful applicants are scored. The extracted rules indicate an 18% application admission with a final score of more than 403. This article is contributed to interpreting the behavioral patterns in INEF and determining who among the applicants has a higher chance of receiving the grant, supporting the policymakers and managers to assign rational budgeting and adopting human resource strategies.

Improving the measurement of self-regulated learning using multi-channel data

Article

Full-text available

Jun 2022

In recent years, unobtrusive measures of self-regulated learning (SRL) processes based on log data recorded by digital learning environments have attracted increasing attention. However, researchers have also recognised that simple navigational log data or time spent on pages are often not fine-grained enough to study complex SRL processes. Recent advances in data-capturing technologies enabled researchers to go beyond simple navigational logs to measure SRL processes with multi-channel data. What multi-channel data can reveal about SRL processes, and to what extent can the addition of peripheral and eye-tracking data with navigational log data change and improve the measurement of SRL are key questions that require further investigation. Hence, we conducted a study and collected learning trace data generated by 25 university students in a laboratory setting, that aimed to address this problem by enhancing navigational log data with peripheral and eye-tracking data. We developed a trace-based measurement protocol of SRL, which interpreted raw trace data from multi-channel data into SRL processes. Specifically, the study compared the frequency and duration of SRL processes detected, how much duration and times of occurrences of the detected SRL processes were affected or refined. We also used a process mining technique to analyses how temporal sequencing of the detected SRL processes changed by enriching navigational log data with peripheral and eye-tracking data. The results revealed that by adding new data channels, we improved the capture of learning actions and detected SRL processes while enhancing the granularity of the measurement. In comparison to the use of navigational logs only, the completeness of temporal sequencing relationships between SRL processes with multi-channel data improved. In addition, we concluded that eye-tracking data is valuable for measuring and extracting SRL processes, and it should receive more attention in the future.

Exploring the link between self‐regulated learning and learner behaviour in a massive open online course

Article

Full-text available

Apr 2022
J COMPUT ASSIST LEAR

Background Learners in Massive Open Online Courses (MOOCs) are presented with great autonomy over their learning process. Learners must engage in self‐regulated learning (SRL) to handle this autonomy. It is assumed that learners' SRL, through monitoring and control, influences learners' behaviour within the MOOC environment (e.g., watching videos). The exact relationship between SRL and learner behaviour has however not been investigated. Objectives We explored whether differences in SRL are related to differences in learner behaviour in a MOOC. As insight in this relationship could improve our understanding of the influence of SRL on behaviour, could help explain the variety in online learner behaviour, and could be useful for the development of successful SRL support for learners. Methods MOOC learners were grouped based on their self‐reported SRL. Next, we used process mining to create process models of learners' activities. These process models were compared between groups of learners. Results and conclusions Four clusters emerged: average regulators, help seekers, self‐regulators, and weak regulators. Learners in all clusters closely followed the designed course structure. However, the process models also showed differences which could be linked to differences in the SRL scores between clusters. Takeaways The study shows that SRL may explain part of the variability in online learner behaviour. Implications for the design of SRL interventions include the necessity to integrate support for weak regulators in the course structure.

Tracking Changes in Students’ Online Self-Regulated Learning Behaviors and Achievement Goals Using Trace Clustering and Process Mining

Article

Full-text available

Mar 2022

Success in online and blended courses requires engaging in self-regulated learning (SRL), especially for challenging STEM disciplines, such as physics. This involves students planning how they will navigate course assignments and activities, setting goals for completion, monitoring their progress and content understanding, and reflecting on how they completed each assignment. Based on Winne & Hadwin’s COPES model, SRL is a series of events that temporally unfold during learning, impacted by changing internal and external factors, such as goal orientation and content difficulty. Thus, as goal orientation and content difficulty change throughout a course, so might students’ use of SRL processes. This paper studies how students’ SRL behavior and achievement goal orientation change over time in a large (N = 250) college introductory level physics course taught online. Students’ achievement goal orientation was measured by repeated administration of the achievement goals questionnaire-revised (AGQ-R). Students’ SRL behavior was measured by analyzing their clickstream event traces interacting with online learning modules via a combination of trace clustering and process mining. Event traces were first divided into groups similar in nature using agglomerative clustering, with similarity between traces determined based on a set of derived characteristics most reflective of students’ SRL processes. We then generated causal nets for each cluster of traces via process mining and interpreted the underlying behavior and strategy of each causal net according to the COPES SRL framework. We then measured the frequency at which students adopted each causal net and assessed whether the adoption of different causal nets was associated with responses to the AGQ-R. By repeating the analysis for three sets of online learning modules assigned at the beginning, middle, and end of the semester, we examined how the frequency of each causal net changed over time, and how the change correlated with changes to the AGQ-R responses. Results have implications for measuring the temporal nature of SRL during online learning, as well as the factors impacting the use of SRL processes in an online physics course. Results also provide guidance for developing online instructional materials that foster effective SRL for students with different motivational profiles.

Analyzing and Interpreting Students’ Self-regulated Learning Patterns Combining Time-series Feature Extraction, Segmentation, and Clustering

Article

Jan 2022

In online learning, students’ learning behavior might change as the course progresses. How students adjust learning behaviors aligned with course requirements reflects their self-regulated learning strategies. Analyzing students’ learning patterns can help instructors understand how the course design or activities shape students’ learning behaviors, including their learning beliefs and motivation, and facilitate teaching decision makings accordingly. This study aims to propose a scientific analytic method to understand students’ self-regulated learning (SRL) patterns. The whole process includes the following four steps: (1) encoding behavioral patterns; (2) detecting turning points and chunking behavioral patterns; (3) grouping similar patterns; and (4) interpreting results. A case study with 4604 K-12 students from 476 courses was conducted to validate the proposed method. Five successful patterns, three at-risk patterns, and three average patterns were identified. The case study indicated that successful students showed at least one of the following characteristics: (1) Balanced, (2) Proactive and Balanced, and (3) Balanced with one highly engaged behavior. The at-risk students showed the following characteristics: (1) Oscillatory and (2) Low Engaged. Patterns which led to successful or at-risk conditions are compared and connected with corresponding SRL strategies. Practical and research implications are discussed in the article as well.

Do Video Modeling and Metacognitive Prompts Improve Self-Regulated Scientific Inquiry?

Article

Full-text available

Jun 2022
EDUC PSYCHOL REV

Guided inquiry learning is an effective method for learning about scientific concepts. The present study investigated the effects of combining video modeling (VM) examples and metacognitive prompts on university students’ (N = 127) scientific reasoning and self-regulation during inquiry learning. We compared the effects of watching VM examples combined with prompts (VMP) to watching VM examples only, and to unguided inquiry (control) in a training and a transfer task. Dependent variables were scientific reasoning ability, hypothesis and argumentation quality, and scientific reasoning and self-regulation processes. Participants in the VMP and VM conditions had higher hypothesis and argumentation quality in the training task and higher hypothesis quality in the transfer task compared to the control group. There was no added benefit of the prompts. Screen captures and think aloud protocols during the two tasks served to obtain insights into students’ scientific reasoning and self-regulation processes. Epistemic network analysis (ENA) and process mining were used to model the co-occurrence and sequences of these processes. The ENA identified stronger co-occurrences between scientific reasoning and self-regulation processes in the two VM conditions compared to the control condition. Process mining revealed that in the VM conditions these processes occurred in unique sequences and that self-regulation processes had many self-loops. Our findings show that video modeling examples are a promising instructional method for supporting inquiry learning on both the process and the learning outcomes level.

Prompts instrucionais como estratégia metacognitiva no ensino de arritmias cardíacas supraventriculares para estudantes de medicina

Article

Full-text available

Jan 2022

Objetivo: Comparar o efeito de dois métodos de ensino (prompts instrucionais e estudo autodirigido) na competência diagnóstica de arritmias supraventriculares em estudantes de medicina. Métodos: Estudo experimental com três fases: avaliação inicial (fase 1), treinamento (fase 2) e avaliação tardia (fase 3). A amostra foi composta por 24 estudantes, que na fase 1 forneceram os diagnósticos dos ritmos cardíacos de 11 traçados de Eletrocardiograma (ECG). Na fase 2, a amostra foi dividida em dois grupos. O Grupo 1 adotou o estudo autodirigido, e o Grupo 2 prompts instrucionais. Na fase 3 (após uma semana), toda a amostra forneceu os diagnósticos dos ritmos cardíacos de 11 novos traçados de ECG. Resultados: Após a fase 2, os alunos melhoraram os escores diagnósticos em relação à fase 1, independentemente o método de ensino utilizado. Na fase 3, a média das notas do Grupo 2 superou o Grupo 1 (p = 0,034). Conclusão: O uso de prompts instrucionais no ensino de ritmos cardíacos, foi capaz de melhorar a acurácia diagnóstica dos estudantes para as arritmias estudadas.

Han, F., & Ellis, R. A. (2021). Explaining medical students’ learning outcomes in blended course designs: Combining self-reported and observational learning experience. Advances in Physiology Education. https://doi.org/10.1152/advan.00121.2021

Article

Nov 2021

Blended course designs have been increasingly adopted in medical education. However, research on the relations between the key aspects of students' learning experience and their learning outcomes often only measure students' self-reported experience, neglecting what they actually do in learning. This study combined both self-reported and observational measures of students' learning experiences and examined the relations between the two sets of measures and their contributions to learning outcomes. Australian medical students were asked to report their approaches to, and perceptions of, learning. Their frequency and duration of their interactions with both online formative and summative tasks were observed and recorded. Correlation analyses showed that the learning outcomes was positively related to deep approaches to using online technologies and duration of interactions with online summative tasks. The hierarchical multiple regression analysis found that the self-reported approaches and duration of interactions jointed explained the learning outcomes, accounting for 6% of the variance. The study demonstrated the complementary nature of using both self-reported and observational measures of students' learning experiences to explain the learning outcomes in blended course designs.

Process Science in Action: A Literature Review on Process Mining in Business Management

Article

Nov 2021
TECHNOL FORECAST SOC

Process Mining is a new kind of Business Analytics and has emerged as a powerful family of Process Science techniques for analysing and improving business processes. Although Process Mining has managerial benefits, such as better decision making, the scientific literature has investigated it mainly from a computer science standpoint and appears to have overlooked various possible applications. We reviewed management-orientated literature on Process Mining and Business Management to assess the state of the art and to pave the way for further research. We built a seven-dimension framework to develop and guide the review. We selected and analysed 145 papers and identified eleven research gaps sorted into four categories. Our findings were formalised in a structured research agenda suggesting twenty-five research questions. We believe that these questions may stimulate the application of Process Mining in promising, albeit little explored, business contexts and in mostly unaddressed managerial areas.

Time-driven modeling of student self-regulated learning in network-based tutors

Article

Mar 2021

This study applies a time-driven approach to model self-regulated learning (SRL) on the basis of elapsed time metrics in the context of open-ended learning environments (OELEs), specifically, network-based tutors. In doing so, we examine how students allocated attentional resources to distinct phases of SRL as a measure of depth of information processing. Student teachers (N=68) were assigned to two different versions of nBrowser: a static version where the network did not converge on the basis of student interactions and a dynamic version where the network was continually updated by the system. Students designed a lesson plan and completed pre- and post-test self-report measures of knowledge gains. In both the experimental conditions, the results show four distinct SRL profiles that are relatively consistent and can be detected on the basis of behavioral patterns logged by the system across behaviors, namely, planning, requesting hints, studying examples, and monitoring. Although students who allocated more attentional resources to studying examples performed more poorly, their efforts to engage in planning, requesting hints, and monitoring were found to predict knowledge gains and design skills. Furthermore, students assigned to the dynamic version of the system outperformed those assigned to the static version in pedagogical knowledge gains.

Process Analysis of Teachers’ Self-Regulated Learning Patterns in Technological Pedagogical Content Knowledge Development

Article

Feb 2021
COMPUT EDUC

Self-regulated learning (SRL) has a predictable and instrumental effect on learning complicated knowledge. This study investigates the role of SRL in acquiring technological pedagogical content knowledge (TPACK), an important aspect of teachers’ effective technology use. The present study identified several regulatory procedural patterns used by teachers in the context of their TPACK achievements. A computer-based context, nBrowser, was used to facilitate teachers lesson planning around technology usage. Teachers log file data were analysed using process mining approaches. Findings indicate that high TPACK performers are more likely to perform self-regulative activities (e.g., monitoring) in developing TPACK compared to the low performers. Higher TPACK performers are more goal-oriented, demonstrate more monitoring and are more iterative in using all SRL processes in contrast to low performers who only partially regulate their problem solving. Such findings support previous research. This study adopts a novel approach for understanding the relations between SRL and TPACK. It offers opportunities to examine how teachers enact SRL as they move from the beginning to later stages of designing lessons and provides insights to researchers who study SRL in TPACK domains. Furthermore, the findings can assist educational designers in developing interventions for promoting TPACK development by concentrating on teachers’ SRL abilities.

Analyzing Multimodal Multichannel Data about Self-Regulated Learning with Advanced Learning Technologies: Issues and Challenges

Article

Full-text available

Jul 2019
COMPUT HUM BEHAV

Analyzing multimodal multichannel data about self-regulated learning (SRL) obtained during the use of advanced learning technologies such as intelligent tutoring systems, serious games, hypermedia, and immersive virtual learning environments is key to understanding the interplay among cognitive, affective, metacognitive, and social processes and their impact on learning, problem solving, reasoning, and conceptual understanding in learners of all ages and contexts. In this special issue of Computers in Human Behavior, we report six studies conducted by interdisciplinary teams’ use of various trace methodologies such as eye tracking, log-files, physiological data, facial expressions of emotions, screen recordings, concurrent think-alouds, and linguistic analyses of discourse. The research studies focus on how these data were analyzed using a combination of traditional statistical techniques as well as educational data-mining procedures to detect, measure, and infer cognitive, metacognitive, and social processes related to regulating the self and others across several tasks, domains, ages, and contexts. The results of these studies point to future work necessitating interdisciplinary researchers’ collaboration to use theoretically based and empirically derived approaches to collecting, measuring, and modeling multimodal multichannel SRL data to extend our current models, frameworks, and theories by making them more predictive by elucidating the nature, complexity, and temporality of underlying processes. Lastly, analyses of multimodal multichannel SRL process data can significantly augment advanced learning technologies by providing real-time, intelligent, adaptive, individualized scaffolding and feedback to address learners’ self-regulatory needs.

Examining self-efficacy during learning: variability and relations to behavior, performance, and learning

Article

Full-text available

Apr 2015

Self-regulated learning (SRL) theorists propose that learners' motivations and cognitive and metacognitive processes interact dynamically during learning, yet researchers typically measure motivational constructs as stable factors. In this study, self-efficacy was assessed frequently to observe its variability during learning and how learners' efficacy related to their problem-solving performance and behavior. Students responded to self-efficacy prompts after every fourth problem of an algebra unit completed in an intelligent tutoring system. The software logged students' problem-solving behaviors and performance. The results of stability and change, path, and correlational analyses indicate that learners' feelings of efficacy varied reliably over the learning task. Their prior performance (i.e., accuracy) predicted subsequent self-efficacy judgments, but this relationship diminished over time as judgments were decreasingly informed by accuracy and increasingly informed by fluency. Controlling for prior achievement and self-efficacy, increases in efficacy during one problem-solving period predicted help-seeking behavior, performance, and learning in the next period. Findings suggest that self-efficacy varies during learning, that students consider multiple aspects of performance to inform their efficacy judgments, and that changes in efficacy influence self-regulated learning processes and outcomes.

A Framework for Evaluating and Enhancing Alignment in Self-Regulated Learning Research

Article

Full-text available

Apr 2015

We discuss the articles of this special issue with reference to an important yet previously only implicit dimension of study quality: alignment across the theoretical and methodological decisions that collectively define an approach to self-regulated learning. Integrating and extending work by leaders in the field, we propose a framework for evaluating alignment in the way self-regulated learning research is both conducted and reported. Within this framework, the special issue articles provide a springboard for discussing methodological promises and pitfalls of increasingly sophisticated research on the dynamic, contingent, and contextualized features of self-regulated learning.

Addressing complexities in self-regulated learning: a focus on contextual factors, contingencies, and dynamic relations

Article

Full-text available

Apr 2015

IntroductionThe conceptualization of self-regulated learning (SRL) has been profoundly articulated in recent special issues focused on the similarities and distinctions between SRL and related phenomena like self-regulation, metacognition, and motivation as well as the development of new environments and methods that improve SRL research (Alexander 2008; Azevedo 2005a, 2005b, 2007; Azevedo and Hadwin 2005; Greene and Azevedo 2010; Gress and Hadwin 2010; Järvelä and Hadwin 2013; Molenaar and Järvelä 2014; Perry 2002; Winne and Baker 2013). These special issues have examined how the affordances of (usually computer-based) learning environments can provide the means to observe phenomena described in SRL models and to empirically test theoretical assumptions (Azevedo 2005b; Azevedo and Hadwin 2005; Efklides 2011; Winne and Hadwin 2008; Winne 2010, 2011; Zimmerman and Schunk 2008, 2011). In this special issue, we tighten the focus of this ongoing discussion by focusing on three key features ...

Issues in dealing with sequential and temporal characteristics of self- and socially-regulated learning

Article

Full-text available

Aug 2014

Roger Azevedo

Sequential and temporal characteristics of self and socially regulated learning

Article

Full-text available

Aug 2014

Students’ Adaptation and Transfer of Strategies across Levels of Scaffolding in an Exploratory Environment

Conference Paper

Jun 2014

While the effect of scaffolding on learning has received much attention, less is known about its effect on students’ strategy use, especially in transfer activities. This study focuses on students’ adaptive behaviours as a function of given scaffolding and when transitioning from a scaffolded to an unstructured activity. We study this in the context of a complex physics simulation in which students choose between 124 different actions. We evaluate (i) how the scaffolding affects students’ building and testing behaviours, (ii) whether these behaviours transfer to an unstructured activity, and (iii) the relationship between the adapted behaviours and learning. A repeated-measures MANOVA suggests that students adapt their learning behaviours according to the demands and affordances of the task and the environment, and that these strategies transfer from a scaffolded to an unstructured activity. No significant relationships were found between these patterns and learning.

The effectiveness of self-regulated learning scaffolds on academic performance in computer-based learning environments: a meta-analysis

Article

Apr 2016

Lanqin Zheng

This meta-analysis examined research on the effects of self-regulated learning scaffolds on academic performance in computer-based learning environments from 2004 to 2015. A total of 29 articles met inclusion criteria and were included in the final analysis with a total sample size of 2648 students. Moderator analyses were performed using a random effects model that focused on the three main areas of scaffold characteristics (including the mechanism, functions, delivery forms, mode, and number of scaffolds; how to promote self-regulated learning by scaffolds); demographics of the selected studies (including sample groups, sample size, learning domain, research settings, and types of computer-based learning environments); and research methodological features (including research methods, types of research design, types of organization for treatment, and duration of treatment). Findings revealed that self-regulated learning scaffolds in computer-based learning environments generally produced a significantly positive effect on academic performance (ES = 0.438). It is also suggested that both domain-general and domain-specific scaffolds can support the entire process of self-regulated learning since they demonstrated substantial effects on academic performance. Different impacts of various studies and their methodological features are presented and discussed.

Issues in researching self-regulated learning as patterns of events

Article

Aug 2014

Philip H Winne

New methods for gathering and analyzing data about events that comprise self-regulated learning (SRL) support discoveries about patterns among events and tests of hypotheses about roles patterns play in learning. Five such methodologies are discussed in the context of four key questions that shape investigations into patterns in SRL. A framework for this review is provided by a model that structures SRL in terms of: conditions of a task, operations, products generated by operations, evaluations of work and standards used in evaluations (COPES; Winne in Journal of Educational Psychology, 89, 397–410, 1997). Four recommendations are made for future work on SRL as patterned activity: prune models of SRL with experimental tests, explicitly include goals in data, ensure learners have options for SRL by training them in tactics and strategies, and provide learners access to accurate displays about the events and patterns that comprise SRL.

Process mining techniques for analysing patterns and strategies in students’ self-regulated learning

Article

Aug 2013

Referring to current research on self-regulated learning, we analyse individual regulation in terms of a set of specific sequences of regulatory activities. Successful students perform regulatory activities such as analysing, planning, monitoring and evaluating cognitive and motivational aspects during learning not only with a higher frequency than less successful learners, but also in a different order—or so we hypothesize. Whereas most research has concentrated on frequency analysis, so far, little is known about how students’ regulatory activities unfold over time. Thus, the aim of our approach is to also analyse the temporal order of spontaneous individual regulation activities. In this paper, we demonstrate how various methods developed in process mining research can be applied to identify process patterns in self-regulated learning events as captured in verbal protocols. We also show how theoretical SRL process models can be tested with process mining methods. Thinking aloud data from a study with 38 participants learning in a self-regulated manner from a hypermedia are used to illustrate the methodological points.

Using Process Mining to Examine the Sustainability of Instructional Support: How Stable Are the Effects of Metacognitive Prompting on Self-Regulatory Behavior?

Abstract

No full-text available

Recommended publications

Analyzing temporal data for understanding the learning process induced by metacognitive prompts