Jaclyn Ocumpaugh

• Managing Director at University of Pennsylvania

Detection of carelessness in digital learning platforms has relied on the contextual slip model, which leverages conditional probability and Bayesian Knowledge Tracing (BKT) to identify careless errors, where students make mistakes despite having the knowledge. However, this model cannot effectively assess carelessness in questions tagged with mult...

This study investigates student learning and interest within the context of a single-player, open-world game designed for microbiology inquiry. The game immerses players in the role of investigative scientists tasked with diagnosing a mysterious illness on a remote island. Ordered Network Analysis (ONA) was combined with clustering techniques to an...

This study explores the potential of the large language model GPT-4 as an automated tool for qualitative data analysis by educational researchers, exploring which techniques are most successful for different types of constructs. Specifically, we assess three different prompt engineering strategies-Zero-shot, Few-shot, and Few-shot with contextual i...

Detection of carelessness in digital learning platforms has relied on the contextual slip model, which leverages conditional probability and Bayesian Knowledge Tracing (BKT) to identify careless errors, where students make mistakes despite having the knowledge. However, this model cannot effectively assess carelessness in questions tagged with mult...

This study explores how student actions in Minecraft-based virtual environments designed to simulate astronomical phenomena shift over time, as their interest in astronomy changes. We analyze observations made by middle school learners participating in the What-if Hypothetical Implementations in Minecraft (WHIMC) project, which adapts the game to i...

The artificial intelligence in education (AIED) community has produced technologies that are widely used to support learning, teaching, assessment, and administration. This work has successfully enhanced test scores, course grades, skill acquisition, comprehension, engagement, and related outcomes. However, the prevailing approach to adaptive and p...

In this paper, we propose a new method for selecting cases for in situ, immediate interview research: Detector-Driven Classroom Interviewing (DDCI). Published work in educational data mining and learning analytics has yielded highly scalable measures that can detect key aspects of student interaction with computer-based learning in close to real-ti...

Prior research has shown that digital games can enhance STEM education by providing learners with immersive and authentic scientific experiences. However, optimizing the learning outcomes of students engaged in game-based environments requires aligning the game design with diverse student needs. Therefore, an in-depth understanding of player behavi...

Help from virtual pedagogical agents has the potential to improve student learning. Yet students often do not seek help when they need it, do not use help effectively , or ignore the agent's help altogether. This paper seeks to better understand stu-dents' patterns of accepting and seeking help in a computer-based science program called Betty's Bra...

There exist several online applications for automated testing of the computer programs that students write in computer science education. Use of such systems enables self-paced learning with automated feedback delivered by the application. However, due to the complexity of programming languages, even the easiest tasks made available through such sy...

Game-based learning offers rich learning opportunities, but open-ended games make it difficult to identify struggling students. Prior work compares student paths to a single expert’s “golden path.” This effort focuses on efficiency, but additional pathways may be required for learning. We examine data from middle schoolers who played Crystal Island...

Pedagogical agents offer significant promise for engaging students in learning. In this paper, we investigate students’ conversational interactions with a pedagogical agent in a game-based learning environment for middle school science education. We utilize word embeddings of student-agent conversations along with features distilled from students’...

There exist several online applications for automated testing of the computer programs that students write in computer science education. Use of such systems enables self-paced learning with automated feedback delivered by the application. However, due to the complexity of programming languages, even the easiest tasks made available through such sy...

Feedback is essential to successful learning and instruction. Much of the recent feedback literature has focused on instructors providing feedback to students (Van Boekel et al., 2021), however, there is also a growing body of work considering peer feedback, where learners provide feedback to other learners. The effect of peer feedback is two-fold:...

Help from virtual pedagogical agents has the potential to improve student learning. Yet students often do not seek help when they need it, do not use help effectively, or ignore the agent’s help altogether. This paper seeks to better understand students’ patterns of accepting and seeking help in a computer-based science program called Betty’s Brain...

Self-regulated learning (SRL) is a critical component of mathematics problem-solving. Students skilled in SRL are more likely to effectively set goals, search for information, and direct their attention and cognitive process so that they align their efforts with their objectives. An influential framework for SRL, the SMART model (Winne, 2017), prop...

Background Providing adaptive scaffolds to help learners develop effective self‐regulated learning (SRL) behaviours has been an important goal for intelligent learning environments. Adaptive scaffolding is especially important in open‐ended learning environments (OELE), where novice learners often face difficulties in completing their learning task...

This study uses a mixed methods approach to explore relationships between goal achievement orientation and changes in student achievement, behavior, and affect while using Betty's Brain. Qualitative coding was applied to student interviews to identify either performance or mastery goal approaches applied during their interaction with the game. A co...

Self-regulated learning (SRL) is a critical component of mathematics problem solving. Students skilled in SRL are more likely to effectively set goals, search for information, and direct their attention and cognitive process so that they align their efforts with their objectives. An influential framework for SRL, the SMART model, proposes that five...

full paper view link https://urldefense.com/v3/__https:/rdcu.be/cK86e__;!!DZ3fjg!viPalScMMCiA9Bt3jnu3WSYZ1TXWSby157lusNo9TAhBRtPb8W2TUJxVp4C7lIRBe9I$ The feelings of difficulty and familiarity (FOD and FOF) are two types of metacognitive experiences. Both may influence student engagement and the application of metacognitive strategies, but these r...

Automated, data‐driven decision making is increasingly common in a variety of application domains. In educational software, for example, machine learning has been applied to tasks like selecting the next exercise for students to complete. Machine learning methods, however, are not always equally effective for all groups of students. Current approac...

There is a growing interest in viewing self-regulated learning as events unfolding over time, especially when students perform learning tasks in computer-based environments. Metacognitive activities are critical events in self-regulated learning. This study investigated the evolution of metacognitive strategy use in an open-ended computer-based lea...

Providing adaptive scaffolds to help learners develop self-regulated learning (SRL) processes has been an important goal for intelligent learning environments. In this paper, we develop a systematic framework for adaptive scaffolding in Betty's Brain, an open-ended learning-by-teaching environment that helps middle school students learn science by...

This study integrates the analysis of student interaction data with classroom interviews in order to better understand how students’ trait-level anxiety relates to in-the-moment measures of students’ affective experiences (i.e., boredom, confusion, delight, engaged concentration, and frustration). We first use quantitative data to drive the data co...

This study integrates the analysis of student interaction data with classroom interviews in order to better understand how students' trait-level anxiety relates to in-the-moment measures of students' affective experiences (i.e., boredom , confusion, delight, engaged concentration, and frustration). We first use quantitative data to drive the data c...

Artificial Intelligence in Education research for STEM domains has largely been quantitative in nature, but qualitative research offers several advantages as part of a mixed-methods approach. In particular, qualitative research enables researchers to develop deeper phenomenological understanding of how learners represent their activity to themselve...

This paper examines how interviews with students-at critical moments of the learning process-may be leveraged to improve the design of educational software. Specifically, we discuss iterative work to improve the design of a pedagogical agent in the Betty's Brain learning environment, Mr. Davis. Students interacted with the pedagogical agent in Bett...

Confusion may benefit learning when it is resolved or partially resolved. Metacognitive strategies (MS) may help learners to resolve confusion when it occurs during learning and problem solving. This study examined the relationship between confusion and MS that students invoked in Betty’s Brain, a computer-based learning-by modeling environment whe...

Educational technology (EdTech) designers need to ensure population validity as they attempt to meet the individual needs of all students. EdTech researchers often have access to larger and more diverse samples of student data to test replication across broad demographic contexts as compared to either the small-scale experiments or the larger conve...

Research shows that anxiety can disrupt learning processes, but few studies have examined anxiety's relationships to online learning behaviors. This study considers the interplay between students' anxiety about science and behavior within an online system designed to support self-regulated science inquiry. Using the searching, monitoring, assessing...

Frustration is a natural part of learning in AIED systems but remains relatively poorly understood. In particular, it remains unclear how students' perceptions about the learning activity drive their experience of frustration and their subsequent choices during learning. In this paper, we adopt a mixed-methods approach , using automated detectors o...

Self-regulated learning (SRL) is a critical 21 st-century skill. In this paper, we examine SRL through the lens of the searching, monitoring, assessing, rehearsing, and translating (SMART) schema for learning operations. We use microanalysis to measure SRL behaviors as students interact with a computer-based learning environment, Betty's Brain. We...

Affect dynamics, the study of how affect develops and manifests over time, has become a popular area of research in affective computing for learning. In this paper, we first provide a detailed analysis of prior affect dynamics studies, elaborating both their findings and the contextual and methodological differences between these studies. We then a...

Students in computerized learning environments often direct their own learning processes, which requires metacognitive awareness of what should be learned next. We investigated a novel method of measuring verbalized metacognition by applying natural language processing (NLP) to transcripts of interviews conducted in a classroom with 99 middle schoo...

Filipino learners’ lack of English language proficiency is a major barrier to higher education opportunities and participation in high-value industries. Computer-based learning systems have the potential to increase educational quality, equity, and efficacy in the Global South. However, a key challenge is to design systems that are developmentally...

This paper explores how early grade school students’ math performance relates to human ratings of students’ affect, identity, and social awareness based on the content of messages to an online tutoring system avatar. There is an expanding body of research which investigates connections between these features and success in mathematics. This study u...

Commercial facial affect detection software is typically trained on large databases and achieves high accuracy in detecting basic emotions, but their use in educational settings is unclear. The goal of this research is to determine how basic emotions relate to the achievement emotion states that are more relevant in academic settings. Such relation...

Now that the modeling of affective states is beginning to mature, understanding affect dynamics has become an increasingly realistic endeavor. However, the results from empirical studies have not always matched those of theoretical models, which raises questions as to why. In this study, we explore the relationship between affective sequences that...

Research that has examined relationships among mathematics success and student language patterns has typically focused on features of students’ language production such as linguistic sophistication, sentiment, and cognitive measures. There is also a small but growing body of research on how motivational and affective measures derived through survey...

The growing use of machine learning for the data-driven study of social issues and the implementation of data-driven decision processes has required researchers to reexamine the often implicit assumption that data-driven models are neutral and free of biases. The careful examination of machine-learned models has identified examples of how existing...

Confusion has been shown to be prevalent during complex learning and has mixed effects on learning. Whether confusion facilitates or hampers learning may depend on whether it is resolved or not. Confusion resolution, behind which is the resolution of cognitive disequilibrium, requires learners to possess some skills, but it is unclear what these sk...

We investigated the affective states (both individual and shared emotions) of students using a collaborative and educational game for English called Ibigkas! Our goal was two-fold: (1) To determine the incidence and persistence of affective states exhibited by the students when working individually and in groups, and (2) to adapt the Baker Rodrigo...

Discussion forum participation represents a crucial support for learning and often the only way of supporting social interactions in on-line settings. However, learner behavior varies considerably in these forums, including positive behaviors such as sharing new ideas or asking thoughtful questions, but also verbally abusive behaviors, which could...

Discussion forum participation represents a crucial support for learning and often the only way of supporting social interactions in on-line settings. However, learner behavior varies considerably in these forums, including positive behaviors such as sharing new ideas or asking thoughtful questions, but also verbally abusive behaviors, which could...

Previous studies have demonstrated strong links between students' linguistic knowledge, their affective language patterns and their success in math. Other studies have shown that demographic and click-stream variables in online learning environments are important predictors of math success. This study builds on this research in two ways. First, it...

Demographic information often proves useful for finding subpopulations in educational data. Unfortunately, it is often not collected in the log files of online learning systems, which serve as one of the primary sources of data for the Educational Data Mining community. Recent work has sought to address this issue by investigating school-level diff...

Affect dynamics, the study of how affect develops and manifests over the course of learning, has become a popular area of research in learning analytics. Despite some shared metrics and research questions, researchers in this area have some differences in how they pre-process the data for analysis [17]. Specifically, researchers differ in how they...

Ibigkas! is a team-based mobile-assisted language learning application that provides students with English language practice. Working collaboratively rather than competitively, players must find the rhyme, synonym, or antonym of a given target word among different lists of words on their mobile phones. At this time, Ibigkas! is not adaptive. In ord...

Filipino learners’ lack of English language proficiency is a major barrier to higher education opportunities and participation in high-value industries. Computer-based learning systems have the potential to increase educational quality, equity, and efficacy in the Global South. However, a key challenge is to design systems that are developmentally...

Education research has explored the role of students' affective states in learning, but some evidence suggests that existing models may not fully capture the meaning or frequency of how students transition between different states. In this study we examine the patterns of educationally-relevant affective states within the context of Betty's Brain,...

Education research has explored the role of students' affective states in learning, but some evidence suggests that existing models may not fully capture the meaning or frequency of how students transition between different states. In this study we examine the patterns of educationally-relevant affective states within the context of Betty's Brain,...

A better understanding of the relationship between self-concept in mathematics and fine-grained behavior logs from students' interactions with intelligent tutoring systems (ITSs) could help researchers better understand self-concept, which in turn could lead to improved designs in interventions intended to improve a student's self-concept. Yet, to...

In this chapter, we discuss over a decade of research to establish the Baker Rodrigo Ocumpaugh Monitoring Protocol (BROMP) as a method for conducting rapid, highly-quality, and time- synchronized quantitative field observations. We discuss work to establish standards and scalable training methods for the protocol in four countries, as well as work...

Considerable evidence demonstrates that motivational constructs predict educational outcomes, but little research has examined how these constructs manifest within online learning systems. This study addresses this gap by surveying Math Identity measures (self-concept, value, and interest in mathematics) and correlating them to behavior and perform...

D'Mello and Graesser's (2012) highly-cited model of affect dynamics proposes a sequence of theoretically-grounded transitions between affective states during learning. However, empirical studies in a range of contexts have not produced the predicted results. Several factors may explain this lack of replication, including the demographics of the pop...

In this paper, we discuss some of the results of a participatory design workshop used to elicit design guidelines for an education game for phonemic awareness intended for use by disadvantaged students. Using a grounded theory approach, we analyze facilitators’ observations from the workshop and related findings to well-established game design guid...

Note-taking is important for academic success and has been thoroughly studied in traditional classroom contexts. Recent advancements of technology have led to more students taking notes on computers, and in different situations than are common in traditional instructional contexts. However, research on computer-based note-taking is still an emergin...

The relationship between learners' cognitive and affective states has become a topic of increased interest, especially because it is an important component of self-regulated learning (SRL) processes. This paper studies sixth grade students' SRL processes as they work in Betty's Brain, an agent-based open-ended learning environment (OELE). In this e...

This paper investigates the interactions between learners' cognitive strategies and affective states; both important components of self-regulated learning (SRL) processes that influence student learning. We study cognitive-affective relationships in high versus low performing students as they worked on a model building task to teach their agent in...

The past few years have seen a surge of interest in deep neural networks. The wide application of deep learning in other domains such as image classification has driven considerable recent interest and efforts in applying these methods in educational domains. However, there is still limited research comparing the predictive power of the deep learni...

Student affect has been found to correlate with short-and long-term learning outcomes, including college attendance as well as interest and involvement in Science, Technology, Engineering, and Mathematics (STEM) careers. However, there still remain significant questions about the processes by which affect shifts and develops during the learning pro...

The past few years have seen a surge of interest in deep neural networks. The wide application of deep learning in other domains such as image classification has driven considerable recent interest and efforts in applying these methods in educational domains. However, there is still limited research comparing the predictive power of the deep learni...

A number of studies have demonstrated strong links between students' language features (as found in spoken and written production) and their math performance. However, no studies have examined links between the students' language features and measures of their Math Identity. This project extends prior studies that use natural language processing (N...

Background Interactive learning environments often provide help strategies to facilitate learning. Hints, for example, help students recall relevant concepts, identify mistakes, and make inferences. However, several studies have shown cases of ineffective help use. Findings from an initial study on the availability of hints in a mathematics problem...

Considerable evidence demonstrates that constructs that influence motivation, such as self-concept, interest, and value, are predictive of student outcomes. However, the ways in which such constructs influence behaviour has not been comprehensively studied within online learning systems. This study presents evidence addressing this gap by collectin...

This study examines patterns of rhyme identification among English Language Learners (ELLs) towards the development of an educational game, JOLLY, intended to improve phonemic awareness among school-aged children in the Philippines. Leveraging on students’ intrinsic interest in Western popular music, we ask students to identify rhyming words from a...

This paper examines the effect of different linguistic features (as identified through Natural Language Processing tools) on affective measures of student engagement using a discovery with models approach. We build on previous literature, using automated detectors that identify when a middle-school student using an online mathematics tutor is exper...

The role of affect in learning has received increasing attention from AIED researchers seeking to understand how emotion and cognition interact in learning contexts. The dynamics of affect over time have been explored in a variety of research environments, allowing researchers to determine the extent to which common patterns are captured by hypothe...

Advances in the learning analytics community have created opportunities to deliver early warnings that alert teachers and instructors when a student is at risk of not meeting academic goals [6], [71]. Alert systems have also been developed for school district leaders [33] and for academic advisors in higher education [39], but other professionals i...

The role of affect in learning has received increasing attention from AIED researchers seeking to understand how emotion and cognition interact in learning contexts. The dynamics of affect over time have been explored in a variety of research environments, allowing researchers to determine the extent to which common patterns are captured by hypothe...

This study uses correlation mining to investigate relationships between the linguistic properties of math problems and student outcomes. We find that linguistic properties associated with boredom were negatively associated with engaged concentration, an emotion which is boredom’s inverse in terms of activation (intensity of emotion) and valence (po...

Research suggests that trajectories toward careers in science, technology, engineering, and mathematics (STEM) emerge early and are influenced by multiple factors. This paper presents a longitudinal study, which uses data from 76 high school students to explore how a student’s vocational self-efficacy and interest are related to his or her middle s...

Affect detection is a key component in intelligent educational interfaces that respond to students' affective states. We use computer vision and machine-learning techniques to detect students' affect from facial expressions (primary channel) and gross body movements (secondary channel) during interactions with an educational physics game. We collec...

On-demand help in intelligent learning environments is typically linked to better learning, but may lead to longer completion times. This present work provides an analysis of how students interacted with a summer learning assignment when on-demand help was available, compared to when it was not. When hints were available from the start, students we...

The creation of crowd-sourced content in learning systems is a powerful method for adapting learning systems to the needs of a range of teachers in a range of domains, but the quality of this content can vary. This study explores linguistic differences in teacher-created problem content in ASSISTments using a combination of discovery with models an...

Recent interest in online assessment of scientific inquiry has led to several new online systems that attempt to assess these skills, but producing models that detect when students are successfully practising these skills can be challenging. In this paper, we study models that assess student inquiry in an immersive virtual environment, where a stud...

The expanded use of online and blended learning programs in K-12 STEM education has led researchers to propose design principles for effective e-learning systems. Much of this research has focused on the impact on learning, but not how instructional design impacts student engagement, which has a critical impact both on short-term learning and long-...

This study investigated the relationships among incoming knowledge, persistence, affective states, in-game progress, and consequently learning outcomes for students using the game Physics Playground. We used structural equation modeling to examine these relations. We tested three models, obtaining a model with good fit to the data. We found evidenc...

This paper evaluates the Human Affect Recording Tool (HART), a Computer Assisted Direct Observation (CADO) application that facilitates scientific sampling. HART enforces an established method for systematic direct observation in Educational Data Mining (EDM) research, the Baker Rodrigo Ocumpaugh Monitoring Protocol [25] [26]. This examination prov...

Replicable research on the behavior known as gaming the system, in which students try to succeed by exploiting the functionalities of a learning environment instead of learning the material, has shown it is negatively correlated with learning outcomes. As such, many have developed models that can automatically detect gaming behaviors, towards deplo...

Increased attention to the relationships between affect and learning has led to the development of machine-learned models that are able to identify students' affective states in computerized learning environments. Data for these affect detectors have been collected from multiple modalities including physical sensors, dialogue logs, and logs of stud...

Student engagement indicators, such as behavior and affective states, are known to impact learning. This study uses an established quantitative field observation method to evaluate engagement during students’ use of a new version of an online learning system (Reasoning Mind’s Genie 3). Improvements to Genie 3’s design intended to increase engagemen...