Albert T. Corbett’s research while affiliated with Carnegie Mellon University and other places

Bayesian Knowledge Tracing (BKT) has been employed successfully in intelligent learning environments to individualize curriculum sequencing and help messages. Standard BKT employs four parameters, which are estimated separately for individual knowledge components, but not for individual students. Studies have shown that individualizing the parameter estimates for students based on existing data logs improves goodness of fit and leads to substantially different practice recommendations. This study investigates how well BKT parameters in a tutor lesson can be individualized ahead of time, based on learners' prior activities, including reading text and completing prior tutor lessons. We find that directly applying best-fitting individualized parameter estimates from prior tutor lessons does not appreciably improve BKT goodness of fit for a later tutor lesson, but that individual differences in the later lesson can be effectively predicted from measures of learners' behaviors in reading text and in completing the prior tutor lessons.

Background Across computer-based and traditional classroom settings, recent studies have identified motivational orientation, prior knowledge, self-regulation, and cognitive load as possible factors that affect help-seeking behaviors and their impact on learning. However, the question of whether there is an optimal point for determining when a student needs help has not been fully explored. Purpose of Study Using data from two modules of the Genetics Cognitive Tutor, the present study investigates this question by examining whether the relationship of help avoidance (failing to seek help when it is needed) and student learning is dependent on the student's level of prior knowledge. We also investigate how the relationship between help avoidance and student learning is mediated by the amount of prior practice, or the number of attempts at a problem step. Research Design We obtained existing data from the use of the Genetics Cognitive Tutor. We conducted a series of correlational analyses to better understand the relationship between help avoidance and student learning. We correlated students’ proportions of help avoidance at different levels of knowledge with measures of robust learning. We also analyzed the relationship between students’ proportions of help avoidance and measures of robust learning, taking the amount of practice or the number of attempts at a problem step into account. Results Our findings suggest that, except at very high or very low knowledge, help avoidance is generally stably (negatively) related to robust learning outcomes. Our results also indicate that help avoidance is more strongly associated with learning outcomes early in the practice sequence, suggesting that students should be encouraged to seek help on problem-solving skills on the first problem, rather than waiting until later problems. Similarly, our results reveal that help avoidance is more negatively associated with learning outcomes on early attempts at a problem step than on later attempts, indicating that students should be encouraged to seek help on the first attempt if help is needed. Conclusions These findings represent a step toward understanding when students should seek help, with the potential of improving the design of metacognitive support within adaptive learning systems.

This study examines how accurately individual student differences in learning can be predicted from prior student learning activities. Bayesian Knowledge Tracing (BKT) predicts learner performance well and has often been employed to implement cognitive mastery. Standard BKT individualizes parameter estimates for knowledge components, but not for learners. Studies have shown that individualizing parameters for learners improves the quality of BKT fits and can lead to very different (and potentially better) practice recommendations. These studies typically derive best-fitting individualized learner parameters from learner performance in existing data logs, making the methods difficult to deploy in actual tutor use. In this work, we examine how well BKT parameters in a tutor lesson can be individualized based on learners' prior performance in reading instructional text, taking a pretest, and completing an earlier tutor lesson. We find that best-fitting individual difference estimates do not directly transfer well from one tutor lesson to another, but that predictive models incorporating variables extracted from prior reading, pretest and tutor activities perform well, when compared to a standard BKT model and a model with best-fitting individualized parameter estimates.

Past studies have shown that Bayesian Knowledge Tracing (BKT) can predict student performance and implement Cognitive Mastery successfully. Standard BKT individualizes parameter estimates for skills, also referred to as knowledge components (KCs), but not for students. Studies deriving individual student parameters from the data logs of student tutor performance have shown improvements to the standard BKT model fits, and result in different practice recommendations for students. This study investigates whether individual student parameters, specifically individual difference weights (IDWs) [1], can be derived from student activities prior to tutor use. We find that student performance measures in reading instructional text and in a conceptual knowledge pretest can be employed to predict IDWs. Further, we find that a model incorporating these predicted IDWs performs well, in terms of model fit and learning efficiency, when compared to a standard BKT model and a model with best-fitting IDWs derived from tutor performance.

We describe a programming tutor framework that consists of two configurable components, a guided-planning component and an assisted-coding component that offers task relevant automatically-generated hints on demand to students. We evaluate the effectiveness of the new integrated planning and coding environment by comparing it to three other tutor conditions: planning-only, coding-only, and planning-only interleaved with planning-coding. We conclude that the integrated planning and coding tutor environment is more effective than tutored planning-only activities and that students make more efficient use of tutor feedback in the integrated environment than in the coding only environment.

Recently, there has been growing emphasis on supporting robust learning within intelligent tutoring systems, assessed by measures such as transfer to related skills, preparation for future learning, and longer term retention. It has been shown that different pedagogical strategies promote robust learning to different degrees. However, the student modeling methods embedded within intelligent tutoring systems remain focused on assessing basic skill learning rather than robust learning. Recent work has proposed models, developed using educational data mining, that infer whether students are acquiring learning that transfers to related skills, and prepares the student for future learning (PFL). In this earlier work, evidence was presented that these models achieve superior prediction of robust learning to what can be achieved by traditional methods for student modeling. However, using these models to drive intervention by educational software depends on evidence that these models remain effective within new populations. To this end, we analyze the degree to which these detectors remain accurate for an entirely new population of high school students. We find limited evidence of degradation for transfer. More degradation is seen for PFL. This degradation appears to occur in part because it is generally more difficult to infer this construct within the new population. (PsycINFO Database Record (c) 2013 APA, all rights reserved)

This paper describes two types of Conceptually Grounded Learning Activities designed to foster more robust learning in the Genetics Cognitive Tutor: interleaved worked examples and genetic-process reasoning scaffolds. We report three empirical studies that evaluate the impact of these learning activities on three diverse genetics problem-solving topics in the tutor. We found that interleaved worked examples yielded less basic-skill learning than conventional problem solving, unlike many prior ITS studies of worked examples. We also found preliminary evidence that scaffolded reasoning tasks in conjunction with conventional problem solving leads to more robust understanding than conventional problem solving alone. Implications for the use of contextually grounded learning activities are discussed.

In this chapter, we will discuss our work to understand why students game the system. This work leverages models of student gaming, termed “detectors”, which can infer student gaming in log files of student interaction with educational software. These detectors are developed using a combination of human observation and annotation, and educational data mining. We then apply the detectors to large data sets, and analyze the detectors’ predictions, using discovery with models methods, to study the factors associated with gaming behavior. Within this chapter, we will discuss the work to develop these detectors, and what we have discovered through these analyses based on these detectors. We will discuss evidence for how gaming the system impacts learning and evidence for why students choose to game. We will also discuss attempts to address gaming the system through adaptive scaffolding.