Chapter

Mastery Learning Heuristics and Their Hidden Models

Authors:
To read the full-text of this research, you can request a copy directly from the author.

Abstract

Mastery learning algorithms are used in many adaptive learning technologies to assess when a student has learned a particular concept or skill. To assess mastery, some technologies utilize data-driven models while others use simple heuristics. Prior work has suggested that heuristics may often perform comparably to model-based algorithms. But is there any reason we should expect these heuristics to be reasonable? In this paper, we show that two prominent mastery learning heuristics can be reinterpreted as model-based algorithms. In particular, we show that the N-Consecutive Correct in a Row heuristic and a simplified version of ALEKS’ mastery learning heuristic are both optimal policies for variants of the Bayesian knowledge tracing model. By putting mastery learning heuristics on the same playing field as model-based algorithms, we can gain insights on their hidden assumptions about learning and why they might perform well in practice.

No full-text available

Request Full-text Paper PDF

To read the full-text of this research,
you can request a copy directly from the author.

... The idea of making a student reach a mastery level is by feeding him/her with just the proper amount of education based on their needs on that topic before moving forward to another. In recent years, mastery learning, positive persistence, and successful learning have been the goal of many learning technologies like Khan Academy, Duolingo, ASSISTments, ALEKS, and cognitive educators such as MATHia and CAT [31]. ...
... In the light of choosing the best method for analysing mastery learning heuristic, the researcher in [31] has examined mastery learning from different types of educational datasets. The N-Consecutive Correct answers in a Row (N-CCR) heuristic, ASSISTments, and a simplified version of ALEKS mastery learning heuristic are ideal strategies for the Bayesian knowledge tracing model (BKT). ...
Article
Full-text available
Digital learning environments have offered new opportunities to stream educational materials such as courses, educational videos, forums and provide outcomes of the students, grades, engagement details, and learning patterns. These valuable educational data has emerged with the latest technologies and software tools to provide researchers and decision-makers with a better understanding of students' behaviours. These virtual learning environments can professionally aid struggling students by observing, learning, and identifying different learning patterns. Many researchers have discussed that even if there are instructions and helping tools within these environments, some students remain at risk of negative learning behaviours such as boredom, disengagement, and failure. Particularly when approaching complex or new educational content. Previous researchers have observed that the students exhibit high persistence levels when spending too much time on a particular task when they are learning remotely or too many tries to solve a specific task without reaching the success level. Students' persistence is identified as a prominent learning skill contributing to confirmed success while learning new education materials. Many works of literature recognised the value of persistence. They reached a fundamental fact that not all persistence is considered productive, especially when spending more time and effort without moving toward a state of mastery in learning new skills and topics. This scenario may eventually lead to frustration and disengagement; in the worst-case scenarios, the students will finally drop the course or just quit learning. By examining the most relevant literature, this paper discusses the main factors affecting students' digital learning persistence. Different models performed at each learning opportunity are observed to categorise when involvements may be arranged to best aid the learners facing learning struggles.
... Such various groupings provide access to individuals looking for very specific assignments, readings, videos, etc., but also individuals looking for an overview, or a comprehensive picture, of a subject area. Students in the Levels II and III categories must demonstrate proficiency after completing the courses (Doroudi, 2020). Developing assessments to measure the knowledge and skills in their content domain is challenging and expensive on the frontend of development (Towns, 2014;Bearman et al., 2017), but techniques are evolving rapidly to improve this, even without human intervention (Kurnia et al., 2001). ...
Article
Full-text available
The COVID19 pandemic has revealed deep, ingrained problems with higher education, but also opportunities for positive transformation. In the post-COVID world, education at all levels has the chance to become: (1) universally available at low cost; (2) focused on developing competencies, (3) empowering fulfilling lives, not merely job training; and (4) engaged with communities to solve real-world problems. Achieving this will require overcoming the mass production model of higher education by utilizing the full potential of the Internet in creative ways balanced with face-to-face solutions-based integrated learning, research, and outreach agenda. Building a global collaborative consortium of universities and other educational institutions can move this agenda forward. We describe how this “MetaUniversity” could be structured and how it would serve to advance this agenda and lead the way to a sustainable well-being future for humanity and the rest of nature.
... programming skills) need to be attained. Existing systems lack modern techniques or tools for students to practice and master their skills (Doroudi, 2020). Thus, based on their capabilities, we recommend that the various techniques (AI and data analytics techniques), which we classified into three basic categories (descriptive, predictive and prescriptive analytics), be used to address the problems related to the complexity of learning systems models, the lack of evaluation standards and methods for AI-enabled learning systems and the difficulties in efficiently organising complex information to support students in skill attainment. ...
Article
Full-text available
Mobile internet, cloud computing, big data technologies, and significant breakthroughs in Artificial Intelligence (AI) have all transformed education. In recent years, there has been an emergence of more advanced AI-enabled learning systems, which are gaining traction due to their ability to deliver learning content and adapt to the individual needs of students. Yet, even though these contemporary learning systems are useful educational platforms that meet students’ needs, there is still a low number of implemented systems designed to address the concerns and problems faced by many students. Based on this perspective, a systematic mapping of the literature on AI-enabled adaptive learning systems was performed in this work. A total of 147 studies published between 2014 and 2020 were analysed. The major findings and contributions of this paper include the identification of the types of AI-enabled learning interventions used, a visualisation of the co-occurrences of authors associated with major research themes in AI-enabled learning systems and a review of common analytical methods and related techniques utilised in such learning systems. This mapping can serve as a guide for future studies on how to better design AI-enabled learning systems to solve specific learning problems and improve users’ learning experiences.
Chapter
Covid-19 has catalyzed positive disruptive change in the contemporary education system. It has accelerated the necessity of an innovation implementation process, technology-enhanced learning environment, customized and student-centered learning. It has enabled educators to tailor course content and pathways to students’ strengths, skills, interests and learning profiles. Adaptive learning is a contemporary method orchestrating human and digital resources to meet the needs of individual learners. The paper gives a review of the innovation process in education and the implementation of theoretical adaptive learning framework in Russian universities. It identifies innovators and early adopters who can assist two Russian Universities in attempts to implement new educational approaches. The authors also consider if there are any correlations between professors’ self-categorization and their practical application of adaptive learning in terms of the pace of learning, type of content presentation, knowledge state and students’ interests. Moreover, the researchers investigate and evaluate the Public Relations students’ perceived importance of factors affecting their learning: subject choice, subject knowledge acquisition, a type of subject content (text, audio, video), an individual pace in studying the subject, a lecturer’s guidance and group size.
Article
Full-text available
Prior research has shown that tutored problem solving with intelligent software tutors is an effective instructional method, and that worked examples are an effective complement to this kind of tutored problem solving. The work on the expertise reversal effect suggests that it is desirable to tailor the fading of worked examples to individual students’ growing expertise levels. One lab and one classroom experiment were conducted to investigate whether adaptively fading worked examples in a tutored problem-solving environment can lead to higher learning gains. Both studies compared a standard Cognitive Tutor with two example-enhanced versions, in which the fading of worked examples occurred either in a fixed manner or in a manner adaptive to individual students’ understanding of the examples. Both experiments provide evidence of improved learning results from adaptive fading over fixed fading over problem solving. We discuss how to further optimize the fading procedure matching each individual student’s changing knowledge level. KeywordsCognitive tutor-Worked examples-Adaptive fading-Expertise reversal effect
Conference Paper
Full-text available
This study examined the effectiveness of an educational data mining method - Learning Factors Analysis (LFA) - on improving the learning efficiency in the Cognitive Tutor curriculum. LFA uses a statistical model to predict how students perform in each practice of a knowledge component (KC), and identifies over-practiced or under-practiced KCs. By using the LFA findings on the Cognitive Tutor geometry curriculum, we optimized the curriculum with the goal of improving student learning efficiency. With a control group design, we analyzed the learning performance and the learning time of high school students participating in the Optimized Cognitive Tutor geometry curriculum. Results were compared to students participating in the traditional Cognitive Tutor geometry curriculum. Analyses indicated that students in the optimized condition saved a significant amount of time in the optimized curriculum units, compared with the time spent by the control group. There was no significant difference in the learning performance of the two groups in either an immediate post test or a two-week-later retention test. Findings support the use of this data mining technique to improve learning efficiency with other computer-tutor-based curricula.
Conference Paper
Full-text available
Intelligent tutoring systems that utilize Bayesian Knowledge Tracing have achieved the ability to accurately predict student performance not only within the intelligent tutoring system, but on paper post-tests outside of the system. Recent work has suggested that contextual estimation of student guessing and slipping leads to better prediction within the tutoring software (Baker, Corbett, & Aleven, 2008a, 2008b). However, it is not yet clear whether this new variant on knowledge tracing is effective at predicting the latent student knowledge that leads to successful post-test performance. In this paper, we compare the Contextual-Guess-and-Slip variant on Bayesian Knowledge Tracing to classical four-parameter Bayesian Knowledge Tracing and the Individual Difference Weights variant of Bayesian Knowledge Tracing (Corbett & Anderson, 1995), investigating how well each model variant predicts post-test performance. We also test other ways to utilize contextual estimation of slipping within the tutor in post-test prediction, and discuss hypotheses for why slipping during tutor use is a significant predictor of post-test performance, even after Bayesian Knowledge Tracing estimates are controlled for.
Conference Paper
A common personalization approach in educational systems is mastery learning. A key step in this approach is a criterion that determines whether a learner has achieved mastery. We thoroughly analyze several mastery criteria for the basic case of a single well-specified knowledge component. For the analysis we use experiments with both simulated and real data. The results show that the choice of data sources used for mastery decision and setting of thresholds are more important than the choice of a learner modeling technique. We argue that a simple exponential moving average method is a suitable technique for mastery criterion and propose techniques for the choice of a mastery threshold.
Conference Paper
Knowledge Tracing (KT) is a task of tracing evolving knowledge state of students with respect to one or more concepts as they engage in a sequence of learning activities. One important purpose of KT is to personalize the practice sequence to help students learn knowledge concepts efficiently. However, existing methods such as Bayesian Knowledge Tracing and Deep Knowledge Tracing either model knowledge state for each predefined concept separately or fail to pinpoint exactly which concepts a student is good at or unfamiliar with. To solve these problems, this work introduces a new model called Dynamic Key-Value Memory Networks (DKVMN) that can exploit the relationships between underlying concepts and directly output a student's mastery level of each concept. Unlike standard memory-augmented neural networks that facilitate a single memory matrix or two static memory matrices, our model has one static matrix called key, which stores the knowledge concepts and the other dynamic matrix called value, which stores and updates the mastery levels of corresponding concepts. Experiments show that our model consistently outperforms the state-of-the-art model in a range of KT datasets. Moreover, the DKVMN model can automatically discover underlying concepts of exercises typically performed by human annotations and depict the changing knowledge state of a student.
Article
A control chart is considered for the problem of monitoring a process when all items from the process are inspected and classified into one of two categories. The objective is to detect changes in the proportion, p, of items in the first category. The control chart being considered is a cumulative sum (CUSUM) chart based on the Bernoulli observations corresponding to the inspection of the individual items. Bernoulli CUSUM charts can be constructed to detect increases in p, decreases in p, or both increases and decreases in p. The properties of the Bernoulli CUSUM chart are evaluated using exact Markov chain methods and by using a corrected diffusion theory approximation. The corrected diffusion theory approximation provides a relatively simple method of designing the chart for practical applications. It is shown that the Bernoulli CUSUM chart will detect changes in p substantially faster than the traditional approach of grouping items into samples and applying a Shewhart p-chart. The Bernoulli CUSUM chart is also better than grouping items into samples and applying a CUSUM chart to the sample statistics. The Bernoulli CUSUM chart is equivalent to a geometric CUSUM chart which is based on counting the number of items in the second category that occur between items in the first category.
Article
The initial vision for intelligent tutoring systems involved powerful, multi-faceted systems that would leverage rich models of students and pedagogies to create complex learning interactions. But the intelligent tutoring systems used at scale today are much simpler. In this article, I present hypotheses on the factors underlying this development, and discuss the potential of educational data mining driving human decision-making as an alternate paradigm for online learning, focusing on intelligence amplification rather than artificial intelligence.
Conference Paper
The concept of mastery learning is powerful: rather than a fixed number of practices, students continue to practice a skill until they have mastered it. However, an implicit assumption in this formulation is that students are capable of mastering the skill. Such an assumption is crucial in computer tutors, as their repertoire of teaching actions may not be as effective as commonly believed. What if a student lacks sufficient knowledge to solve problems involving the skill, and the computer tutor is not capable of providing sufficient instruction? This paper introduces the concept of “wheel-spinning;” that is, students who do not succeed in mastering a skill in a timely manner. We show that if a student does not master a skill in ASSISTments or the Cognitive Tutor quickly, the student is likely to struggle and will probably never master the skill. We discuss connections between such lack of learning and negative student behaviors such as gaming and disengagement, and discuss alterations to ITS design to overcome this issue.
Article
The spacing and effectiveness functions of a quality control chart used either alone or in sets of two or more are derived for production at a constant level and for erratic production. The spacing of decision points is considered from a general point of view. The theory developed is fundamental in deciding which of two different decision techniques in quality control, each using the same spacing function, is the more effective.* Many of the results given herewith were obtained independently by the two authors.
Article
We treat a dynamic programming problem concerned with an application of tailoring programmed instruction to the individual student. We use a model of learning based on stimulus-sampling theory in which a subject is to be taught n items in the course of N trials. The problem is to determine a strategy of trial-by-trial item selection to maximize the expected terminal level of achievement of the subject; a trial consists of a test on a selected item followed by a reinforcement or teaching action relative to the item. A subject is either in the "conditioned" or "unconditioned" state with respect to an item. His response to a test is either correct or incorrect, and the probability of a correct response depends upon his state; thus, the state is not in exact correspondence with the response. The reinforcement action permits a probabilistic transition from the unconditioned to the conditioned state during a trial. States are not observable; a strategy is based upon the history of responses to items presented up to the current trial. Associated with a subject is a current state probability vector (\lambda<sub>1</sub>, \lambda<sub>2</sub>,..., \lambda<sup>n</sup>), \lambda<sub>i</sub>- probability of conditioned state relative to item i, given the subject's history to date. We prove that the following (locally optimal) strategy is (globally) optimal: In each trial, present any item for which the current probability of the conditioned state is least.
Article
The proposal is made to consider a paired-associate item as becoming conditioned to its correct response in all-or-none fashion, and that prior to this conditioning event the subject guesses responses at random to an unlearned item. These simple assumptions enable the derivation of an extensive number of predictions about paired-associate learning. The predictions compare very favorably with the results of an experiment discussed below.
Article
This paper describes an effort to model students' changing knowledge state during skill acquisition. Students in this research are learning to write short programs with the ACT Programming Tutor (APT). APT is constructed around a production rule cognitive model of programming knowledge, called theideal student model. This model allows the tutor to solve exercises along with the student and provide assistance as necessary. As the student works, the tutor also maintains an estimate of the probability that the student has learned each of the rules in the ideal model, in a process calledknowledge tracing. The tutor presents an individualized sequence of exercises to the student based on these probability estimates until the student has mastered each rule. The programming tutor, cognitive model and learning and performance assumptions are described. A series of studies is reviewed that examine the empirical validity of knowledge tracing and has led to modifications in the process. Currently the model is quite successful in predicting test performance. Further modifications in the modeling process are discussed that may improve performance levels.
Article
The ACT Programming Tutor (APT) is a problem solving environment constructed around an executable cognitive model of the programming knowledge students are acquiring.
How Khan Academy is using machine learning to assess student mastery
  • D Hu
Defining mastery: knowledge tracing versus n-consecutive correct responses
  • K M Kelly
  • Y Wang
  • T Thompson
  • N T Heffernan
The Practice of Teaching in the Secondary School
  • H C Morrison
  • HC Morrison