Article

Skill acquisition and the LISP Tutor

Authors:
To read the full-text of this research, you can request a copy directly from the authors.

Abstract

An analysis of student learning with the LISP tutor indicates that while LISP is complex, learning it is simple. The key to factoring out the complexity of LISP is to monitor the learning of the 500 productions in the LISP tutor which describe the programming skill. The learning of these productions follows the power-law learning curve typical of skill acquisition. There is transfer from other programming experience to the extent that this programming experience involves the same productions. Subjects appear to differ only on the general dimensions of how well they acquire the productions and how well they retain the productions. Instructional manipulations such as remediation, content of feedback, and timing of feedback are effective to the extent they give students more practice programming, and explain to students why correct solutions work.

No full-text available

Request Full-text Paper PDF

To read the full-text of this research,
you can request a copy directly from the authors.

... The tradeoff has recently been called the 'assistance dilemma' [31]. To resolve this dilemma, some results of earlier studies suggest that on-demand graded hints, with each giving progressively more specific advice until the learners judge they accomplished the task, might be effective for learning [39], [42], [43], [44], [45], [46], [47], [48], [49], [50], [51], [52]. ...
... Variants of this technique are used in a number of ITS fields (e.g., [38], [39], [40], [41]). However, some ITS researchers have pointed out that the adaptive help function tends to provide over-assistance and that it might obstruct effective learning [39], [42], [43], [44], [45], [46], [47]. A tradeoff exists between information giving and withholding to achieve optimal learning, which has recently been called the "assistance dilemma' [31]. ...
... A tradeoff exists between information giving and withholding to achieve optimal learning, which has recently been called the "assistance dilemma' [31]. Anderson et al. (1989) conducted a study that evaluated the effects of the tutor's mastery learning method and of explanatory content in both the tutor's hints and its feedback messages [42]. They compared the regular Lisp Tutor, which provides explanatory content in its hints and in some of its error feedback messages, with a version that simply told students they were wrong when they made errors, or gave them the correct answer when they requested a hint. ...
Article
Full-text available
Over the past few decades, many studies conducted in the field of learning science have described that scaffolding plays an important role in human learning. To scaffold a learner efficiently, a teacher should predict how much support a learner must have to complete tasks and then decide the optimal degree of assistance to support the learner's development. Nevertheless, it is difficult to ascertain the optimal degree of assistance for learner development. For this study, it is assumed that optimal scaffolding is based on a probabilistic decision rule: given a teacher's assistance to facilitate the learner development, an optimal probability exists for a learner to solve a task. To ascertain that optimal probability, we developed a scaffolding system that provides adaptive hints to adjust the predictive probability of the learner's successful performance to the previously determined certain value, using a probabilistic model, i.e., item response theory (IRT). Furthermore, using the scaffolding system, we compared learning performances by changing the predictive probability. Results show that scaffolding to achieve 0.5 learner success probability provides the best performance. Additionally, results demonstrate that a scaffolding system providing 0.5 probability decreases the number of hints (amount of support) automatically as a fading function according to the learner's growth capability.
... Programming tutors have been built since the 1970s, for programming languages such as Lisp (Anderson et al., 1986), Prolog (Hong, 2004), Java (Holland et al., 2009), Pascal (Johnson and Soloway, 1985), C (Wang et al., 2011), and many more. None of these tutors supports automatic feedback on the incremental development of programs for class 3 exercises, in which teachers can easily add programming exercises and adapt feedback. ...
... None of these tutors supports automatic feedback on the incremental development of programs for class 3 exercises, in which teachers can easily add programming exercises and adapt feedback. For example, the Lisp tutor (Anderson et al., 1986) does give feedback on the steps a student takes towards a solution to a programming exercise, and allows students to solve a program flexibly in that students do not have to follow a strict top to bottom, left to right order (Corbett et al., 1988), but it o↵ers class 2 exercises. Also, adding an exercise to the Lisp tutor is non-trivial. ...
... Tools that support students in learning programming have been developed since the 1960s (Ulloa, 1980;Douce et al., 2005). Some of these tools analyse incremental steps a student takes (Anderson et al., 1986), and/or support the development of programming plans (Soloway, 1985;Johnson and Soloway, 1985). The early work in this area primarily targeted the programming languages Lisp and Pascal (including Pascal-like imperative languages); a nice overview is given by Vanneste et al. (1996). ...
Article
Full-text available
Ask-Elle is a tutor for learning the higher-order, strongly-typed functional programming language Haskell. It supports the stepwise development of Haskell programs by verifying the correctness of incomplete programs, and by providing hints. Programming exercises are added to Ask-Elle by providing a task description for the exercise, one or more model solutions, and properties that a solution should satisfy. The properties and model solutions can be annotated with feedback messages, and the amount of flexibility that is allowed in student solutions can be adjusted. The main contribution of our work is the design of a tutor that combines (1) the incremental development of different solutions in various forms to a programming exercise with (2) automated feedback and (3) teacher-specified programming exercises, solutions, and properties. The main functionality is obtained by means of strategy-based model tracing and property-based testing. We have tested the feasibility of our approach in several experiments, in which we analyse both intermediate and final student solutions to programming exercises, amongst others.
... On the other hand, some early programming tutors have explored domain modelling on the level of integrative skills by using relatively complex production rules or algorithmic patterns (Anderson et al., 1989;Spohrer, 1992;Weber, 1996), and provided initial evidence of the effectiveness of such integration-level representations in supporting learning. For example, Weber (1996) represented programming skills as algorithmic patterns stored in a hierarchy of episodic frames and associated concepts. ...
... Although they allow for a more active learning process, learning-by-doing, such tools often only offer support at a coarse-grained problem level whereby students get feedback relative to the final output or final value of a variable of a code-tracing problem (Hsiao et al., 2010;Shah et al., 2021;Thomas et al., 2019) or the final submission of code (Ihantola et al., 2010), limiting their effectiveness for supporting learning. Anderson et al. (1989), in their experiments with the LISP programming tutor, found that students who received step-level support (i.e., error feedback and hints) learned more than students with program solution evaluation only at the end of each problem, and they did so in one third the time. ...
Article
Full-text available
Background Skill integration is vital in students' mastery development and is especially prominent in developing code tracing skills which are foundational to programming, an increasingly important area in the current STEM education. However, instructional design to support skill integration in learning technologies has been limited. Objectives The current work presents the development and empirical evaluation of instructional design targeting students' difficulties in code tracing particularly in integrating component skills in the Trace Table Tutor (T3), an intelligent tutoring system. Methods Beyond the instructional features of active learning, step‐level support, and individualized problem selection of intelligent tutoring systems (ITS), the instructional design of T3 (e.g., hints, problem types, problem selection) was optimized to target skill integration based on a domain model where integrative skills were represented as combinations of component skills. We conducted an experimental study in a university‐level introductory Python programming course and obtained three findings. Results and Conclusions First, the instructional features of the ITS technology support effective learning of code tracing, as evidenced by significant learning gains (medium‐to‐large effect sizes). Second, performance data supports the existence of integrative skills beyond component skills. Third, an instructional design focused on integrative skills yields learning benefits beyond a design without such focus, such as improving performance efficiency (medium‐to‐large effect sizes). Major Takeaways Our work demonstrates the value of designing for skill integration in learning technologies and the effectiveness of the ITS technology for computing education, as well as provides general implications for designing learning technologies to foster robust learning.
... AISs have shown promise to fill this gap. For instance, Anderson et al. (1989) demonstrated how their adaptive LISP tutor could benefit learning programming skills (or "production rules") to create working LISP code. The tutor provided remedial instruction when students were incorrect and customized the feedback according to the type of error when possible. ...
... A critical complement to learner models' knowledge estimates are the PDRs that dictate, for the purpose of the instructional sequence decisions, what should be practiced based on the learner model predictions (Katz & Albacete, 2013). For instance, Anderson et al. (1989) had students proceed to a new KC once their learner model predicted the probability of solving problems for a KC exceeded 95%. Some AIS do not express PDR as probabilities (Heffernan & Heffernan, 2014;Canfield, 2001) instead opting for rules-of-thumb such as dropping content from practice once the student has correctly answered three times in a row. ...
Article
Full-text available
An important component of many Adaptive Instructional Systems (AIS) is a ‘Learner Model’ intended to track student learning and predict future performance. Predictions from learner models are frequently used in combination with mastery criterion decision rules to make pedagogical decisions. Important aspects of learner models, such as learning rate and item difficulty, can be estimated from prior data. A critical function of AIS is to have students practice new content once the AIS predicts that they have ‘mastered’ current content or learned it to some criterion. For making this prediction, individual student parameters (e.g., for learning rate) are frequently unavailable due to having no prior data about a student, and thus population-level parameters or rules-of-thumb are typically applied instead. In this paper, we will argue and demonstrate via simulation and data analysis that even in best-case scenarios, learner models assuming equal learning rates for students will inevitably lead to systematic errors that result in suboptimal pedagogical decisions for most learners. This finding leads us to conclude that systematic errors should be expected, and mechanisms to adjust predictions to account for them should be included in AIS. We introduce two solutions that can adjust for student differences “online” in a running system: one that tracks systemic errors of the learner model (not the student) and adjusts accordingly, and a student-level performance adaptive feature. We demonstrate these solutions’ efficacy and practicality on six large educational datasets and show that these features improved model accuracy in all tested datasets.
... Much of this research has focused on specific skills. For example, researchers have captured low-level programming knowledge as part of learning technologies (Anderson et al. 1989). A long history of work has theorized about program comprehension skills, describing the bottom up, top down, and opportunistic strategies that developers use at a high level (Roehm et al. 2012;Von Mayrhauser and Vans 1995). ...
... For example, studies have shown that the use of explicit slicing strategies in debugging (Francel and Rugaber 2001), the use of explicit strategies in tracing program execution (Xie et al. 2018), and the use of explicit strategies to extract requirements from problem statements (Haidry et al. 2017), are either correlated with or cause decreases in task completion time or increases in task performance. Early research on the LISP tutor (Anderson et al. 1989) and software design environments (Rist 1995) similarly showed that by defining expert problem solving strategies, and nudging novices to follow those expert strategies through hints and feedback, novices could approach expert performance. Many have also described explicit strategies for debugging; Metzger, for example, describes debugging strategies as a high-level plan for accomplishing a goal through a sequence of physical and cognitive actions (Metzger 2004) and Zeller presents a series of formal and informal procedures for isolating causes and effects of defects (Zeller 2009). ...
Article
Full-text available
Software developers solve a diverse and wide range of problems. While software engineering research often focuses on tools to support this problem solving, the strategies that developers use to solve problems are at least as important. In this paper, we offer a novel approach for enabling developers to follow explicit programming strategies that describe how an expert tackles a common programming problem. We define explicit programming strategies, grounding our definition in prior work both within software engineering and in other professions which have adopted more explicit procedures for problem solving. We then present a novel notation called Roboto and a novel strategy tracker tool that explicitly represent programming strategies and frame executing strategies as a collaborative effort between human abilities to make decisions and computer abilities to structure process and persist information. In a formative evaluation, 28 software developers of varying expertise completed a design task and a debugging task. We found that, compared to developers who are free to choose their own strategies, developers given explicit strategies experienced their work as more organized, systematic, and predictable, but also more constrained. Developers using explicit strategies were objectively more successful at the design and debugging tasks. We discuss the implications of Roboto and these findings, envisioning a thriving ecosystem of explicit strategies that accelerate and improve developers’ programming problem solving.
... Much of this research has focused on specific skills. For example, researchers have captured low-level programming knowledge as part of learning technologies [3]. A long history of work has theorized about program comprehension skills, describing the bottom up, top down, and opportunistic strategies that developers use at a high level [59,69]. ...
... For example, studies have shown that the use of explicit slicing strategies in debugging [21], the use of explicit strategies in tracing program execution [71], and the use of explicit strategies to extract requirements from problem statements [27], are either correlated with or cause decreases in task completion time or increases in task performance. Early research on the LISP tutor [3] and software design environments [56] similarly showed that by defining expert problem solving strategies, and nudging novices to follow those expert strategies through hints and feedback, novices could approach expert performance. Many have also described explicit strategies for debugging; Metzger, for example, describes debugging strategies as a high-level plan for accomplishing a goal through a sequence of physical and cognitive actions [50] and Zeller presents a series of formal and informal procedures for isolating causes and effects of defects [73]. ...
Preprint
Full-text available
Software developers solve a diverse and wide range of problems. While software engineering research often focuses on tools to support this problem solving, the strategies that developers use to solve problems are at least as important. In this paper, we offer a novel approach for enabling developers to follow explicit programming strategies that describe how an expert tackles a common programming problem. We define explicit programming strategies, grounding our definition in prior work both within software engineering and in other professions which have adopted more explicit procedures for problem solving. We then present a novel notation called Roboto and a novel StrategyTracker tool that explicitly represents programming strategies and frame executing strategies as a collaborative effort between human abilities to make decisions and computer abilities to structure process and persist information. Ina formative evaluation, 28 software developers of varying expertise completed a design task and a debugging task. We found that, compared to developers who are free to choose their strategies, developers gave explicit strategies experienced their work as more organized, systematic, and predictable, but also more constrained. Developers using explicit strategies were objectively more successful at the design and debugging tasks. We discuss the implications of Roboto and these findings, envisioning a thriving ecosystem of explicit strategies that accelerate and improve developers programming problem solving.
... The second coder and a third coder (research intern) then independently categorized the interactions into the existing categories. We calculated a Fleiss' 3 The five interactions were excluded either because of errors with recording (2) or prominent unconsented student voices (3) 4 Grounded theory analysis contends that we can understand the content in communication by analyzing the type of "common ground" shared by the speakers. This standard qualitative analysis method, like most qualitative research, relies on two or more "coders" who categorize interactions according to the given property being studied and the research goals. ...
... ITS, which allow students to obtain real-time feedback while working on problems as part of a pre-specified curriculum, have been successful in domains such as K-12 math [5] and programming. The LISP tutor [3] has specific relevance to this work, as we use the Scheme programming language in our prototype implementation. Additional work has augmented ITS feedback by using real student data for Python programming [33]. ...
Conference Paper
Students learning to program often need help completing assignments and understanding why their code does not work as they expect it to. One common place where they seek such help is at teaching assistant office hours. We found that teaching assistants in introductory programming (CS1) courses frequently answer some variant of the question ``Am I on the Right Track?''. The goal of this work is to develop an automated tool that provides similar feedback for students in real-time from within an IDE as they are writing their program. Existing automated tools lack the generality that we seek, often assuming a single approach to a problem, using hand-coded error models, or applying sample fixes from other students. In this paper, we explore the use of program synthesis to provide less constrained automated answers to ``Am I on the Right Track'' (AIORT) questions. We describe an observational study of TA-student interactions that supports targeting AIORT questions, as well as the development of and design considerations behind a prototype integrated development environment (IDE). The IDE uses an existing program synthesis engine to determine if a student is on the right track and we present pilot user studies of its use.
... Digital technologies, such as Intelligent Tutoring Systems (ITS), can be used to provide learners with different learning interventions such as scaffolding, feedback, etc., hence supporting them with the deliberate practice of their skills. An example is the classic LISP tutor that helps learners to become acquainted with the LISP programming language [4]. Traditional ITSs have shown positive effects supporting skills in wellformed topics and knowledge domains [5] such as mathematics [6] or programming languages [4,7]. ...
... An example is the classic LISP tutor that helps learners to become acquainted with the LISP programming language [4]. Traditional ITSs have shown positive effects supporting skills in wellformed topics and knowledge domains [5] such as mathematics [6] or programming languages [4,7]. Learners can practice this type of skills while directly interacting with the ITS through precise and unambiguous operations (e.g., keyboard strokes, button clicks). ...
Article
Full-text available
The development of multimodal sensor-based applications designed to support learners with the improvement of their skills is expensive since most of these applications are tailor-made and built from scratch. In this paper, we show how the Presentation Trainer (PT), a multimodal sensor-based application designed to support the development of public speaking skills, can be modularly extended with a Virtual Reality real-time feedback module (VR module), which makes usage of the PT more immersive and comprehensive. The described study consists of a formative evaluation and has two main objectives. Firstly, a technical objective is concerned with the feasibility of extending the PT with an immersive VR Module. Secondly, a user experience objective focuses on the level of satisfaction of interacting with the VR extended PT. To study these objectives, we conducted user tests with 20 participants. Results from our test show the feasibility of modularly extending existing multimodal sensor-based applications, and in terms of learning and user experience, results indicate a positive attitude of the participants towards using the application (PT+VR module).
... When detected, a relevant feedback message was delivered (either on-demand when the code was executed or while the student was editing code) using BlockPy's built-in feedback mechanisms that are normally used to deliver runtime and output errors. We chose immediate feedback delivery because of its acquisitional efficiency for verbal knowledge and procedural skills [2,6,9,11]. In developing the feedback message, the instructors aimed to explain the misconception found and the corresponding mistake's location, rather than how the student should fix the mistake. ...
... The second post-test contained two programming problems -each related to one of the learning objectives. The statement of the three programming problems are:(1) The data block in the BlockPy canvas below provides a list of the number of students taking the 2015 SAT test in each state.Write an algorithm to compute and print the total number of students taking the SAT test in 2015.(2) The data block in the BlockPy canvas below provides a list of the per capita income of each state. ...
Conference Paper
Full-text available
The feedback given to novice programmers can be substantially improved by delivering advice focused on learners' cognitive misconceptions contextualized to the instruction. Building on this idea, we present Misconception-Driven Feedback (MDF); MDF uses a cognitive student model and program analysis to detect mistakes and uncover underlying misconceptions. To evaluate the impact of MDF on student learning, we performed a quasi-experimental study of novice programmers that compares conventional run-time and output check feedback against MDF over three semesters. Inferential statistics indicates MDF supports significantly accelerated acquisition of conceptual knowledge and practical programming skills. Additionally, we present descriptive analysis from the study indicating the MDF student model allows for complex analysis of student mistakes and misconceptions that can suggest improvements to the feedback, the instruction, and to specific students.
... Most of this work has not investigated the teaching of programming problem-solving, although there are some exceptions. The LISP tutor built models of the program solution space and monitored learners' traversals through this space, intervening with corrective feedback if learners encountered error states or made mistakes the tutor had previously observed [3]. Other more recent efforts to teach problem-solving to programmers have found that having teachers prompt novices about the strategies they are using and whether those strategies are appropriate and effective can greatly improve novices' abilities to problem-solve independently [39]. ...
... However, the experimental group received four interventions that the control group did not: (1) A short lecture about problem solving and metacognition in programming; (2) A set of prompts to be answered when campers asked helpers for assistance; (3) a physical, paper handout of a problem solving model; and (4) the Idea Garden prototype for Cloud9. As we explained above, the Idea Garden prototype supported anti-patterns observed in previous studies as well as problem-solving strategy hints consistent with interventions (1)- (3). ...
Article
Many systems are designed to help novices who want to learn programming, but few support those who are not necessarily interested in learning programming. This paper targets the subset of end-user programmers (EUPs) in this category. We present a set of principles on how to help EUPs like this learn just a little when they need to overcome a barrier. We then instantiate the principles in a prototype and empirically investigate them in three studies: a formative think-aloud study, a pair of summer camps attended by 42 teens, and a third summer camp study featuring a different environment attended by 48 teens. Finally, we present a generalized architecture to facilitate the inclusion of Idea Gardens into other systems, illustrating with examples from Idea Garden prototypes. Results have been very encouraging. For example, under our principles, Study #2’s camp participants required significantly less in-person help than in a previous camp to learn the same amount of material in the same amount of time.
... Although we have made these investigations and conclusions in the context of a relatively simple and artificial task, we think the same mix of learning processes are involved in more complex problem-solving tasks. Indeed, the task used in these experiments was originally motivated to study more systematically the phenomena we observed in LISP programming (Anderson, Conrad, & Corbett, 1989) where we have found similar effects of directional asymmetry and problem repetition. ...
Article
Full-text available
In 3 experiments, participants memorized 8 examples, each exemplifying a different rule. Participants were asked to extend these rules to new examples. They practiced applications of the rules to examples over a period of 4 days (Experiment 1) or 5 days (Experiments 2 and 3). Although these rules were bidirectional, an asymmetry gradually built up such that participants became more facile in using the rules in the practiced direction. Participants also showed an advantage when the initial study example was repeated or when test examples were repeated. It is argued that skill acquisition involves development of a complex set of strategies based on use of rules and retrieval of examples. Four overlapping stages of skill acquisition are described.
... 6 Different levels of transfer are achieved based on what elements are and are not identical. Cognitive abstractions allow one to specify the elements of cognitive tasks (Anderson et al. , 1989;Bovair et al. 1990). Positive transfer is predicted by holding constant elements of cognitive tasks across learning and performance situations. ...
... Process-level data is likely to help understand these differences better. A key theoretical assumption that has emerged from past ITS research is that eventually-correct steps are instrumental in student learning from deliberate practice [12,21,36]. Following past research, we consider a competency to be learned as comprising multiple Knowledge Components (KCs) [22]. ...
Chapter
Full-text available
In numerous studies, intelligent tutoring systems (ITSs) have proven effective in helping students learn mathematics. Prior work posits that their effectiveness derives from efficiently providing eventually-correct practice opportunities. Yet, there is little empirical evidence on how learning processes with ITSs compare to other forms of instruction. The current study compares problem-solving with an ITS versus solving the same problems on paper. We analyze the learning process and pre-post gain data from N = 97 middle school students practicing linear graphs in three curricular units. We find that (i) working with the ITS, students had more than twice the number of eventually-correct practice opportunities than when working on paper and (ii) omission errors on paper were associated with lower learning gains. Yet, contrary to our hypothesis, tutor practice did not yield greater learning gains, with tutor and paper comparing differently across curricular units. These findings align with tutoring allowing students to grapple with challenging steps through tutor assistance but not with eventually-correct opportunities driving learning gains. Gaming-the-system, lack of transfer to an unfamiliar test format, potentially ineffective tutor design, and learning affordances of paper can help explain this gap. This study provides first-of-its-kind quantitative evidence that ITSs yield more learning opportunities than equivalent paper-and-pencil practice and reveals that the relation between opportunities and learning gains emerges only when the instruction is effective.
... It is also the case that there is a vast middle ground of educational research in which the distinction between basic versus domain specifi c work is often blurred (Anderson, Conrad, & Corbett, 1989;Corbett & An derson, 1988;Singley & Anderson, 1989) . However, at the further extreme are the attempts to model rare forms of expertise, such as that possessed by Submarine Com manders (Ehret, Gray, & Kirschenbaum, 2000;Gray, Kirschenbaum, & Ehret, 1997;Kirschen baum & Gray, 2000), uninhabited air ve hicle (UA V) operators (Gluck et al., 2007), or airline pilots (Byrne & Kirlik, 2005). ...
Chapter
Full-text available
The Cambridge Handbook of Computational Cognitive Sciences is a comprehensive reference for this rapidly developing and highly interdisciplinary field. Written with both newcomers and experts in mind, it provides an accessible introduction of paradigms, methodologies, approaches, and models, with ample detail and illustrated by examples. It should appeal to researchers and students working within the computational cognitive sciences, as well as those working in adjacent fields including philosophy, psychology, linguistics, anthropology, education, neuroscience, artificial intelligence, computer science, and more.
... Model-tracing tutors guide students as they solve complex problems, that is, problems that have multiple possible solution paths, each with multiple steps. Many modeltracing tutors have been described in the AIED literature, including Cognitive Tutors for middle-school and high-school mathematics (Koedinger & Corbett, 2006), the Genetics Tutor (Corbett et al., 2010), Cognitive Tutors for Lisp, Pascal, and Prolog programming (Anderson et al., 1989;Anderson et al., 1993), Lynnette (middle-school equation solving; Vincent Aleven, Jonathan Rowe, Yun Huang, and Antonija Mitrovic -9781800375413 Downloaded from PubFactory at 05/24/2023 08:10:02AM via communal account Long & Aleven, 2014), MATHia (middle-and high-school mathematics; Ritter et al., 2007), Andes (physics;VanLehn et al., 2005), SlideTutor (skin pathology; Crowley & Medvedeva, 2006), and MATHESIS (high-school algebra; Sklavakis & Refanidis, 2013). Model-tracing tutors for mathematics learning are being used widely in American mathematics learning (Ritter et al., 2007). ...
Chapter
=== Feel free to contact me (Yun) through full-text request in ResearchGate! === A central component of many AIED systems is a “domain model,” that is, a representation of knowledge of the domain of instruction. The system uses the model in many ways to provide instruction that adapts to learners. Not all AIED systems have an elaborate domain model, but in those that do, the domain model is central to the system’s functioning. In fact, domain models fulfill so many important functions within AIED systems that entire classes of AIED systems are defined in terms of the types of domain model they use (such as model-tracing tutors, constraint-based tutors, example-tracing tutors, and issue-based approaches to build- ing tutoring systems). Across AIED projects, systems, and paradigms, the types of domain models used span the gamut of AI representations. AIED systems use their domain models for many different purposes, chief among them assessing student work, which is foundational for other functionality. This chapter reviews major approaches to domain modeling used in AIED systems and briefly touches on the corresponding student models and the way they are used to track an individual student’s knowledge growth. (We do not discuss student models that target other aspects, such as affect, motivation, self-regulation, or metacognition.) We discuss, in turn: rule-based models, constraint-based models, Bayesian networks, machine-learned models, text-based models, generalized examples, and knowledge spaces. These types of models have been studied extensively in AIED research and have been the foundation for many AIED systems that have been proven to be effective in enhancing student learning or other aspects of the student experience. A number of these approaches are now used in AIED systems that are used on a wide scale in educational practice. The chapter discusses how these approaches support key aspects of an AIED system’s behavior and enable the system to adapt aspects of its instruction to individual student variables. We also highlight challenges that occur when applying the different approaches. We look at the use of machine learning and data-driven methods to create or refine domain models, so they better account for learning data and sup- port more effective adaptive instruction. As well, we make note of connections between a system’s domain model and other key components, including the system’s student model. We base this discussion on traditional views of intelligent tutoring systems (ITSs), which divide the system’s architecture into four main components: a domain model, a student model, a pedagogical model, and a problem-solving environment. We focus on systems that support individual learning. Other types of AIED systems are covered in other chapters.
... Feedback that explains why that choice is correct or incorrect is provided for each answer option to give additional guidance and another opportunity for learning ( Figure 2). Immediate, targeted feedback was shown to reduce the time it took students to reach a desired outcome [2] [13], and feedback in practice testing outperforms no-feedback testing [5] [15]. Formative practice with targeted feedback provides scaffolding and examples that support cognitive structures for effective learning [9] [15] [18]. ...
Article
Full-text available
The doer effect is a learning science principle that proves students who engage with formative practice at the point of learning have higher learning gains than those who only read expository text or watch video. This principle has been demonstrated through both correlational and causal analysis. It is imperative that learning science approaches capable of increasing student learning gains be rigorously tested and replicated to confirm their validity before wide-scale use. Previously we replicated causal doer effect results using student data from courseware used at a major online university. In this paper, we will replicate both the correlational doer effect analysis as well as the causal analysis using both unit tests from the courseware and the course final exam. These multiple analyses of the doer effect on the same course data provide a unique comparison of this method and the impact of the doer effect on near and intermediate learning assessments. Findings of the correlational doer effect analyses confirmed doing was more significant to outcomes than reading, and further analysis determined these results could not be attributed to student characteristics. Results of the causal analysis verified doing was causal to learning on both the unit tests and final exam. The implications of these doer effect replication results and future research will be discussed.
... The introduction of AI in training and development has accelerated the personalized learning experiences among employees. An AI-enabled intelligent tutoring system (ITS) preexisted since 1980s which was used at the college level and in the military services [40,41]. More than two eras ago, experts have discovered that intelligent systems entrenched with techniques built on cognitive science, might drastically enhance the learning results in high school students who were learning algebra [42]. ...
... As it quickly became apparent that the availability of hints did not ensure their effective use, work began to identify the factors that led to a positive relationship between help-seeking behaviors and student learning. In one of the earliest studies, Anderson et al., (1989) compared the use of explanatory hints and so-called bottom-out hints (which simply provided the student with the correct answer) and found that neither hint type was correlated with learning. In part, this may have been due to selection bias. ...
Article
Full-text available
Educational technology (EdTech) designers need to ensure population validity as they attempt to meet the individual needs of all students. EdTech researchers often have access to larger and more diverse samples of student data to test replication across broad demographic contexts as compared to either the small-scale experiments or the larger convenience samples often seen in experimental psychology studies of learning. However, the source of typical EdTech data (i.e., online learning systems) and concerns related to student privacy often limit the opportunity to collect demographic variables from individual students—the sample is diverse, but the researcher does not know how that diversity is realized in individual learners. In order to ensure equitable student outcomes, the EdTech community should make greater efforts to develop new methods for addressing this shortcoming. Recent work has sought to address this issue by investigating publicly-available, school-level differences in demographics, which can be useful when individual-level variation may be difficult or impossible to acquire data for. In this study, we use this approach to better understand the role of social factors in students’ self-regulated learning and motivation-related behaviors, behaviors whose effectiveness appears to be highly variable between groups. We demonstrate that school-level demographics can be significantly associated with the relationships between students’ help-seeking behavior, motivation, and outcomes (math performance and math self-concept). We do so in the context of reasoning mind, an intelligent tutoring system for elementary mathematics. By studying the conditions under which these relationships vary across different demographic contexts, we challenge implicit assumptions of generalizability and provide an evidence-based commentary on future research practices in the EdTech community surrounding how we consider diversity in our field’s investigations.
... The acquisition of skill is divided in three overlapping phases: (1) knowledge acquisition, (2) knowledge association, and (3) autonomous task performance [18], [19]. For programming skill, we use the definition of Bergersen et al. [19], which is in accordance to the definition used in psychology [18], [20], [21], [22], [23]: "the ability to use one's knowledge effectively and readily in execution or performance of programming tasks." In previous work with programmers, participants were often expected to have programming skill in various programming languages such as Python, Java, C, Perl, Haskell, JavaScript, PHP, etc. (e.g., [3], [7], [9], [10], [15], [19], [24], [25], [26], [27]). ...
Preprint
Recruiting professional programmers in sufficient numbers for research studies can be challenging because they often cannot spare the time, or due to their geographical distribution and potentially the cost involved. Online platforms such as Clickworker or Qualtrics do provide options to recruit participants with programming skill; however, misunderstandings and fraud can be an issue. This can result in participants without programming skill taking part in studies and surveys. If these participants are not detected, they can cause detrimental noise in the survey data. In this paper, we develop screener questions that are easy and quick to answer for people with programming skill but difficult to answer correctly for those without. In order to evaluate our questionnaire for efficacy and efficiency, we recruited several batches of participants with and without programming skill and tested the questions. In our batch 42% of Clickworkers stating that they have programming skill did not meet our criteria and we would recommend filtering these from studies. We also evaluated the questions in an adversarial setting. We conclude with a set of recommended questions which researchers can use to recruit participants with programming skill from online platforms.
... Given the challenges related to learning to program, some early work focused on identifying expert programmers' mental representations (Gilmore and Green 1988), and how these representations could be incorporated into teaching activities to help novices learn (Schank et al. 1993;Soloway 1986). There was also interest in computational support for programming activities delivered through tutoring systems (Anderson et al. 1989;Bhuiyan et al. 1994). Jim Greer and colleagues championed some of this work. ...
Article
When students are first learning to program, they not only have to learn to write programs, but also how to trace them. Code tracing involves stepping through a program step-by-step, which helps to predict the output of the program and identify bugs. Students routinely struggle with this activity, as evidenced by prior work and our own experiences in the classroom. To address this, we designed a Code Tracing (CT)-Tutor. We varied the level of assistance provided in the tutor, based on (1) the interface scaffolding available during code tracing, and (2) instructional order, operationalized by when examples were provided, either before or after the corresponding problem was solved. We collected data by having participants use the tutor to solve code tracing problems (N = 97) and analyzed both learning outcomes and process data obtained by extracting features of interest from the log files. We used a multi-layered approach for the analysis, including standard inferential statistics and unsupervised learning to cluster students by their behaviors in the tutor. The results show that the optimal level of assistance for code tracing falls in the middle of the assistance spectrum included in the tutor, but also that there are individual differences in terms of optimal assistance for subgroups of individuals. Based on these results, we outline opportunities for future work around personalizing instruction for code tracing.
... Such a design was found to encourage system gaming behaviors that harm learning [35], [36]. Although hint and feedback were used interchangeably in many computing education studies, the pedagogical differences between them are significant (e.g., Anderson et al [37] combined both feedback and hint in the LISP Tutor, but only used "feedback" to refer to both the facilitative functions). ...
... While I have found no reading-first program tracing tutoring systems, intelligent tutoring systems have included tracing practice without a full unified curriculum. Intelligent tutoring systems like ITEM/IP and a version of the LISP tutor included some tracing in their curriculum but interleaved with writing practice [2,3]. The Problets collection of tutors for mostly tracing practice lack conceptual instruction and a unified curriculum for a Turing-complete programming language [6]. ...
Conference Paper
Full-text available
Two large multinational studies show more than 60% of students incorrectly answer questions about the execution of basic programs. Past work has explored writing-focused curricula, but little work explores a reading-first pedagogy, teaching and assessing programming language semantics before teaching learners to write code. The hypothesis of my thesis is "An adaptive and motivating reading-first tutorial helps people learn program tracing skills and improves later learning of writing and debugging skills." Towards that goal, I built and evaluated a reading-first tutorial (PLTutor) with a fixed, non-adaptive curriculum, showing 60% higher learning gains (3.9 vs 2.4 on the Second CS1 Assessment (SCS1)) than the writing-focused tutorial Codecademy. I designed and did initial validity studies for a formative assessment for program tracing skills, based on a new formal knowledge model. Next, I will build an adaptive version of PLTutor and evaluate it. In the Koli DC I hope to refine my thesis' framing and narrative, get advice on my academic job search, and connect with and potentially help other PhD students and researchers.
... Templates are a form of program schema [16,20] that students can recall and reuse in constructing solutions to new problems. The LISP Tutor [1] builds on a theory that students can recognize and adapt solutions to recursive problems, though without an explicit step of articulating the template independently from the code. ...
Conference Paper
In many universities, Teaching Assistants (TAs) are an important part of students' educational experience. This is especially true in early courses, where students may suffer from inexperience and anxiety, and find fellow students more accessible than professors. Despite its importance, this learning channel has not been studied very much. Part of the difficulty lies in how to meaningfully evaluate it. Any intervention needs to be both unintrusive and lightweight, and yet yield useful data. As a result, to many faculty and researchers, TA office hours remain fairly opaque. This paper presents one approach to studying the technical component (but not the social dynamics) of TA office hours. We use a program-design methodology as a device to help track what students are asking about in hours, using a simple survey-based method to gather data. Data from TAs effectively summarize students' questions. In addition, contrasting data from both TAs and students provides insight into students' progress on program design help-seeking over the course of the semester.
... This is especially true in large classrooms and MOOCs, where instructors cannot interact with every student [17,21]. To address this, researchers have developed intelligent systems for automatically generating support for students during programming [2,13,16,31], such as hints and feedback, which have been shown to improve students' learning outcomes [8,9]. These systems often offer next step, edit-based hints, which suggest an edit that a student should make to bring their code closer to a correct solution [10,28]. ...
Conference Paper
Automated hints, a powerful feature of many programming environments, have been shown to improve students' performance and learning. New methods for generating these hints use historical data, allowing them to scale easily to new classrooms and contexts. These scalable methods often generate next-step, code hints that suggest a single edit for the student to make to their code. However, while these code hints tell the student what to do, they do not explain why, which can make these hints hard to interpret and decrease students' trust in their helpfulness. In this work, we augmented code hints by adding adaptive, textual explanations in a block-based, novice programming environment. We evaluated their impact in two controlled studies with novice learners to investigate how our results generalize to different populations. We measured the impact of textual explanations on novices' programming performance. We also used quantitative analysis of log data, self-explanation prompts, and frequent feedback surveys to evaluate novices' understanding and perception of the hints throughout the learning process. Our results showed that novices perceived hints with explanations as significantly more relevant and interpretable than those without explanations, and were also better able to connect these hints to their code and the assignment. However, we found little difference in novices' performance. Our results suggest that explanations have the potential to make code hints more useful, but it is unclear whether this translates into better overall performance and learning.
... The greatest learning on a skill occurs early on and lessens with additional practice (Newell & Rosenbloom, 1981). However, Anderson et al. (1989) found that this power relationship may not always hold true with complex skills. In general, learning curves can be used to demonstrate students' learning rates over repeated trials practicing a skill (Koedinger & Mathan, 2004). ...
Article
Full-text available
Abstract We have conducted a study on how many opportunities are necessary, on average, for learners to achieve mastery of a skill, also called a knowledge component (KC), as defined in the Open Learning Initiative (OLI) digital courseware. The study used datasets from 74 different course instances in four topic areas comprising 3813 students and 1.2 million transactions. The analysis supports our claim that the number of opportunities to reach mastery gives us new information on both students and the development of course components. Among the conclusions are a minimum of seven opportunities are necessary for each knowledge component, more if the prior knowledge among students are uneven within a course. The number of KCs in a course increases the number of opportunities needed. The number of opportunities to reach mastery can be used to identify KCs that are outliers that may be in need of better explanations or further instruction.
... Intelligent Tutoring Systems (ITSs) have been created for a variety of domains such as the military (Zachary et al., 1999), intelligent computer-assisted language learning (Gamper & Knapp, 2002) and education (Bradáč & Kostolányová, 2016). While they have been successful in providing personalized individual instruction in a variety of domains (e.g., programming, algebra, physics, and on-the-job training) (Anderson, 1989;Arroyo-Figueroa, Hernandez, & Sucar, 2006;Koedinger, 1997;VanLehn, van de Sande, Shelby, & Gershman, 2010), the complexity and difficulty of building an ITS has been well-documented (Murray, Blessing, & Ainsworth, 2003). Given the combinatorics of team member interactions and the need for a team tutor to track those interactions it is expected that building an intelligent team tutoring system (ITTS) is more complex. ...
... It is also seen in the Alef NextGen platform, a platform currently used by learners in the Middle East and New York City, which we will discuss below. Moreso, extended explanations have been a part of many successful intelligent tutoring systems, from the original LISP Tutor (Anderson, Conrad, & Corbett, 1989) to recent extensions to the ASSISTments system today . These systems represent a hybrid between continual learning/assessment and periodic learning/assessment, suggesting that the standard assumptions of most student modeling algorithms may also not hold in these cases. ...
Conference Paper
Full-text available
Student models for adaptive learning environments and intelligent tutoring systems typically assume a paradigm of use where a student completes exercises or activities, and learns from those exercises or activities. However, many modern systems, including MOOCs, intersperse declarative content or lecture with assessment of the learning from this content. In this paper, we present a variant of a common student modeling algorithm, Bayesian Knowledge Tracing, which assumes that most learning occurs during use of declarative content rather than between exercises. We compare this algorithm’s predictive ability to classic Bayesian Knowledge Tracing and another common algorithm, Performance Factors Assessment. We find that our new algorithm, BKT-PL, performs slightly better than algorithms designed for the standard intelligent tutoring paradigm. Moreso, we can use BKT-PL to determine which declarative content is most and least effective, to drive iterative re-design.
... While many students find solving Parsons problems engaging [34], students sometimes struggle to solve the problems, and some students give up without ever solving them [21]. Since learning gains are based on the number of practice problems that students solve and understand [1], scaffolding that allows students to solve more problems should improve learning. However, it is also important that practice problems challenge the learner. ...
Conference Paper
Practice is essential for learning. There is evidence that solving Parsons problems (putting mixed up code blocks in order) is a more efficient, but just as effective, form of practice than writing code from scratch. However, not all students successfully solve every Parsons problem. Making the problems adaptive, so that the difficulty changes based on the learner's performance, should keep the learner in Vygotsky's zone of proximal development and maximize learning gains. This paper reports on a study comparing the efficiency and effectiveness of learning from solving adaptive Parsons problems vs non-adaptive Parsons problem vs writing the equivalent code. The adaptive Parsons problems used both intra-problem and inter-problem adaptation. Intra-problem adaptation means that if the learner is struggling to solve the current problem, the problem can dynamically be made easier. Inter-problem adaptation means that the difficulty of the next problem is modified based on the learner's performance on the previous problem. This study provides evidence that solving intra-problem and inter-problem adaptive Parsons problems is a more efficient, but just as effective, form of practice as writing the equivalent code.
... al 2014;Ritter et. al 2007;Anderson, Conrad, & Corbett 1989). ...
Article
Full-text available
Students have a limited time to study and are typically ineffective at allocating study time. Machine-directed study strategies that identify which items need reinforcement and dictate the spacing of repetition have been shown to help students optimize mastery (Mozer & Lindsey 2017). The large volume of research on this matter is typically conducted in constructed experimental settings with fixed instruction, content, and scheduling; in contrast, we aim to develop methods that can address any demographic, subject matter, or study schedule. We show two methods that model item-specific recall probability for use in a discrepancy-reduction instruction strategy. The first model predicts item recall probability using a multiple logistic regression (MLR) model based on previous answer correctness and temporal spacing of study. Prompted by literature suggesting that forgetting is better modeled by the power law than an exponential decay (Wickelgren 1974), we compare the MLR approach with a Recurrent Power Law (RPL) model which adaptively fits a forgetting curve. We then discuss the performance of these models against study datasets comprised of millions of answers and show that the RPL approach is more accurate and flexible than the MLR model. Finally, we give an overview of promising future approaches to knowledge modeling.
... Computer tutors tried low-level evaluation exercises, but sequenced with writing exercises; this had no bene ts for writing skills vs. only having writing exercises [2]. Other work implements a comprehension-rst pedagogy but has limitations in their evaluations or the breadth of what they teach. ...
Conference Paper
Full-text available
What knowledge does learning programming require? Prior work has focused on theorizing program writing and problem solving skills. We examine program comprehension and propose a formal theory of program tracing knowledge based on control flow paths through an interpreter program's source code. Because novices cannot understand the interpreter's programming language notation, we transform it into causal relationships from code tokens to instructions to machine state changes. To teach this knowledge, we propose a comprehension-first pedagogy based on causal inference, by showing, explaining, and assessing each path by stepping through concrete examples within many example programs. To assess this pedagogy, we built PLTutor, a tutorial system with a fixed curriculum of example programs. We evaluate learning gains among self-selected CS1 students using a block randomized lab study comparing PLTutor with Codecademy, a writing tutorial. In our small study, we find some evidence of improved learning gains on the SCS1, with average learning gains of PLTutor 60% higher than Codecademy (gain of 3.89 vs. 2.42 out of 27 questions). These gains strongly predicted midterms (R²=.64) only for PLTutor participants, whose grades showed less variation and no failures.
... We show in Figure 7 where each concept is faded in the CF version of the game. Unlike textual introductions which must present many concepts "at once"-the complexity of which may unnecessarily hamper or confuse novices [8,2]our approach manages cognitive complexity by introducing concepts in relative isolation, then gradually mixing previously learned concepts to strengthen understanding of the whole, in a repeating pattern informed by theories of elaboration and flow [68,25] and lessons from cognitive tutors [9]. must now also understand replication and first-class function application (functions as input to other functions). ...
Conference Paper
Dominant approaches to programming education emphasize program construction over language comprehension. We present Reduct, an educational game embodying a new, comprehension-first approach to teaching novices core programming concepts which include functions, Booleans, equality, conditionals, and mapping functions over sets. In this novel teaching strategy, the player executes code using reduction-based operational semantics. During gameplay, code representations fade from concrete, block-based graphics to the actual syntax of JavaScript ES2015. We describe our design rationale and report on the results of a study evaluating the efficacy of our approach on young adults (18+) without prior coding experience. In a short timeframe, novices demonstrated promising learning of core concepts expressed in actual JavaScript. We also present results from an online deployment. Finally, we discuss ramifications for the design of future computational thinking games.
... Therefore, example-tracing tutors can meet the first requirement. They also meet the second factor: As mentioned, example-tracing tutors and the Tutorshop support individualized problem selection based on Bayesian Knowledge Tracing Anderson et al. 1989;Corbett et al. 2000;VanLehn 2011). ...
Article
Full-text available
In 2009, we reported on a new Intelligent Tutoring Systems (ITS) technology, example-tracing tutors, that can be built without programming using the Cognitive Tutor Authoring Tools (CTAT). Creating example-tracing tutors was shown to be 4–8 times as cost-effective as estimates for ITS development from the literature. Since 2009, CTAT and its associated learning management system, the Tutorshop, have been extended and have been used for both research and real-world instruction. As evidence that example-tracing tutors are an effective and mature ITS paradigm, CTAT-built tutors have been used by approximately 44,000 students and account for 40 % of the data sets in DataShop, a large open repository for educational technology data sets. We review 18 example-tracing tutors built since 2009, which have been shown to be effective in helping students learn in real educational settings, often with large pre/post effect sizes. These tutors support a variety of pedagogical approaches, beyond step-based problem solving, including collaborative learning, educational games, and guided invention activities. CTAT and other ITS authoring tools illustrate that non-programmer approaches to building ITS are viable and useful and will likely play a key role in making ITS widespread.
... An interesting finding from his laboratory is that if the feedback (KR) is provided after each trial, immediate performance is better than when only summary performance is given every 15 trials; however, long-term or ultimate learning is facilitated by giving only summary feedback intermittently such as after 15 trials. Anderson, Conrad and Corbett (1989) examined feedback in the context of learning the programming language LISP. They taught the language using a computer based tutor and varied whether or not they provided feedback and when it was provided. ...
... It is also the case that there is a vast middle ground of educational research in which the distinction between basic versus domain specifi c work is often blurred (Anderson, Conrad, & Corbett, 1989;Corbett & An derson, 1988;Singley & Anderson, 1989) . However, at the further extreme are the attempts to model rare forms of expertise, such as that possessed by Submarine Com manders (Ehret, Gray, & Kirschenbaum, 2000;Gray, Kirschenbaum, & Ehret, 1997;Kirschen baum & Gray, 2000), uninhabited air ve hicle (UA V) operators (Gluck et al., 2007), or airline pilots (Byrne & Kirlik, 2005). ...
Chapter
Full-text available
[from the Introduction] Cognitive engineering is the application of cognitive science theories to human factors practice. As this description suggests, there are strong symbioses between cognitive engineering and cognitive science, but there are also strong differences. Symbiosis implies a mutual influence, and the history of cognitive engineering supports this characterization in two key areas: the development of cognitive theory and the development of computational modeling software. For theory development, a stringent test of our understanding of cognitive processes is whether we can apply our knowledge to real-world problems. The degree to which we succeed at this task is the degree to which we have developed robust and powerful theories. The degree to which we fail at this task is the degree to which more research and stronger theories are required (Gray, Schoelles, & Myers, 2004).
Conference Paper
Full-text available
The integration of Artificial Intelligence (AI) and Machine Learning (ML) is revolutionizing academic education, redefining traditional teaching and learning methods. This abstract explores the transformative impact of AI and ML in academia. AI-driven technologies, such as intelligent tutoring systems and adaptive learning platforms, are enhancing personalized learning experiences, catering to individual student needs, and improving engagement and performance. Administrative processes in educational institutions are being optimized through automated tasks and data-driven decision-making, enhancing resource allocation and overall education quality. Nonetheless, ethical considerations and data privacy issues pose challenges that require careful management. Striking a balance between innovation and safeguarding sensitive student data is crucial for responsible AI implementation in education. In conclusion, the ongoing fusion of AI and ML with academic education is reshaping the learning landscape and preparing students for an ever-evolving future. As these technologies continue to advance, they promise boundless opportunities to enhance teaching and learning methods, ultimately redefining the way we acquire knowledge and skills in the digital age.
Article
Full-text available
The ability to organize variable sensory signals into discrete categories is a fundamental process in human cognition thought to underlie many real-world learning problems. Decades of research suggests that two learning systems may support category learning and that categories with different distributional structures (rule-based, information-integration) optimally rely on different learning systems. However, it remains unclear how the same individual learns these different categories and whether the behaviors that support learning success are common or distinct across different categories. In two experiments, we investigate learning and develop a taxonomy of learning behaviors to investigate which behaviors are stable or flexible as the same individual learns rule-based and information-integration categories and which behaviors are common or distinct to learning success for these different types of categories. We found that some learning behaviors are stable in an individual across category learning tasks (learning success, strategy consistency), while others are flexibly task-modulated (learning speed, strategy, stability). Further, success in rule-based and information-integration category learning was supported by both common (faster learning speeds, higher working memory ability) and distinct factors (learning strategies, strategy consistency). Overall, these results demonstrate that even with highly similar categories and identical training tasks, individuals dynamically adjust some behaviors to fit the task and success in learning different kinds of categories is supported by both common and distinct factors. These results illustrate a need for theoretical perspectives of category learning to include nuances of behavior at the level of an individual learner.
Preprint
The ability to organize variable acoustic signals into discrete categories is a fundamental process supporting speech perception. Adult learners can acquire complex, multidimensional auditory categories via feedback. There is substantial variability across individuals in how quickly and how well they learn auditory categories; however, it is unclear how the same individual approaches different category learning problems. We trained the same participants on three types of multidimensional auditory categories with different distributional structure (rule-based, information-integration) and from different auditory domains (nonspeech, speech). Consistent with prior work, there was substantial variability in both how well individuals learned the different categories and the strategies they used. As a novel contribution, we found that within an individual, learning outcomes were related across all three tasks and success across tasks was related to working memory capacity. Across tasks, we found that the same individual tailored their strategies for the specific task at hand, rather than systematically applying the same kind of strategy across different tasks. It was also uncommon for participants to shift toward an optimal strategy across different tasks and, instead, most participants used a mix of optimal and suboptimal strategies. These results indicate that auditory category learning may be supported by a category-general learning ability and highlight the importance of considering variability within an individual as they learn categories with different requirements.
Conference Paper
Full-text available
There is a dire need for replication research in the learning sciences, as methods put forth for increasing student learning should be unequivocally grounded in reproducible, reliable research. Learning science research is not only a critical input in the learning engineering process during the development of educational technology tools, such as courseware, but also as an output after student data have been analyzed to determine if the learning methods used were effective for students in their natural learning context. Furthermore, research that can provide causal evidence that a method of learning is effective for students should be reproduced-and the generality for its use expanded-so that methods that cause learning gains can be widely applied. One such method is the doer effect: the principle that students who engage with more practice have higher learning gains than those who only read expository text or watch video. This effect has been shown to be causal in prior research through statistical modeling using data mined from natural learning contexts. The goal of this paper is to replicate this research using a large-scale data set from courseware used at a major online university. The learning-by-doing data recorded by the courseware platform were combined with final exam data to replicate the statistical model of the causal doer effect study. Results from this analysis similarly point to a causal relationship between doing practice and learning outcomes. The implications of these doer effect results and future learning science research using large-scale data analytics will be discussed.
Article
Full-text available
Lernen erfordert vielfältige metakognitive Aktivitäten, die den Wissensaufbau und die Steuerung des eigenen Lernprozesses unterstützen. Dazu zählen die Auswahl und Planung von Zielen, die Anwendung, Beobachtung und Bewertung von Lernstrategien sowie regulierende Eingriffe zum Erreichen gesetzter Ziele. Der Artikel zeigt beispielhaft auf, wie eine KI-gestützte Trainingssoftware Lernende dabei fördern kann, sich Ziele zu setzen, Phasen fokussierten Arbeitens umzusetzen und ein sinnvolles Pausenmanagement einzubinden. Metakognitives Feedback bildet aufbauend auf Prinzipien des maschinellen Lernens den Wert zielgerichteten Handelns ab. In Pilotbefunden zeigen sich bereits erste positive Effekte dieses Ansatzes, die eine weitere Exploration nahelegen. Basierend darauf wird eine mögliche curriculare Einbindung im Lernfeld Schule sowie Implikationen für die Rolle der Lehrkräfte diskutiert.
Preprint
Full-text available
Lernen erfordert vielfältige metakognitive Aktivitäten, die den Wissensaufbau und die Steuerung des eigenen Lernprozesses unterstützen. Dazu zählen die Auswahl und Planung von Zielen, die Anwendung, Beobachtung und Bewertung von Lernstrategien sowie regulierende Eingriffe zum Erreichen gesetzter Ziele. Der Artikel zeigt beispielhaft auf, wie eine KI-gestützte Trainingssoftware Lernende dabei fördern kann, sich Ziele zu setzen, Phasen fokussierten Arbeitens umzusetzen und ein sinnvolles Pausenmanagement einzubinden. Metakognitives Feedback bildet aufbauend auf Prinzipien des maschinellen Lernens den Wert zielgerichteten Handelns ab. In Pilotbefunden zeigen sich bereits erste positive Effekte dieses Ansatzes, die eine weitere Exploration nahelegen. Basierend darauf wird eine mögliche curriculare Einbindung im Lernfeld Schule sowie Implikationen für die Rolle der Lehrkräfte diskutiert.
Article
In recent years, learning process data have become increasingly easy to collect through computer-based learning environments. This has led to increased interest in the field of learning analytics, which is concerned with leveraging learning process data in order to better understand, and ultimately to improve, teaching and learning. In computing education, the logical place to collect learning process data is through integrated development environments (IDEs), where computing students typically spend large amounts of time working on programming assignments. While the primary purpose of IDEs is to support computer programming, they might also be used as a mechanism for delivering learning interventions designed to enhance student learning. The possibility of using IDEs both to collect learning process data, and to strategically intervene in the learning process, suggests an exciting design space for computing education research: that of IDE-based learning analytics. In order to facilitate the systematic exploration of this design space, we present an IDE-based data analytics process model with four primary activities: (1) Collect data, (2) Analyze data, (3) Design intervention, and (4) Deliver intervention. For each activity, we identify key design dimensions and review relevant computing education literature. To provide guidance on designing effective interventions, we describe four relevant learning theories, and consider their implications for design. Based on our review, we present a call-to-action for future research into IDE-based learning analytics.
Article
Data-driven intelligent tutoring systems learn to provide feedback based on past student behavior, reducing the effort required for their development. A major obstacle to applying data-driven methods in the programming domain is the lack of meaningful observable actions for describing the students' problem-solving process. We propose rewrite rules as a language-independent formalization of programming actions in terms of code edits. We describe a method for automatically extracting rewrite rules from students' program-writing traces, and a method for debugging new programs using these rules. We used these methods to automatically provide hints in a web application for learning programming. In-class evaluation showed that students receiving automatic feedback solved problems faster and submitted fewer incorrect programs. We believe that rewrite rules provide a good basis for further research into how humans write and debug programs.
Conference Paper
Intelligent Tutoring Systems are highly effective at helping students learn, but have required intensive amounts of development time in the past, keeping teachers from making their own. Data-driven tutoring has made it possible to build these tutors more efficiently. For my thesis work, I intend to build an authoring tool for data-driven tutors that is designed to be used by computer science teachers. I plan to design this system based on data gathered in interviews with CS educators and evaluate it on its usability for new users.
Article
Full-text available
The major results reported by R. A. Carlson et al (see record 1989-24864-001) confirm predictions of J. R. Anderson's (1983) ACT * theory. In particular, ACT * predicts the detrimental effects of the transition to randomized practice because of the need to learn new productions, the complexity effect of gate and judgment type because of more complex production conditions, the effects of practice and its interaction with complexity because of the strengthening mechanisms, and the effects of memory load because of the need to hold information active in working memory so that it can be matched by production conditions. (PsycINFO Database Record (c) 2012 APA, all rights reserved)
Article
Full-text available
Several theories assume that practice (a) results in restructuring of component processes and (b) reduces demand on working memory. Eight subjects practiced judgments about digital logic gates for over 8,000 trials. At two practice levels, subjects made judgments while retaining short-term memory loads irrelevant to the judgments, relevant but not accessed, or accessed to make the judgments. Four phenomena together provide constraints for theory: First, performance declined in moving from blocked practice to randomized practice. Second, gate and judgment type strongly affected latency. Third, these effects declined but did not disapppear with practice. Fourth, the cost of accessing information in working memory remained substantial. These results are interpreted as reflecting a serial process with constant structure, while component processes become faster. The results challenge theories assuming that all learning results from restructuring or that restructuring is an automatic consequence of practice, and they support a distributed view of working memory. (PsycINFO Database Record (c) 2012 APA, all rights reserved)
Article
Full-text available
This article presents a theory in which automatization is construed as the acquisition of a domain-specific knowledge base, formed of separate representations, instances, of each exposure to the task. Processing is considered automatic if it relies on retrieval of stored instances, which will occur only after practice in a consistent environment. Practice is important because it increases the amount retrieved and the speed of retrieval; consistency is important because it ensures that the retrieved instances will be useful. The theory accounts quantitatively for the power-function speed-up and predicts a power-function reduction in the standard deviation that is constrained to have the same exponent as the power function for the speed-up. The theory accounts for qualitative properties as well, explaining how some may disappear and others appear with practice. More generally, it provides an alternative to the modal view of automaticity. (PsycINFO Database Record (c) 2012 APA, all rights reserved)
Article
Full-text available
Proposed and evaluated 4 optimization strategies for the learning of a large German-English vocabulary using 120 undergraduates. The 1st strategy involved presenting items in a random order and served as a benchmark against which the others could be evaluated. The 2nd strategy permitted S to determine on each trial of the experiment which item was to be presented, thus placing instruction under "learner control." The 3rd and 4th strategies were based on a mathematical model of the learning process; these strategies were computer controlled and took account of S's response history in making decisions about which items to present next. Performance on a delayed test administered 1 wk. after the instructional session indicated that the learner-controlled strategy yielded a gain of 53% when compared to the random procedure, whereas the best of the 2 computer-controlled strategies yielded a gain of 108%. Implications for a theory of instruction are considered. (PsycINFO Database Record (c) 2012 APA, all rights reserved)
Article
I shall describe a model of the evolution of rule-structured knowledge that serves as a cornerstone of our development of computer-based coaches. The key idea is a graph structure whose nodes represent rules, and whose links represent various evolutionary relationships such as generalization, correction, and refinement. I shall define this graph and describe a student simulation testbed which we are using to analyze different genetic graph formulations of the reasoning skills required to play an elementary mathematical game.
Article
Recent researches are cited which suggest that the acquisition of manual speed-skill proceeds by a certain type of selective action. A formal theoretical model is developed, and its predictions compared with the experimental results. Certain complications of the theory, and conclusions from it are outlined, and the nature of the selective mechanism is discussed. Some implications for training are indicated.
Article
Cognitive skills are encoded by a set of productions, which are organized according to a hierarchical goal structure. People solve problems in new domains by applying weak problem-solving procedures to declarative knowledge they have about this domain. From these initial problem solutions, production rules are compiled that are specific to that domain and that use of the knowledge. Numerous experimental results may be predicted from this conception of skill organization and skill acquisition. These include predictions about transfer among skills, differential improvement on problem types, effects of working memory limitations, and applications to instruction. The theory implies that all varieties of skill acquisition, including those typically regarded as inductive, conform to this characterization.
Article
Studies of students learning cognitive skills have revealed that private tutoring appears to be much more effective than conventional classroom instruction. A computer-based tutor is considered that is as effective in teaching LISP as a human tutor. GREATERP (Goal-Restricted Environment for Tutoring and Educational Research on Programming) is an attempt to combine artificial-intelligence technology and a psychological theory of skill acquisition into an effective teaching device. Two program listings give examples of dialogues between student and computer tutor.
Article
We describe the current version of the Why System, a script-based socratic tutor which uses tutoring strategies formulated as production rules. The current system is capable of carrying on a dialogue about the factors influencing rainfall by presenting different cases to the student, asking for predictions, probing for relevent factors, entrapping the student when he has not identified all necessary factors, and presenting counterexamples. The current system is incomplete because it lacks a goal structure to guide the tutorial sessions. We outline a more complete theory of the goal structure of Socratic tutors based on analysis of human tutorial dialogues. There are two top level goals: (1) refinement of the student's causal model and (2) refinement of the student's predictive abilities. The subgoals are diagnosis of bugs in the student's knowledge and correction of the bugs. This goal-driven control mechanism governs the selection of examples and teaching strategies used by the tutor.
Article
This article discusses cognitive models of learning to program recursion and their relation to lessons on recursion in an intelligent computer tutor for LISP programming (the LISP Tutor). The cognitive models are implemented as production systems in which programming skill is characterized as the decomposition of programming goals into subgoals and elementary actions via the application of programming plans. Two sets of learning mechanisms are used in the cognitive models. Analogical problem-solving mechanisms use declarative knowledge of example program solutions to overcome problem-solving impasses. Knowledge compilation mechanisms summarize problem solutions into efficient problem-solving skill. Analyses and simulations of novice and expert programming were used to develop ideal models of the programming knowledge to confer upon students and bugs that characterize common misconceptions. The LISP Tutor uses the ideal models and bugs to guide its interactions with students. Experimental evaluations of the LISP Tutor indicate that it is more efficient and effective than classroom instruction.
Article
Describes cyclopean methodology, based on the author's stereopsis model in which binocular depth perception is a central process occurring at a specific site on the visual cortex termed the "cyclopean retina." Evidence of cyclopean perception is reviewed, and over 400 computer-generated, random-dot stereograms and anaglyphs are presented. (16 p. ref.) (PsycINFO Database Record (c) 2012 APA, all rights reserved)
Article
In the past two decades cognitive psychologists have started to adopt "production systems" as the principle theoretical medium in which to cast complex theories of human intelligence. Production systems are a class of computer simulation models that are stated in terms of condition-action rules, and that make strong assumptions about the nature of the cognitive architecture. The initial use of production systems was to provide theoretical accounts of human performance on a variety of tasks. However, one fundamental feature that distinguishes production systems from most other forms of cognitive simulation is their capability for self-modification. Because of this feature it quickly became apparent that they were eminently well suited for dealing with issues of development and learning. There has been no single volume exclusively devoted to production-system models of human cognition. For this reason we came to feel that a volume that enables readers to compare and contrast those research efforts would have special importance. All of the chapters in this book deal with the issues that arise when one constructs self-modifying production systems. The book should be of interest to any cognitive scientist interested in the processes underlying the acquisition of knowledge and skills, particularly researchers in psychology with an information-processing orientation, as well as cognitively oriented researchers in artificial intelligence. In planning this book, we sought to accomplish for the production-system paradigm what "Explorations in Cognition" did for the LNR Group, or what "Scripts, Plans, Goals and Understanding" did for the Yale AI group. That is, we have tried to provide a single volume that provides access to state-of-the-art research results and does so in a way that allows readers to gain some perspective on the underlying research approach. (PsycINFO Database Record (c) 2012 APA, all rights reserved)
Article
Reviews existing theories of fluency in high-proficiency skills such as speech, and proposes that execution of behavior involves the activation of a hierarchy of nodes in proper serial order within an output system. Activating a node at any level in the system activates its connected nodes, and repeated activation increases the rate of priming per unit time, thereby allowing a faster rate of output at the lowest, muscle movement level. Relevance of this theory for several related issues is discussed: why behavior becomes more flexible with practice, transferring readily from one response mechanism to another; why there is almost perfect transfer from one hand to the other for simple skills, but less than perfect transfer for complex skills; why skills at higher, semantic levels transfer to new behavioral sequences, as when bilinguals produce a word-for-word translation of a practiced sentence in their other language. The theory also provides a new way of looking at motor equivalence, automaticity, speed–accuracy trade-off, subordinate autonomy, and the motor program. (32 ref) (PsycINFO Database Record (c) 2012 APA, all rights reserved)
Article
We have gathered protocols of subjects in their first 30 hours of learning LISP. The processes by which subjects write LISP functions to meet problem specifications has been modeled in a simulation program called GRAPES (Goal Restricted Production System). The GRAPES system embodies the goal-restricted architecture for production systems as specified in the ACT* theory ( [3] and [4]). We compare our simulation to human protocols on a number of problems. GRAPES simulates the top-down, depth-first flow of control exhibited by subjects and produces code very similar to subject code. Special attention is given to modeling student solutions by analogy, how students learn from doing, and how failures of working memory affect the course of problem-solving. Of major concern is the process by which GRAPES compiles operators in solving one problem to facilitate the solution of later problems.
Article
Three studies investigate how problem solvers learn to apply appropriate actions in problem solving. Part of this knowledge appears to result from learning the sets of problem features (schemata) that predict the success of different problem-solving actions (operators). A major claim is that this learning can be produced, in part, by the same mechanisms that produce concept formation and abstraction of object schemata. Studies using geometry proof problems and an abstract maze-searching task produce results similar to common findings in the schema abstraction literature: Performance improves as the prototypicality of the feature sets increases. Also examined are the effects of active processing of problem features, relevance of the discriminating features to the problem solution, amount of practice, and delay of feedback regarding the accuracy of operator selection. Subjects learn when to apply an operator better during active, deliberate hypothesis testing, regardless of feature relevance. Delayed feedback produces poorer performance with extended practice. A model of classification learning and a simulation of explicit hypothesis testing produced reasonable fits to the data. The failure to find evidence for unconscious learning is evidence against the automatic discrimination mechanism proposed in ACT∗.
Conference Paper
The tutor for doing proofs in high school geometry consists of a cot of ideal and buggy rules (IRR), a tutor, and an interface. The IBR is responsible for ehiuertly computing matcher, to all the correct and incorrect rules The interface is responsible for interacting with the student and graphically representing the proof. The tutor is responsible for directing the IBR and interface to achieve a current tutorial strategy. The strategy wo employ involves tracing the student's behavior in terms of what rules in the IBR it instantiates, correcting the student when behavior deviates below a minimum threshold, and helping the student over hurdles. While incomplete, current evidence indicates the geometry tutor is guite effective.
Article
Despite extensive discussion in the literature about the diagnosis and subsequent remediation of students' errors, few studies have compared the effects of different styles of error-based remediation. Swan (1983) found that a conflict approach (pointing out errors made by students and demonstrating their consequences) was more effective than simple reteaching, but Bunderson & Olsen (1983) found no difference between error-specific remediation and reteaching. More studies are needed in order to understand the factors which lead to successful remediation. The three studies discussed in this article compared error-specific or model-based remediation (MBR) with reteaching in algebra. MBR bases its remediation on the model inferred for an individual student before reteaching the correct procedure. Reteaching simply shows students the correct procedure without addressing specific errors. The results show that MBR and reteaching are both more effective than no tutoring; however, MBR is not clearly more effective than reteaching. The results are discussed in terms of stability of errors, their relevance to educational practice and to intelligent tutoring systems (ITS). Although the studies were carried out using human tutors, the results suggest that for the purpose of remediation in the algebra domain, when taught procedurally, “classical” computer-assisted instruction (CAI) would be as effective as an ITS.
Article
Thesis (Ph. D.)--Carnegie-Mellon University, 1988. Includes bibliographical references (leaves 129-131). Microfiche. s
Transfer of skills between inconsistent editors. Human-Computer Interaction The chunking of goal hierarchies: A generalized model of practice Learning by chunkhrg: A production system model of practice The transfer of cognitive skills
  • Pg Poison
  • E Muncher
  • D E Kieras
  • P S Rosenbloom
  • A D &newell
  • P K&r
  • R Langley
  • Neches
Poison, PG., Muncher, E., & Kieras, D.E. (ii press). Transfer of skills between inconsistent editors. Human-Computer Interaction. Rosenbloom, P.S, &Newell, A. (1986). The chunking of goal hierarchies: A generalized model of practice. In R.S. Michalski, J.G. Carbonell. & T.M. Mitchell (Eds.), Machine Learn-ing (Vol. II, pp. 247-288). Los Altos, CA: Morgan Kaufmann. Rosenbloom, P., & Newell, A. (1987). Learning by chunkhrg: A production system model of practice. In D. K&r, P. Langley, & R. Neches, (Eds.), Production System Models of Learning and Development. Cambridge, MA: MIT Press. Singley, K., & Anderson, J.R. (1989). The transfer of cognitive skills. Cambridge, MA: Harvard Press.
wfects of immediacy of feedback on learning proof skills
  • M W Lewis
  • J R Anderson
Lewis, M.W., & Anderson, J.R. (1989). wfects of immediacy of feedback on learning proof skills. Manuscript in preparation.
Transfer of programming skills in novice LISP learners. Unpublished doctoral dissertatioan, Carnegie-Mellon University Production system models ofpractice e$fects. Unpublished doctoral dis-sertation
  • K Kessler
Kessler, K. (1988). Transfer of programming skills in novice LISP learners. Unpublished doctoral dissertatioan, Carnegie-Mellon University, Pittsburgh, PA. Lewis, C.H. (1978). Production system models ofpractice e$fects. Unpublished doctoral dis-sertation. University of Michigan, Ann Arbor.
Leaming to program recursion
  • P L Pirolli
  • J R &anderson
Pirolli, P.L., &Anderson, J.R. (1984). Leaming to program recursion. In Sixth Annual Cog-nitive Science Meetings (pp. 277-280). Boulder, CO.
Automatic programming. In Artificial Intelligence Handbook
  • A Barr
  • E A Feigenbaum
Barr, A., & Feigenbaum, E. A. (1982). Automatic programming. In Artificial Intelligence Handbook. Los Altos, CA: Kaufmann.
Impact of feedback during complex skill acquisition Frequency and practice effects on the composition on knowledge in LISP evaluation (pp. 236-259) Cognitive aspects of human-computer interaction
  • J E Mckendree
  • Pa Mckendree
  • J E &anderson
McKendree, J.E. (1986). Impact of feedback during complex skill acquisition. Unpublished doctoral dissertation, Carnegie-Mellon University, Pittsburgh, PA. McKendree, J.E., &Anderson, J.R. (1987). Frequency and practice effects on the composition on knowledge in LISP evaluation (pp. 236-259). In J.M. Carroll (Ed.), Cognitive aspects of human-computer interaction. Cambridge, MA: MIT Press.
Foundationsof Cyclopeanperception
  • B Julesa
Julesa. B. (1971). Foundationsof Cyclopeanperception. Chicago: University of Chicago Press.
Production system models ofpractice e$fects
  • C H Lewis
Lewis, C.H. (1978). Production system models ofpractice e$fects. Unpublished doctoral dissertation. University of Michigan, Ann Arbor.
  • J R Anderson
  • A T Corbett
  • B J Breiser
Anderson, J.R., Corbett, A.T., BReiser, B.J. (1987). ErrentialLISP. Reading, MA: Addison- Wesley.
Impact of feedback during complex skill acquisition. Unpublished doctoral dissertation
  • J E Mckendree
McKendree, J.E. (1986). Impact of feedback during complex skill acquisition. Unpublished doctoral dissertation, Carnegie-Mellon University, Pittsburgh, PA.
  • J R Anderson
  • A T Corbett
  • B J Breiser
Anderson, J.R., Corbett, A.T., BReiser, B.J. (1987). ErrentialLISP. Reading, MA: Addison-Wesley.
Intelligent tutoring systems: At the crossroads of artificial intelligence and education
  • A T Corbett
  • J R Anderson
  • E J Patterson
Corbett, A.T., Anderson, J.R., & Patterson, E.J. (in press). Student modeling and tutoring flexibility in the LISP tutor. In C. Frasson & G. Gauthier (Eds.), Intelligent tutoring systems: At the crossroads of artificial intelligence and education. Nonvood, NJ: Ablex.
A cognitive model and computer tutor for programming recursion
  • P Piiolli
Piiolli, P. (1986). A cognitive model and computer tutor for programming recursion. Human-Computer Interaction, 2, 319-355.