Article

Skill acquisition and the LISP Tutor

Authors:
To read the full-text of this research, you can request a copy directly from the authors.

Abstract

An analysis of student learning with the LISP tutor indicates that while LISP is complex, learning it is simple. The key to factoring out the complexity of LISP is to monitor the learning of the 500 productions in the LISP tutor which describe the programming skill. The learning of these productions follows the power-law learning curve typical of skill acquisition. There is transfer from other programming experience to the extent that this programming experience involves the same productions. Subjects appear to differ only on the general dimensions of how well they acquire the productions and how well they retain the productions. Instructional manipulations such as remediation, content of feedback, and timing of feedback are effective to the extent they give students more practice programming, and explain to students why correct solutions work.

No full-text available

Request Full-text Paper PDF

To read the full-text of this research,
you can request a copy directly from the authors.

... Currently, the major cognitive tutor systems used at PACT are the LISP Tutor [14,12], which teaches basic LISP through a self-paced course given at CMU, and a series of mathematical tutors in the geometry [46,35] and algebra domains [9]. These tutors now covers the curriculum of high school geometry and algebra courses in the US, respectively. ...
... Description of the domain The cognitive tutors developed at PACT have focused on either basic level programming (initially LISP, later other languages such as Pascal) or mathematical proof generation in algebra and geometry. The rst versions of cognitive tutors on these topics was introduced in the 1980s (LISP Tutor [14,12] and Geometry Tutor [15]), and has later been evaluated and revised extensively. Currently, the focus of the mathematically oriented tutors are on adaptation to extensive deployment in high schools, while the programming tutors are more tailored for laboratory experiments. ...
... The results of the evaluation showed that there was no signicant dierence between the gain of the control and experiment groups on the basic argumentation test (see Figure 4. 12), and in addition that CATO performed very well (it was the third best of all answers even in the post-test). However, on the memowriting task, the control group scored signicantly better than the experiment group, and CATO itself did not receive a high grade. ...
... Programming tutors have been built since the 1970s, for programming languages such as Lisp (Anderson et al., 1986), Prolog (Hong, 2004), Java (Holland et al., 2009), Pascal (Johnson and Soloway, 1985), C (Wang et al., 2011), and many more. None of these tutors supports automatic feedback on the incremental development of programs for class 3 exercises, in which teachers can easily add programming exercises and adapt feedback. ...
... None of these tutors supports automatic feedback on the incremental development of programs for class 3 exercises, in which teachers can easily add programming exercises and adapt feedback. For example, the Lisp tutor (Anderson et al., 1986) does give feedback on the steps a student takes towards a solution to a programming exercise, and allows students to solve a program flexibly in that students do not have to follow a strict top to bottom, left to right order (Corbett et al., 1988), but it o↵ers class 2 exercises. Also, adding an exercise to the Lisp tutor is non-trivial. ...
... Tools that support students in learning programming have been developed since the 1960s (Ulloa, 1980;Douce et al., 2005). Some of these tools analyse incremental steps a student takes (Anderson et al., 1986), and/or support the development of programming plans (Soloway, 1985;Johnson and Soloway, 1985). The early work in this area primarily targeted the programming languages Lisp and Pascal (including Pascal-like imperative languages); a nice overview is given by Vanneste et al. (1996). ...
Article
Full-text available
Ask-Elle is a tutor for learning the higher-order, strongly-typed functional programming language Haskell. It supports the stepwise development of Haskell programs by verifying the correctness of incomplete programs, and by providing hints. Programming exercises are added to Ask-Elle by providing a task description for the exercise, one or more model solutions, and properties that a solution should satisfy. The properties and model solutions can be annotated with feedback messages, and the amount of flexibility that is allowed in student solutions can be adjusted. The main contribution of our work is the design of a tutor that combines (1) the incremental development of different solutions in various forms to a programming exercise with (2) automated feedback and (3) teacher-specified programming exercises, solutions, and properties. The main functionality is obtained by means of strategy-based model tracing and property-based testing. We have tested the feasibility of our approach in several experiments, in which we analyse both intermediate and final student solutions to programming exercises, amongst others.
... The tradeoff has recently been called the 'assistance dilemma' [31]. To resolve this dilemma, some results of earlier studies suggest that on-demand graded hints, with each giving progressively more specific advice until the learners judge they accomplished the task, might be effective for learning [39], [42], [43], [44], [45], [46], [47], [48], [49], [50], [51], [52]. ...
... Variants of this technique are used in a number of ITS fields (e.g., [38], [39], [40], [41]). However, some ITS researchers have pointed out that the adaptive help function tends to provide over-assistance and that it might obstruct effective learning [39], [42], [43], [44], [45], [46], [47]. A tradeoff exists between information giving and withholding to achieve optimal learning, which has recently been called the "assistance dilemma' [31]. ...
... A tradeoff exists between information giving and withholding to achieve optimal learning, which has recently been called the "assistance dilemma' [31]. Anderson et al. (1989) conducted a study that evaluated the effects of the tutor's mastery learning method and of explanatory content in both the tutor's hints and its feedback messages [42]. They compared the regular Lisp Tutor, which provides explanatory content in its hints and in some of its error feedback messages, with a version that simply told students they were wrong when they made errors, or gave them the correct answer when they requested a hint. ...
Article
Full-text available
Over the past few decades, many studies conducted in the field of learning science have described that scaffolding plays an important role in human learning. To scaffold a learner efficiently, a teacher should predict how much support a learner must have to complete tasks and then decide the optimal degree of assistance to support the learner's development. Nevertheless, it is difficult to ascertain the optimal degree of assistance for learner development. For this study, it is assumed that optimal scaffolding is based on a probabilistic decision rule: given a teacher's assistance to facilitate the learner development, an optimal probability exists for a learner to solve a task. To ascertain that optimal probability, we developed a scaffolding system that provides adaptive hints to adjust the predictive probability of the learner's successful performance to the previously determined certain value, using a probabilistic model, i.e., item response theory (IRT). Furthermore, using the scaffolding system, we compared learning performances by changing the predictive probability. Results show that scaffolding to achieve 0.5 learner success probability provides the best performance. Additionally, results demonstrate that a scaffolding system providing 0.5 probability decreases the number of hints (amount of support) automatically as a fading function according to the learner's growth capability.
... Much of this research has focused on specific skills. For example, researchers have captured low-level programming knowledge as part of learning technologies (Anderson et al. 1989). A long history of work has theorized about program comprehension skills, describing the bottom up, top down, and opportunistic strategies that developers use at a high level (Roehm et al. 2012;Von Mayrhauser and Vans 1995). ...
... For example, studies have shown that the use of explicit slicing strategies in debugging (Francel and Rugaber 2001), the use of explicit strategies in tracing program execution (Xie et al. 2018), and the use of explicit strategies to extract requirements from problem statements (Haidry et al. 2017), are either correlated with or cause decreases in task completion time or increases in task performance. Early research on the LISP tutor (Anderson et al. 1989) and software design environments (Rist 1995) similarly showed that by defining expert problem solving strategies, and nudging novices to follow those expert strategies through hints and feedback, novices could approach expert performance. Many have also described explicit strategies for debugging; Metzger, for example, describes debugging strategies as a high-level plan for accomplishing a goal through a sequence of physical and cognitive actions (Metzger 2004) and Zeller presents a series of formal and informal procedures for isolating causes and effects of defects (Zeller 2009). ...
Article
Full-text available
Software developers solve a diverse and wide range of problems. While software engineering research often focuses on tools to support this problem solving, the strategies that developers use to solve problems are at least as important. In this paper, we offer a novel approach for enabling developers to follow explicit programming strategies that describe how an expert tackles a common programming problem. We define explicit programming strategies, grounding our definition in prior work both within software engineering and in other professions which have adopted more explicit procedures for problem solving. We then present a novel notation called Roboto and a novel strategy tracker tool that explicitly represent programming strategies and frame executing strategies as a collaborative effort between human abilities to make decisions and computer abilities to structure process and persist information. In a formative evaluation, 28 software developers of varying expertise completed a design task and a debugging task. We found that, compared to developers who are free to choose their own strategies, developers given explicit strategies experienced their work as more organized, systematic, and predictable, but also more constrained. Developers using explicit strategies were objectively more successful at the design and debugging tasks. We discuss the implications of Roboto and these findings, envisioning a thriving ecosystem of explicit strategies that accelerate and improve developers’ programming problem solving.
... Even though model-tracing tutors have been shown to be effective in labs and classrooms in a variety of domains (e.g., programming: Anderson et al. 1989; chemistry: Johnson and Holder 2010; algebra: Koedinger et al. 1997;Ritter et al. 2007; physics: VanLehn, et al. 2005), they are still relatively rare. The one exception is the Carnegie Learning Cognitive Tutor for mathematics, which has been used by hundreds of thousands of students in the United States. ...
... In order for the learner to see the consequence of the action, the tutor could allow the interface to perform the action, but then put up a modal dialog that would force an undo in order to get the user back on the solution path. Research from the original ACT tutors (Anderson et al. 1989) suggests that having learners stay on path is more useful than allowing learners to explore fruitless paths. Also, when building a tutor on top of an existing interface, the tutor must instruct the learner about the interface itself in addition to the domain knowledge relevant to the task. ...
Article
Full-text available
The Extensible Problem Specific Tutor (xPST) allows authors who are not cognitive scientists and not programmers to quickly create an intelligent tutoring system that provides instruction akin to a model-tracing tutor. Furthermore, this instruction is overlaid on existing software, so that the learner’s interface does not have to be made from scratch. The xPST architecture allows for extending its capabilities by the addition of plug-ins that communicate with additional third-party software. After reviewing this general architecture, we describe three major implementations that we have created using the xPST system, each using different third-party software as the learner’s interface. We have conducted three evaluations of authors using xPST to create tutoring content, and these are considered in turn. These evaluations show that xPST authors can quickly learn the system, and can efficiently produce successful embedded instruction.
... Most of this work has not investigated the teaching of programming problem-solving, although there are some exceptions. The LISP tutor built models of the program solution space and monitored learners' traversals through this space, intervening with corrective feedback if learners encountered error states or made mistakes the tutor had previously observed [3]. Other more recent efforts to teach problem-solving to programmers have found that having teachers prompt novices about the strategies they are using and whether those strategies are appropriate and effective can greatly improve novices' abilities to problem-solve independently [39]. ...
... However, the experimental group received four interventions that the control group did not: (1) A short lecture about problem solving and metacognition in programming; (2) A set of prompts to be answered when campers asked helpers for assistance; (3) a physical, paper handout of a problem solving model; and (4) the Idea Garden prototype for Cloud9. As we explained above, the Idea Garden prototype supported anti-patterns observed in previous studies as well as problem-solving strategy hints consistent with interventions (1)- (3). ...
Article
Many systems are designed to help novices who want to learn programming, but few support those who are not necessarily interested in learning programming. This paper targets the subset of end-user programmers (EUPs) in this category. We present a set of principles on how to help EUPs like this learn just a little when they need to overcome a barrier. We then instantiate the principles in a prototype and empirically investigate them in three studies: a formative think-aloud study, a pair of summer camps attended by 42 teens, and a third summer camp study featuring a different environment attended by 48 teens. Finally, we present a generalized architecture to facilitate the inclusion of Idea Gardens into other systems, illustrating with examples from Idea Garden prototypes. Results have been very encouraging. For example, under our principles, Study #2’s camp participants required significantly less in-person help than in a previous camp to learn the same amount of material in the same amount of time.
... When detected, a relevant feedback message was delivered (either on-demand when the code was executed or while the student was editing code) using BlockPy's built-in feedback mechanisms that are normally used to deliver runtime and output errors. We chose immediate feedback delivery because of its acquisitional efficiency for verbal knowledge and procedural skills [2,6,9,11]. In developing the feedback message, the instructors aimed to explain the misconception found and the corresponding mistake's location, rather than how the student should fix the mistake. ...
... The second post-test contained two programming problems -each related to one of the learning objectives. The statement of the three programming problems are:(1) The data block in the BlockPy canvas below provides a list of the number of students taking the 2015 SAT test in each state.Write an algorithm to compute and print the total number of students taking the SAT test in 2015.(2) The data block in the BlockPy canvas below provides a list of the per capita income of each state. ...
Conference Paper
Full-text available
The feedback given to novice programmers can be substantially improved by delivering advice focused on learners' cognitive misconceptions contextualized to the instruction. Building on this idea, we present Misconception-Driven Feedback (MDF); MDF uses a cognitive student model and program analysis to detect mistakes and uncover underlying misconceptions. To evaluate the impact of MDF on student learning, we performed a quasi-experimental study of novice programmers that compares conventional run-time and output check feedback against MDF over three semesters. Inferential statistics indicates MDF supports significantly accelerated acquisition of conceptual knowledge and practical programming skills. Additionally, we present descriptive analysis from the study indicating the MDF student model allows for complex analysis of student mistakes and misconceptions that can suggest improvements to the feedback, the instruction, and to specific students.
... Digital technologies, such as Intelligent Tutoring Systems (ITS), can be used to provide learners with different learning interventions such as scaffolding, feedback, etc., hence supporting them with the deliberate practice of their skills. An example is the classic LISP tutor that helps learners to become acquainted with the LISP programming language [4]. Traditional ITSs have shown positive effects supporting skills in wellformed topics and knowledge domains [5] such as mathematics [6] or programming languages [4,7]. ...
... An example is the classic LISP tutor that helps learners to become acquainted with the LISP programming language [4]. Traditional ITSs have shown positive effects supporting skills in wellformed topics and knowledge domains [5] such as mathematics [6] or programming languages [4,7]. Learners can practice this type of skills while directly interacting with the ITS through precise and unambiguous operations (e.g., keyboard strokes, button clicks). ...
Article
Full-text available
The development of multimodal sensor-based applications designed to support learners with the improvement of their skills is expensive since most of these applications are tailor-made and built from scratch. In this paper, we show how the Presentation Trainer (PT), a multimodal sensor-based application designed to support the development of public speaking skills, can be modularly extended with a Virtual Reality real-time feedback module (VR module), which makes usage of the PT more immersive and comprehensive. The described study consists of a formative evaluation and has two main objectives. Firstly, a technical objective is concerned with the feasibility of extending the PT with an immersive VR Module. Secondly, a user experience objective focuses on the level of satisfaction of interacting with the VR extended PT. To study these objectives, we conducted user tests with 20 participants. Results from our test show the feasibility of modularly extending existing multimodal sensor-based applications, and in terms of learning and user experience, results indicate a positive attitude of the participants towards using the application (PT+VR module).
... The second coder and a third coder (research intern) then independently categorized the interactions into the existing categories. We calculated a Fleiss' 3 The five interactions were excluded either because of errors with recording (2) or prominent unconsented student voices (3) 4 Grounded theory analysis contends that we can understand the content in communication by analyzing the type of "common ground" shared by the speakers. This standard qualitative analysis method, like most qualitative research, relies on two or more "coders" who categorize interactions according to the given property being studied and the research goals. ...
... ITS, which allow students to obtain real-time feedback while working on problems as part of a pre-specified curriculum, have been successful in domains such as K-12 math [5] and programming. The LISP tutor [3] has specific relevance to this work, as we use the Scheme programming language in our prototype implementation. Additional work has augmented ITS feedback by using real student data for Python programming [33]. ...
Conference Paper
Students learning to program often need help completing assignments and understanding why their code does not work as they expect it to. One common place where they seek such help is at teaching assistant office hours. We found that teaching assistants in introductory programming (CS1) courses frequently answer some variant of the question ``Am I on the Right Track?''. The goal of this work is to develop an automated tool that provides similar feedback for students in real-time from within an IDE as they are writing their program. Existing automated tools lack the generality that we seek, often assuming a single approach to a problem, using hand-coded error models, or applying sample fixes from other students. In this paper, we explore the use of program synthesis to provide less constrained automated answers to ``Am I on the Right Track'' (AIORT) questions. We describe an observational study of TA-student interactions that supports targeting AIORT questions, as well as the development of and design considerations behind a prototype integrated development environment (IDE). The IDE uses an existing program synthesis engine to determine if a student is on the right track and we present pilot user studies of its use.
... Much of this research has focused on specific skills. For example, researchers have captured low-level programming knowledge as part of learning technologies [3]. A long history of work has theorized about program comprehension skills, describing the bottom up, top down, and opportunistic strategies that developers use at a high level [59,69]. ...
... For example, studies have shown that the use of explicit slicing strategies in debugging [21], the use of explicit strategies in tracing program execution [71], and the use of explicit strategies to extract requirements from problem statements [27], are either correlated with or cause decreases in task completion time or increases in task performance. Early research on the LISP tutor [3] and software design environments [56] similarly showed that by defining expert problem solving strategies, and nudging novices to follow those expert strategies through hints and feedback, novices could approach expert performance. Many have also described explicit strategies for debugging; Metzger, for example, describes debugging strategies as a high-level plan for accomplishing a goal through a sequence of physical and cognitive actions [50] and Zeller presents a series of formal and informal procedures for isolating causes and effects of defects [73]. ...
Preprint
Full-text available
Software developers solve a diverse and wide range of problems. While software engineering research often focuses on tools to support this problem solving, the strategies that developers use to solve problems are at least as important. In this paper, we offer a novel approach for enabling developers to follow explicit programming strategies that describe how an expert tackles a common programming problem. We define explicit programming strategies, grounding our definition in prior work both within software engineering and in other professions which have adopted more explicit procedures for problem solving. We then present a novel notation called Roboto and a novel StrategyTracker tool that explicitly represents programming strategies and frame executing strategies as a collaborative effort between human abilities to make decisions and computer abilities to structure process and persist information. Ina formative evaluation, 28 software developers of varying expertise completed a design task and a debugging task. We found that, compared to developers who are free to choose their strategies, developers gave explicit strategies experienced their work as more organized, systematic, and predictable, but also more constrained. Developers using explicit strategies were objectively more successful at the design and debugging tasks. We discuss the implications of Roboto and these findings, envisioning a thriving ecosystem of explicit strategies that accelerate and improve developers programming problem solving.
... AISs have shown promise to fill this gap. For instance, Anderson et al. (1989) demonstrated how their adaptive LISP tutor could benefit learning programming skills (or "production rules") to create working LISP code. The tutor provided remedial instruction when students were incorrect and customized the feedback according to the type of error when possible. ...
... A critical complement to learner models' knowledge estimates are the PDRs that dictate, for the purpose of the instructional sequence decisions, what should be practiced based on the learner model predictions (Katz & Albacete, 2013). For instance, Anderson et al. (1989) had students proceed to a new KC once their learner model predicted the probability of solving problems for a KC exceeded 95%. Some AIS do not express PDR as probabilities (Heffernan & Heffernan, 2014;Canfield, 2001) instead opting for rules-of-thumb such as dropping content from practice once the student has correctly answered three times in a row. ...
... Such a design was found to encourage system gaming behaviors that harm learning [35], [36]. Although hint and feedback were used interchangeably in many computing education studies, the pedagogical differences between them are significant (e.g., Anderson et al [37] combined both feedback and hint in the LISP Tutor, but only used "feedback" to refer to both the facilitative functions). ...
... al 2014;Ritter et. al 2007;Anderson, Conrad, & Corbett 1989). ...
Article
Full-text available
Students have a limited time to study and are typically ineffective at allocating study time. Machine-directed study strategies that identify which items need reinforcement and dictate the spacing of repetition have been shown to help students optimize mastery (Mozer & Lindsey 2017). The large volume of research on this matter is typically conducted in constructed experimental settings with fixed instruction, content, and scheduling; in contrast, we aim to develop methods that can address any demographic, subject matter, or study schedule. We show two methods that model item-specific recall probability for use in a discrepancy-reduction instruction strategy. The first model predicts item recall probability using a multiple logistic regression (MLR) model based on previous answer correctness and temporal spacing of study. Prompted by literature suggesting that forgetting is better modeled by the power law than an exponential decay (Wickelgren 1974), we compare the MLR approach with a Recurrent Power Law (RPL) model which adaptively fits a forgetting curve. We then discuss the performance of these models against study datasets comprised of millions of answers and show that the RPL approach is more accurate and flexible than the MLR model. Finally, we give an overview of promising future approaches to knowledge modeling.
... Classical educational hypermedia system mainly adapted to compensation of knowledge deficits, ergonomic reasons, or adaptations to learning styles for an easier introduction into a topic. Furthermore examples of adaptive methods can be found in different research areas as Intelligent Tutoring Systems (Anderson et al., 1989;Carbonell, 1970;Clancey, 1987), Adaptive User Interfaces (Brusilovsky et al., 1995;Carroll, 1984), Adaptive Hypermedia (Brusilovsky, 1996), Intelligent Multimedia or Intelligent Agents for Learning (Feiner & McKeown, 1993;Rickel & Johnson, 1997). Examples found in the literature can be mostly classified to the scheme introduced above. ...
Article
Full-text available
Research report of the ProLearn Network of Excellence (IST 507310), Deliverable 1.11
... Intelligent Tutoring Systems (ITSs) are a natural solution to this need, as they are designed to give individualized feedback and assistance to students who are working on problems. In fact, ITSs have been designed for programming in structured editors in the past, and have been experimentally demonstrated to produce large positive effects on students learning (Anderson et al. 1989). However, traditional ITSs have a major drawback: they take much time and expert knowledge to create. ...
Article
Full-text available
To provide personalized help to students who are working on code-writing problems, we introduce a data-driven tutoring system, ITAP (Intelligent Teaching Assistant for Programming). ITAP uses state abstraction, path construction, and state reification to automatically generate personalized hints for students, even when given states that have not occurred in the data before. We provide a detailed description of the system’s implementation and perform a technical evaluation on a small set of data to determine the effectiveness of the component algorithms and ITAP’s potential for self-improvement. The results show that ITAP is capable of producing hints for almost any given state after being given only a single reference solution, and that it can improve its performance by collecting data over time.
... A) There had been prior field trials including use of the LISP programming tutor with college students starting in 1984 (Anderson et al. 1989). Intelligent tutors for high school math tutors had also been used in smaller-scale experimental comparisons in schools (cf., Schofield et al. 1990). ...
Article
Full-text available
Our 1997 article in IJAIED reported on a study that showed that a new algebra curriculum with an embedded intelligent tutoring system (the Algebra Cognitive Tutor) dramatically enhanced high-school students’ learning. The main motivation for the study was to demonstrate that intelligent tutors that have cognitive science research embedded in them could have real impact in schools. This study was one of the first large-scale classroom evaluations of the integrated use of an Intelligent Tutoring System (ITS) in high schools. A core challenge was figuring out how to embed this new technology into a curriculum and into the existing social context of schools. A key element of the study design was to include multiple kinds of assessments, including standardized test items and items measuring complex problem solving and use of representations. The results were powerful: “On average the 470 students in experimental classes outperformed students in comparison classes by 15 % on standardized tests and 100 % on tests targeting the [course] objectives.” We suggested that the study was evidence “that laboratory tutoring systems can be scaled up and made to work, both technically and pedagogically, in real and unforgiving settings like urban high schools.” Since this study, many more classroom studies comparing instruction that includes an ITS against business as usual have been conducted, often showing advantages for the ITS-enhanced curricula. More rigorous randomized field trials are now more commonplace, but the approach of using multiple assessments in large-scale randomized field trials has not caught on. Cognitive task analysis will remain fundamental to the success of ITSs. A key remaining question for ITS is to find out how they can be used most effectively to support open-ended problem solving, either online or offline. Given all the recent excitement around Massive Open Online Courses (MOOCs), it is interesting to note that our field of Artificial Intelligence in Education has been making huge, less recognized, progress with impact on millions of students and with the majority of those students finishing the course!
... Les travaux complémentaires menés en didactique professionnelle s'intéressent aux difficultés et aux représentations élaborées au cours de l'apprentissage du domaine. Cette approche n'est pas nouvelle [voir par exemple Leplat, 2001], de même que son application à des systèmes pour l'apprentissage ; par exemple [Bisseret, 1969], [Anderson, 1989 ], [Jonson, 1985 ], [Burkhardt, 1998], [Samurcay, 1998], [Betrancourt, 2000] [Boucheix, 1999]. Plus récente dans le contexte des environnements virtuels pour l'apprentissage, cette approche met l'accent sur l'étude et la spécification de modes puissants et originaux de présentation et de structuration de l'information. ...
... We are not aware of any true experimental studies that did exactly that, although a number of studies approached this ideal quite closely. A study by Anderson and colleagues tested the value of explanatory content in hint and feedback messages in the Lisp Tutor (Anderson, Conrad, & Corbett, 1989). This study compared a version of the tutor that included explanatory content in on-demand hints and system-generated feedback messages against a tutor version that gave only bottom-out hints and only correctness feedback (i.e., did not provide explanatory content). ...
Article
Help seeking is an important process in self-regulated learning (SRL). It may influence learning with intelligent tutoring systems (ITSs), because many ITSs provide help, often at the student’s request. The Help Tutor was a tutor agent that gave in-context, real-time feedback on students’ help-seeking behavior, as they were learning with an ITS. Key goals were to help students become better self-regulated learners and help them achieve better domain-level learning outcomes. In a classroom study, feedback on help seeking helped students to use on-demand help more deliberately, even after the feedback was no longer given, but not to achieve better learning outcomes. The work made a number of contributions, including the creation of a knowledge-engineered, rule-based, executable model of help seeking that can drive tutoring. We review these contributions from a contemporary perspective, with a theoretical analysis, a review of recent empirical literature on help seeking with ITSs, and methodological suggestions. Although we do not view on-demand, principle-based help during tutored problem solving as being as important as we once did, we still view it as helpful under certain circumstances, and recommend that it be included in ITSs. We view the goal of helping students become better self-regulated learners as one of the grand challenges in ITSs research today.
... It is also the case that there is a vast middle ground of educational research in which the distinction between basic versus domain specifi c work is often blurred (Anderson, Conrad, & Corbett, 1989;Corbett & An derson, 1988;Singley & Anderson, 1989) . However, at the further extreme are the attempts to model rare forms of expertise, such as that possessed by Submarine Com manders (Ehret, Gray, & Kirschenbaum, 2000;Gray, Kirschenbaum, & Ehret, 1997;Kirschen baum & Gray, 2000), uninhabited air ve hicle (UA V) operators (Gluck et al., 2007), or airline pilots (Byrne & Kirlik, 2005). ...
Chapter
Full-text available
[from the Introduction] Cognitive engineering is the application of cognitive science theories to human factors practice. As this description suggests, there are strong symbioses between cognitive engineering and cognitive science, but there are also strong differences. Symbiosis implies a mutual influence, and the history of cognitive engineering supports this characterization in two key areas: the development of cognitive theory and the development of computational modeling software. For theory development, a stringent test of our understanding of cognitive processes is whether we can apply our knowledge to real-world problems. The degree to which we succeed at this task is the degree to which we have developed robust and powerful theories. The degree to which we fail at this task is the degree to which more research and stronger theories are required (Gray, Schoelles, & Myers, 2004).
... An interesting finding from his laboratory is that if the feedback (KR) is provided after each trial, immediate performance is better than when only summary performance is given every 15 trials; however, long-term or ultimate learning is facilitated by giving only summary feedback intermittently such as after 15 trials. Anderson, Conrad and Corbett (1989) examined feedback in the context of learning the programming language LISP. They taught the language using a computer based tutor and varied whether or not they provided feedback and when it was provided. ...
... As it quickly became apparent that the availability of hints did not ensure their effective use, work began to identify the factors that led to a positive relationship between help-seeking behaviors and student learning. In one of the earliest studies, Anderson et al., (1989) compared the use of explanatory hints and so-called bottom-out hints (which simply provided the student with the correct answer) and found that neither hint type was correlated with learning. In part, this may have been due to selection bias. ...
Article
Full-text available
Educational technology (EdTech) designers need to ensure population validity as they attempt to meet the individual needs of all students. EdTech researchers often have access to larger and more diverse samples of student data to test replication across broad demographic contexts as compared to either the small-scale experiments or the larger convenience samples often seen in experimental psychology studies of learning. However, the source of typical EdTech data (i.e., online learning systems) and concerns related to student privacy often limit the opportunity to collect demographic variables from individual students—the sample is diverse, but the researcher does not know how that diversity is realized in individual learners. In order to ensure equitable student outcomes, the EdTech community should make greater efforts to develop new methods for addressing this shortcoming. Recent work has sought to address this issue by investigating publicly-available, school-level differences in demographics, which can be useful when individual-level variation may be difficult or impossible to acquire data for. In this study, we use this approach to better understand the role of social factors in students’ self-regulated learning and motivation-related behaviors, behaviors whose effectiveness appears to be highly variable between groups. We demonstrate that school-level demographics can be significantly associated with the relationships between students’ help-seeking behavior, motivation, and outcomes (math performance and math self-concept). We do so in the context of reasoning mind, an intelligent tutoring system for elementary mathematics. By studying the conditions under which these relationships vary across different demographic contexts, we challenge implicit assumptions of generalizability and provide an evidence-based commentary on future research practices in the EdTech community surrounding how we consider diversity in our field’s investigations.
... Therefore, example-tracing tutors can meet the first requirement. They also meet the second factor: As mentioned, example-tracing tutors and the Tutorshop support individualized problem selection based on Bayesian Knowledge Tracing Anderson et al. 1989;Corbett et al. 2000;VanLehn 2011). ...
Article
Full-text available
In 2009, we reported on a new Intelligent Tutoring Systems (ITS) technology, example-tracing tutors, that can be built without programming using the Cognitive Tutor Authoring Tools (CTAT). Creating example-tracing tutors was shown to be 4–8 times as cost-effective as estimates for ITS development from the literature. Since 2009, CTAT and its associated learning management system, the Tutorshop, have been extended and have been used for both research and real-world instruction. As evidence that example-tracing tutors are an effective and mature ITS paradigm, CTAT-built tutors have been used by approximately 44,000 students and account for 40 % of the data sets in DataShop, a large open repository for educational technology data sets. We review 18 example-tracing tutors built since 2009, which have been shown to be effective in helping students learn in real educational settings, often with large pre/post effect sizes. These tutors support a variety of pedagogical approaches, beyond step-based problem solving, including collaborative learning, educational games, and guided invention activities. CTAT and other ITS authoring tools illustrate that non-programmer approaches to building ITS are viable and useful and will likely play a key role in making ITS widespread.
... Given the challenges related to learning to program, some early work focused on identifying expert programmers' mental representations (Gilmore and Green 1988), and how these representations could be incorporated into teaching activities to help novices learn (Schank et al. 1993;Soloway 1986). There was also interest in computational support for programming activities delivered through tutoring systems (Anderson et al. 1989;Bhuiyan et al. 1994). Jim Greer and colleagues championed some of this work. ...
Article
When students are first learning to program, they not only have to learn to write programs, but also how to trace them. Code tracing involves stepping through a program step-by-step, which helps to predict the output of the program and identify bugs. Students routinely struggle with this activity, as evidenced by prior work and our own experiences in the classroom. To address this, we designed a Code Tracing (CT)-Tutor. We varied the level of assistance provided in the tutor, based on (1) the interface scaffolding available during code tracing, and (2) instructional order, operationalized by when examples were provided, either before or after the corresponding problem was solved. We collected data by having participants use the tutor to solve code tracing problems (N = 97) and analyzed both learning outcomes and process data obtained by extracting features of interest from the log files. We used a multi-layered approach for the analysis, including standard inferential statistics and unsupervised learning to cluster students by their behaviors in the tutor. The results show that the optimal level of assistance for code tracing falls in the middle of the assistance spectrum included in the tutor, but also that there are individual differences in terms of optimal assistance for subgroups of individuals. Based on these results, we outline opportunities for future work around personalizing instruction for code tracing.
... Computer tutors tried low-level evaluation exercises, but sequenced with writing exercises; this had no bene ts for writing skills vs. only having writing exercises [2]. Other work implements a comprehension-rst pedagogy but has limitations in their evaluations or the breadth of what they teach. ...
Conference Paper
Full-text available
What knowledge does learning programming require? Prior work has focused on theorizing program writing and problem solving skills. We examine program comprehension and propose a formal theory of program tracing knowledge based on control flow paths through an interpreter program's source code. Because novices cannot understand the interpreter's programming language notation, we transform it into causal relationships from code tokens to instructions to machine state changes. To teach this knowledge, we propose a comprehension-first pedagogy based on causal inference, by showing, explaining, and assessing each path by stepping through concrete examples within many example programs. To assess this pedagogy, we built PLTutor, a tutorial system with a fixed curriculum of example programs. We evaluate learning gains among self-selected CS1 students using a block randomized lab study comparing PLTutor with Codecademy, a writing tutorial. In our small study, we find some evidence of improved learning gains on the SCS1, with average learning gains of PLTutor 60% higher than Codecademy (gain of 3.89 vs. 2.42 out of 27 questions). These gains strongly predicted midterms (R²=.64) only for PLTutor participants, whose grades showed less variation and no failures.
... We show in Figure 7 where each concept is faded in the CF version of the game. Unlike textual introductions which must present many concepts "at once"-the complexity of which may unnecessarily hamper or confuse novices [8,2]our approach manages cognitive complexity by introducing concepts in relative isolation, then gradually mixing previously learned concepts to strengthen understanding of the whole, in a repeating pattern informed by theories of elaboration and flow [68,25] and lessons from cognitive tutors [9]. must now also understand replication and first-class function application (functions as input to other functions). ...
Conference Paper
Dominant approaches to programming education emphasize program construction over language comprehension. We present Reduct, an educational game embodying a new, comprehension-first approach to teaching novices core programming concepts which include functions, Booleans, equality, conditionals, and mapping functions over sets. In this novel teaching strategy, the player executes code using reduction-based operational semantics. During gameplay, code representations fade from concrete, block-based graphics to the actual syntax of JavaScript ES2015. We describe our design rationale and report on the results of a study evaluating the efficacy of our approach on young adults (18+) without prior coding experience. In a short timeframe, novices demonstrated promising learning of core concepts expressed in actual JavaScript. We also present results from an online deployment. Finally, we discuss ramifications for the design of future computational thinking games.
... While many students find solving Parsons problems engaging [34], students sometimes struggle to solve the problems, and some students give up without ever solving them [21]. Since learning gains are based on the number of practice problems that students solve and understand [1], scaffolding that allows students to solve more problems should improve learning. However, it is also important that practice problems challenge the learner. ...
Conference Paper
Practice is essential for learning. There is evidence that solving Parsons problems (putting mixed up code blocks in order) is a more efficient, but just as effective, form of practice than writing code from scratch. However, not all students successfully solve every Parsons problem. Making the problems adaptive, so that the difficulty changes based on the learner's performance, should keep the learner in Vygotsky's zone of proximal development and maximize learning gains. This paper reports on a study comparing the efficiency and effectiveness of learning from solving adaptive Parsons problems vs non-adaptive Parsons problem vs writing the equivalent code. The adaptive Parsons problems used both intra-problem and inter-problem adaptation. Intra-problem adaptation means that if the learner is struggling to solve the current problem, the problem can dynamically be made easier. Inter-problem adaptation means that the difficulty of the next problem is modified based on the learner's performance on the previous problem. This study provides evidence that solving intra-problem and inter-problem adaptive Parsons problems is a more efficient, but just as effective, form of practice as writing the equivalent code.
... The greatest learning on a skill occurs early on and lessens with additional practice (Newell & Rosenbloom, 1981). However, Anderson et al. (1989) found that this power relationship may not always hold true with complex skills. In general, learning curves can be used to demonstrate students' learning rates over repeated trials practicing a skill (Koedinger & Mathan, 2004). ...
Article
Full-text available
Abstract We have conducted a study on how many opportunities are necessary, on average, for learners to achieve mastery of a skill, also called a knowledge component (KC), as defined in the Open Learning Initiative (OLI) digital courseware. The study used datasets from 74 different course instances in four topic areas comprising 3813 students and 1.2 million transactions. The analysis supports our claim that the number of opportunities to reach mastery gives us new information on both students and the development of course components. Among the conclusions are a minimum of seven opportunities are necessary for each knowledge component, more if the prior knowledge among students are uneven within a course. The number of KCs in a course increases the number of opportunities needed. The number of opportunities to reach mastery can be used to identify KCs that are outliers that may be in need of better explanations or further instruction.
... While I have found no reading-first program tracing tutoring systems, intelligent tutoring systems have included tracing practice without a full unified curriculum. Intelligent tutoring systems like ITEM/IP and a version of the LISP tutor included some tracing in their curriculum but interleaved with writing practice [2,3]. The Problets collection of tutors for mostly tracing practice lack conceptual instruction and a unified curriculum for a Turing-complete programming language [6]. ...
Conference Paper
Full-text available
Two large multinational studies show more than 60% of students incorrectly answer questions about the execution of basic programs. Past work has explored writing-focused curricula, but little work explores a reading-first pedagogy, teaching and assessing programming language semantics before teaching learners to write code. The hypothesis of my thesis is "An adaptive and motivating reading-first tutorial helps people learn program tracing skills and improves later learning of writing and debugging skills." Towards that goal, I built and evaluated a reading-first tutorial (PLTutor) with a fixed, non-adaptive curriculum, showing 60% higher learning gains (3.9 vs 2.4 on the Second CS1 Assessment (SCS1)) than the writing-focused tutorial Codecademy. I designed and did initial validity studies for a formative assessment for program tracing skills, based on a new formal knowledge model. Next, I will build an adaptive version of PLTutor and evaluate it. In the Koli DC I hope to refine my thesis' framing and narrative, get advice on my academic job search, and connect with and potentially help other PhD students and researchers.
... The acquisition of skill is divided in three overlapping phases: (1) knowledge acquisition, (2) knowledge association, and (3) autonomous task performance [18], [19]. For programming skill, we use the definition of Bergersen et al. [19], which is in accordance to the definition used in psychology [18], [20], [21], [22], [23]: "the ability to use one's knowledge effectively and readily in execution or performance of programming tasks." In previous work with programmers, participants were often expected to have programming skill in various programming languages such as Python, Java, C, Perl, Haskell, JavaScript, PHP, etc. (e.g., [3], [7], [9], [10], [15], [19], [24], [25], [26], [27]). ...
Preprint
Recruiting professional programmers in sufficient numbers for research studies can be challenging because they often cannot spare the time, or due to their geographical distribution and potentially the cost involved. Online platforms such as Clickworker or Qualtrics do provide options to recruit participants with programming skill; however, misunderstandings and fraud can be an issue. This can result in participants without programming skill taking part in studies and surveys. If these participants are not detected, they can cause detrimental noise in the survey data. In this paper, we develop screener questions that are easy and quick to answer for people with programming skill but difficult to answer correctly for those without. In order to evaluate our questionnaire for efficacy and efficiency, we recruited several batches of participants with and without programming skill and tested the questions. In our batch 42% of Clickworkers stating that they have programming skill did not meet our criteria and we would recommend filtering these from studies. We also evaluated the questions in an adversarial setting. We conclude with a set of recommended questions which researchers can use to recruit participants with programming skill from online platforms.
... Templates are a form of program schema [16,20] that students can recall and reuse in constructing solutions to new problems. The LISP Tutor [1] builds on a theory that students can recognize and adapt solutions to recursive problems, though without an explicit step of articulating the template independently from the code. ...
Conference Paper
In many universities, Teaching Assistants (TAs) are an important part of students' educational experience. This is especially true in early courses, where students may suffer from inexperience and anxiety, and find fellow students more accessible than professors. Despite its importance, this learning channel has not been studied very much. Part of the difficulty lies in how to meaningfully evaluate it. Any intervention needs to be both unintrusive and lightweight, and yet yield useful data. As a result, to many faculty and researchers, TA office hours remain fairly opaque. This paper presents one approach to studying the technical component (but not the social dynamics) of TA office hours. We use a program-design methodology as a device to help track what students are asking about in hours, using a simple survey-based method to gather data. Data from TAs effectively summarize students' questions. In addition, contrasting data from both TAs and students provides insight into students' progress on program design help-seeking over the course of the semester.
... This is especially true in large classrooms and MOOCs, where instructors cannot interact with every student [17,21]. To address this, researchers have developed intelligent systems for automatically generating support for students during programming [2,13,16,31], such as hints and feedback, which have been shown to improve students' learning outcomes [8,9]. These systems often offer next step, edit-based hints, which suggest an edit that a student should make to bring their code closer to a correct solution [10,28]. ...
Conference Paper
Automated hints, a powerful feature of many programming environments, have been shown to improve students' performance and learning. New methods for generating these hints use historical data, allowing them to scale easily to new classrooms and contexts. These scalable methods often generate next-step, code hints that suggest a single edit for the student to make to their code. However, while these code hints tell the student what to do, they do not explain why, which can make these hints hard to interpret and decrease students' trust in their helpfulness. In this work, we augmented code hints by adding adaptive, textual explanations in a block-based, novice programming environment. We evaluated their impact in two controlled studies with novice learners to investigate how our results generalize to different populations. We measured the impact of textual explanations on novices' programming performance. We also used quantitative analysis of log data, self-explanation prompts, and frequent feedback surveys to evaluate novices' understanding and perception of the hints throughout the learning process. Our results showed that novices perceived hints with explanations as significantly more relevant and interpretable than those without explanations, and were also better able to connect these hints to their code and the assignment. However, we found little difference in novices' performance. Our results suggest that explanations have the potential to make code hints more useful, but it is unclear whether this translates into better overall performance and learning.
... Intelligent Tutoring Systems (ITSs) have been created for a variety of domains such as the military (Zachary et al., 1999), intelligent computer-assisted language learning (Gamper & Knapp, 2002) and education (Bradáč & Kostolányová, 2016). While they have been successful in providing personalized individual instruction in a variety of domains (e.g., programming, algebra, physics, and on-the-job training) (Anderson, 1989;Arroyo-Figueroa, Hernandez, & Sucar, 2006;Koedinger, 1997;VanLehn, van de Sande, Shelby, & Gershman, 2010), the complexity and difficulty of building an ITS has been well-documented (Murray, Blessing, & Ainsworth, 2003). Given the combinatorics of team member interactions and the need for a team tutor to track those interactions it is expected that building an intelligent team tutoring system (ITTS) is more complex. ...
... It is also seen in the Alef NextGen platform, a platform currently used by learners in the Middle East and New York City, which we will discuss below. Moreso, extended explanations have been a part of many successful intelligent tutoring systems, from the original LISP Tutor (Anderson, Conrad, & Corbett, 1989) to recent extensions to the ASSISTments system today . These systems represent a hybrid between continual learning/assessment and periodic learning/assessment, suggesting that the standard assumptions of most student modeling algorithms may also not hold in these cases. ...
Conference Paper
Full-text available
Student models for adaptive learning environments and intelligent tutoring systems typically assume a paradigm of use where a student completes exercises or activities, and learns from those exercises or activities. However, many modern systems, including MOOCs, intersperse declarative content or lecture with assessment of the learning from this content. In this paper, we present a variant of a common student modeling algorithm, Bayesian Knowledge Tracing, which assumes that most learning occurs during use of declarative content rather than between exercises. We compare this algorithm’s predictive ability to classic Bayesian Knowledge Tracing and another common algorithm, Performance Factors Assessment. We find that our new algorithm, BKT-PL, performs slightly better than algorithms designed for the standard intelligent tutoring paradigm. Moreso, we can use BKT-PL to determine which declarative content is most and least effective, to drive iterative re-design.
Conference Paper
Full-text available
There is a dire need for replication research in the learning sciences, as methods put forth for increasing student learning should be unequivocally grounded in reproducible, reliable research. Learning science research is not only a critical input in the learning engineering process during the development of educational technology tools, such as courseware, but also as an output after student data have been analyzed to determine if the learning methods used were effective for students in their natural learning context. Furthermore, research that can provide causal evidence that a method of learning is effective for students should be reproduced-and the generality for its use expanded-so that methods that cause learning gains can be widely applied. One such method is the doer effect: the principle that students who engage with more practice have higher learning gains than those who only read expository text or watch video. This effect has been shown to be causal in prior research through statistical modeling using data mined from natural learning contexts. The goal of this paper is to replicate this research using a large-scale data set from courseware used at a major online university. The learning-by-doing data recorded by the courseware platform were combined with final exam data to replicate the statistical model of the causal doer effect study. Results from this analysis similarly point to a causal relationship between doing practice and learning outcomes. The implications of these doer effect results and future learning science research using large-scale data analytics will be discussed.
Book
Full-text available
Design Recommendations for Intelligent Tutoring Systems (ITSs) explores the impact of intelligent tutoring system design on education and training. Specifically, this volume examines “Authoring Tools and Expert Modeling Techniques”. The “Design Recommendations book series examines tools and methods to reduce the time and skill required to develop Intelligent Tutoring Systems with the goal of improving the Generalized Intelligent Framework for Tutoring (GIFT). GIFT is a modular, service-oriented architecture developed to capture simplified authoring techniques, promote reuse and standardization of ITSs along with automated instructional techniques and effectiveness evaluation capabilities for adaptive tutoring tools and methods.
Article
Work in cognitive and educational psychology examines a variety of phenomena related to the learning and retrieval of information. Indeed, Alice Healy, our honoree, and her colleagues have conducted a large body of groundbreaking research on this topic. In this article we discuss how 3 learning principles (the generation effect, deliberate practice and feedback, and antidotes to disengagement) discussed in Healy, Schneider, and Bourne (2012) have influenced the design of 2 intelligent tutoring systems that attempt to incorporate principles of skill and knowledge acquisition. Specifically, this article describes iSTART-2 and the Writing Pal, which provide students with instruction and practice using comprehension and writing strategies. iSTART-2 provides students with training to use effective comprehension strategies while self-explaining complex text. The Writing Pal provides students with instruction and practice to use basic writing strategies when writing persuasive essays. Underlying these systems are the assumptions that students should be provided with initial instruction that breaks down the tasks into component skills and that deliberate practice should include active generation with meaningful feedback, all while remaining engaging. The implementation of these assumptions is complicated by the ill-defined natures of comprehension and writing and supported by the use of various natural language processing techniques. We argue that there is value in attempting to integrate empirically supported learning principles into educational activities, even when there is imperfect alignment between them. Examples from the design of iSTART-2 and Writing Pal guide this argument.
Article
In recent years, learning process data have become increasingly easy to collect through computer-based learning environments. This has led to increased interest in the field of learning analytics, which is concerned with leveraging learning process data in order to better understand, and ultimately to improve, teaching and learning. In computing education, the logical place to collect learning process data is through integrated development environments (IDEs), where computing students typically spend large amounts of time working on programming assignments. While the primary purpose of IDEs is to support computer programming, they might also be used as a mechanism for delivering learning interventions designed to enhance student learning. The possibility of using IDEs both to collect learning process data, and to strategically intervene in the learning process, suggests an exciting design space for computing education research: that of IDE-based learning analytics. In order to facilitate the systematic exploration of this design space, we present an IDE-based data analytics process model with four primary activities: (1) Collect data, (2) Analyze data, (3) Design intervention, and (4) Deliver intervention. For each activity, we identify key design dimensions and review relevant computing education literature. To provide guidance on designing effective interventions, we describe four relevant learning theories, and consider their implications for design. Based on our review, we present a call-to-action for future research into IDE-based learning analytics.
Article
The ability to solve complex problems in which considerable latitude in solution is allowed is an important goal in engineering education. While feedback is critical to learning, traditional grading of handwritten homework rarely provides effective, timely feedback on attempts to solve complex problems. To address the research question of whether automated feedback can be provided on complex problem solving, a cognitive tutor approach is pursued, with truss problems in statics used as a test case. The tutor maintains a cognitive model of problem solving of the class of problems, and associates various solution steps with distinct skills or knowledge components. One can determine whether students learn individual skills by measuring the error rate as a function of practice. Data was obtained from students using the tutor to solve truss problems in place of the weekly homework assignment. Statistical analyses of the data showed that distinct skills involved very different error rates, justifying tracking them separately. Furthermore, for most skills, the error rate either was low from the start or decreased with practice, providing evidence that feedback improved performance.
Article
Engineering professional societies, engineering education thought leaders, and academics in engineering education have established the need to promote systems thinking in engineering students so they can deal effectively with the future complexity of engineering practice. Universities and schools have taught systems thinking, but currently there are limitations in measuring the effectiveness of these instructional interventions. Current research on assessment states that it is necessary to align modern theories of learning, the task and the interpretation of the outcomes from the task. Theories about the development of systems thinking are still scarce and are not based in modern theories of learning. This work intends to contribute to closing this gap by identifying the variation of experiences people have when dealing with problems in complex systems. The research approach used to answer this question is phenomenography (Marton's approach). Participants will be asked to perform a task, and each common way of dealing with the task will be identified as a category. These categories will be hierarchically arranged, yielding a developmental path. A developmental path explaining the development of systems thinking will facilitate the development of tools to assess the level of systems thinking which impacts academy and industry.
Article
Data-driven intelligent tutoring systems learn to provide feedback based on past student behavior, reducing the effort required for their development. A major obstacle to applying data-driven methods in the programming domain is the lack of meaningful observable actions for describing the students' problem-solving process. We propose rewrite rules as a language-independent formalization of programming actions in terms of code edits. We describe a method for automatically extracting rewrite rules from students' program-writing traces, and a method for debugging new programs using these rules. We used these methods to automatically provide hints in a web application for learning programming. In-class evaluation showed that students receiving automatic feedback solved problems faster and submitted fewer incorrect programs. We believe that rewrite rules provide a good basis for further research into how humans write and debug programs.
Psychologists are interested in developing instructional policies that boost student learning. An instructional policy specifies the manner and content of instruction. For example, in the domain of concept learning, a policy might specify the nature of exemplars chosen over a training sequence. Traditional psychological studies compare several hand-selected policies, e.g., contrasting a policy that selects only difficult-to-classify exemplars with a policy that gradually progresses over the training sequence from easy exemplars to more difficult (known as fading). We propose an alternative to the traditional methodology in which we define a parameterized space of policies and search this space to identify the optimal policy. For example, in concept learning, policies might be described by a fading function that specifies exemplar difficulty over time. We propose an experimental technique for searching policy spaces using Gaussian process surrogate-based optimization and a generative model of student performance. Instead of evaluating a few experimental conditions each with many human subjects, as the traditional methodology does, our technique evaluates many experimental conditions each with a few subjects. Even though individual subjects provide only a noisy estimate of the population mean, the optimization method allows us to determine the shape of the policy space and to identify the global optimum, and is as efficient in its subject budget as a traditional A-B comparison. We evaluate the method via two behavioral studies, and suggest that the method has broad applicability to optimization problems involving humans outside the educational arena.
Preprint
Lernen erfordert vielfältige metakognitive Aktivitäten, die den Wissensaufbau und die Steuerung des eigenen Lernprozesses unterstützen. Dazu zählen die Auswahl und Planung von Zielen, die Anwendung, Beobachtung und Bewertung von Lernstrategien sowie regulierende Eingriffe zum Erreichen gesetzter Ziele. Der Artikel zeigt beispielhaft auf, wie eine KI-gestützte Trainingssoftware Lernende dabei fördern kann, sich Ziele zu setzen, Phasen fokussierten Arbeitens umzusetzen und ein sinnvolles Pausenmanagement einzubinden. Metakognitives Feedback bildet aufbauend auf Prinzipien des maschinellen Lernens den Wert zielgerichteten Handelns ab. In Pilotbefunden zeigen sich bereits erste positive Effekte dieses Ansatzes, die eine weitere Exploration nahelegen. Basierend darauf wird eine mögliche curriculare Einbindung im Lernfeld Schule sowie Implikationen für die Rolle der Lehrkräfte diskutiert.
Article
Full-text available
Lernen erfordert vielfältige metakognitive Aktivitäten, die den Wissensaufbau und die Steuerung des eigenen Lernprozesses unterstützen. Dazu zählen die Auswahl und Planung von Zielen, die Anwendung, Beobachtung und Bewertung von Lernstrategien sowie regulierende Eingriffe zum Erreichen gesetzter Ziele. Der Artikel zeigt beispielhaft auf, wie eine KI-gestützte Trainingssoftware Lernende dabei fördern kann, sich Ziele zu setzen, Phasen fokussierten Arbeitens umzusetzen und ein sinnvolles Pausenmanagement einzubinden. Metakognitives Feedback bildet aufbauend auf Prinzipien des maschinellen Lernens den Wert zielgerichteten Handelns ab. In Pilotbefunden zeigen sich bereits erste positive Effekte dieses Ansatzes, die eine weitere Exploration nahelegen. Basierend darauf wird eine mögliche curriculare Einbindung im Lernfeld Schule sowie Implikationen für die Rolle der Lehrkräfte diskutiert.
Conference Paper
Intelligent Tutoring Systems are highly effective at helping students learn, but have required intensive amounts of development time in the past, keeping teachers from making their own. Data-driven tutoring has made it possible to build these tutors more efficiently. For my thesis work, I intend to build an authoring tool for data-driven tutors that is designed to be used by computer science teachers. I plan to design this system based on data gathered in interviews with CS educators and evaluate it on its usability for new users.
Article
I shall describe a model of the evolution of rule-structured knowledge that serves as a cornerstone of our development of computer-based coaches. The key idea is a graph structure whose nodes represent rules, and whose links represent various evolutionary relationships such as generalization, correction, and refinement. I shall define this graph and describe a student simulation testbed which we are using to analyze different genetic graph formulations of the reasoning skills required to play an elementary mathematical game.
Article
Recent researches are cited which suggest that the acquisition of manual speed-skill proceeds by a certain type of selective action. A formal theoretical model is developed, and its predictions compared with the experimental results. Certain complications of the theory, and conclusions from it are outlined, and the nature of the selective mechanism is discussed. Some implications for training are indicated.
Article
Cognitive skills are encoded by a set of productions, which are organized according to a hierarchical goal structure. People solve problems in new domains by applying weak problem-solving procedures to declarative knowledge they have about this domain. From these initial problem solutions, production rules are compiled that are specific to that domain and that use of the knowledge. Numerous experimental results may be predicted from this conception of skill organization and skill acquisition. These include predictions about transfer among skills, differential improvement on problem types, effects of working memory limitations, and applications to instruction. The theory implies that all varieties of skill acquisition, including those typically regarded as inductive, conform to this characterization.
Article
Studies of students learning cognitive skills have revealed that private tutoring appears to be much more effective than conventional classroom instruction. A computer-based tutor is considered that is as effective in teaching LISP as a human tutor. GREATERP (Goal-Restricted Environment for Tutoring and Educational Research on Programming) is an attempt to combine artificial-intelligence technology and a psychological theory of skill acquisition into an effective teaching device. Two program listings give examples of dialogues between student and computer tutor.
Article
We describe the current version of the Why System, a script-based socratic tutor which uses tutoring strategies formulated as production rules. The current system is capable of carrying on a dialogue about the factors influencing rainfall by presenting different cases to the student, asking for predictions, probing for relevent factors, entrapping the student when he has not identified all necessary factors, and presenting counterexamples. The current system is incomplete because it lacks a goal structure to guide the tutorial sessions. We outline a more complete theory of the goal structure of Socratic tutors based on analysis of human tutorial dialogues. There are two top level goals: (1) refinement of the student's causal model and (2) refinement of the student's predictive abilities. The subgoals are diagnosis of bugs in the student's knowledge and correction of the bugs. This goal-driven control mechanism governs the selection of examples and teaching strategies used by the tutor.
Article
This article discusses cognitive models of learning to program recursion and their relation to lessons on recursion in an intelligent computer tutor for LISP programming (the LISP Tutor). The cognitive models are implemented as production systems in which programming skill is characterized as the decomposition of programming goals into subgoals and elementary actions via the application of programming plans. Two sets of learning mechanisms are used in the cognitive models. Analogical problem-solving mechanisms use declarative knowledge of example program solutions to overcome problem-solving impasses. Knowledge compilation mechanisms summarize problem solutions into efficient problem-solving skill. Analyses and simulations of novice and expert programming were used to develop ideal models of the programming knowledge to confer upon students and bugs that characterize common misconceptions. The LISP Tutor uses the ideal models and bugs to guide its interactions with students. Experimental evaluations of the LISP Tutor indicate that it is more efficient and effective than classroom instruction.
Article
The major results reported by R. A. Carlson et al (see record 1989-24864-001) confirm predictions of J. R. Anderson's (1983) ACT * theory. In particular, ACT * predicts the detrimental effects of the transition to randomized practice because of the need to learn new productions, the complexity effect of gate and judgment type because of more complex production conditions, the effects of practice and its interaction with complexity because of the strengthening mechanisms, and the effects of memory load because of the need to hold information active in working memory so that it can be matched by production conditions. (PsycINFO Database Record (c) 2012 APA, all rights reserved)
Article
Several theories assume that practice (a) results in restructuring of component processes and (b) reduces demand on working memory. Eight subjects practiced judgments about digital logic gates for over 8,000 trials. At two practice levels, subjects made judgments while retaining short-term memory loads irrelevant to the judgments, relevant but not accessed, or accessed to make the judgments. Four phenomena together provide constraints for theory: First, performance declined in moving from blocked practice to randomized practice. Second, gate and judgment type strongly affected latency. Third, these effects declined but did not disapppear with practice. Fourth, the cost of accessing information in working memory remained substantial. These results are interpreted as reflecting a serial process with constant structure, while component processes become faster. The results challenge theories assuming that all learning results from restructuring or that restructuring is an automatic consequence of practice, and they support a distributed view of working memory. (PsycINFO Database Record (c) 2012 APA, all rights reserved)
Article
This article presents a theory in which automatization is construed as the acquisition of a domain-specific knowledge base, formed of separate representations, instances, of each exposure to the task. Processing is considered automatic if it relies on retrieval of stored instances, which will occur only after practice in a consistent environment. Practice is important because it increases the amount retrieved and the speed of retrieval; consistency is important because it ensures that the retrieved instances will be useful. The theory accounts quantitatively for the power-function speed-up and predicts a power-function reduction in the standard deviation that is constrained to have the same exponent as the power function for the speed-up. The theory accounts for qualitative properties as well, explaining how some may disappear and others appear with practice. More generally, it provides an alternative to the modal view of automaticity. (PsycINFO Database Record (c) 2012 APA, all rights reserved)
Article
Describes cyclopean methodology, based on the author's stereopsis model in which binocular depth perception is a central process occurring at a specific site on the visual cortex termed the "cyclopean retina." Evidence of cyclopean perception is reviewed, and over 400 computer-generated, random-dot stereograms and anaglyphs are presented. (16 p. ref.) (PsycINFO Database Record (c) 2012 APA, all rights reserved)
Article
In the past two decades cognitive psychologists have started to adopt "production systems" as the principle theoretical medium in which to cast complex theories of human intelligence. Production systems are a class of computer simulation models that are stated in terms of condition-action rules, and that make strong assumptions about the nature of the cognitive architecture. The initial use of production systems was to provide theoretical accounts of human performance on a variety of tasks. However, one fundamental feature that distinguishes production systems from most other forms of cognitive simulation is their capability for self-modification. Because of this feature it quickly became apparent that they were eminently well suited for dealing with issues of development and learning. There has been no single volume exclusively devoted to production-system models of human cognition. For this reason we came to feel that a volume that enables readers to compare and contrast those research efforts would have special importance. All of the chapters in this book deal with the issues that arise when one constructs self-modifying production systems. The book should be of interest to any cognitive scientist interested in the processes underlying the acquisition of knowledge and skills, particularly researchers in psychology with an information-processing orientation, as well as cognitively oriented researchers in artificial intelligence. In planning this book, we sought to accomplish for the production-system paradigm what "Explorations in Cognition" did for the LNR Group, or what "Scripts, Plans, Goals and Understanding" did for the Yale AI group. That is, we have tried to provide a single volume that provides access to state-of-the-art research results and does so in a way that allows readers to gain some perspective on the underlying research approach. (PsycINFO Database Record (c) 2012 APA, all rights reserved)
Article
Proposed and evaluated 4 optimization strategies for the learning of a large German-English vocabulary using 120 undergraduates. The 1st strategy involved presenting items in a random order and served as a benchmark against which the others could be evaluated. The 2nd strategy permitted S to determine on each trial of the experiment which item was to be presented, thus placing instruction under "learner control." The 3rd and 4th strategies were based on a mathematical model of the learning process; these strategies were computer controlled and took account of S's response history in making decisions about which items to present next. Performance on a delayed test administered 1 wk. after the instructional session indicated that the learner-controlled strategy yielded a gain of 53% when compared to the random procedure, whereas the best of the 2 computer-controlled strategies yielded a gain of 108%. Implications for a theory of instruction are considered. (PsycINFO Database Record (c) 2012 APA, all rights reserved)
Article
Reviews existing theories of fluency in high-proficiency skills such as speech, and proposes that execution of behavior involves the activation of a hierarchy of nodes in proper serial order within an output system. Activating a node at any level in the system activates its connected nodes, and repeated activation increases the rate of priming per unit time, thereby allowing a faster rate of output at the lowest, muscle movement level. Relevance of this theory for several related issues is discussed: why behavior becomes more flexible with practice, transferring readily from one response mechanism to another; why there is almost perfect transfer from one hand to the other for simple skills, but less than perfect transfer for complex skills; why skills at higher, semantic levels transfer to new behavioral sequences, as when bilinguals produce a word-for-word translation of a practiced sentence in their other language. The theory also provides a new way of looking at motor equivalence, automaticity, speed–accuracy trade-off, subordinate autonomy, and the motor program. (32 ref) (PsycINFO Database Record (c) 2012 APA, all rights reserved)
Article
We have gathered protocols of subjects in their first 30 hours of learning LISP. The processes by which subjects write LISP functions to meet problem specifications has been modeled in a simulation program called GRAPES (Goal Restricted Production System). The GRAPES system embodies the goal-restricted architecture for production systems as specified in the ACT* theory ( [3] and [4]). We compare our simulation to human protocols on a number of problems. GRAPES simulates the top-down, depth-first flow of control exhibited by subjects and produces code very similar to subject code. Special attention is given to modeling student solutions by analogy, how students learn from doing, and how failures of working memory affect the course of problem-solving. Of major concern is the process by which GRAPES compiles operators in solving one problem to facilitate the solution of later problems.
Article
Three studies investigate how problem solvers learn to apply appropriate actions in problem solving. Part of this knowledge appears to result from learning the sets of problem features (schemata) that predict the success of different problem-solving actions (operators). A major claim is that this learning can be produced, in part, by the same mechanisms that produce concept formation and abstraction of object schemata. Studies using geometry proof problems and an abstract maze-searching task produce results similar to common findings in the schema abstraction literature: Performance improves as the prototypicality of the feature sets increases. Also examined are the effects of active processing of problem features, relevance of the discriminating features to the problem solution, amount of practice, and delay of feedback regarding the accuracy of operator selection. Subjects learn when to apply an operator better during active, deliberate hypothesis testing, regardless of feature relevance. Delayed feedback produces poorer performance with extended practice. A model of classification learning and a simulation of explicit hypothesis testing produced reasonable fits to the data. The failure to find evidence for unconscious learning is evidence against the automatic discrimination mechanism proposed in ACT∗.
Conference Paper
The tutor for doing proofs in high school geometry consists of a cot of ideal and buggy rules (IRR), a tutor, and an interface. The IBR is responsible for ehiuertly computing matcher, to all the correct and incorrect rules The interface is responsible for interacting with the student and graphically representing the proof. The tutor is responsible for directing the IBR and interface to achieve a current tutorial strategy. The strategy wo employ involves tracing the student's behavior in terms of what rules in the IBR it instantiates, correcting the student when behavior deviates below a minimum threshold, and helping the student over hurdles. While incomplete, current evidence indicates the geometry tutor is guite effective.
Article
Despite extensive discussion in the literature about the diagnosis and subsequent remediation of students' errors, few studies have compared the effects of different styles of error-based remediation. Swan (1983) found that a conflict approach (pointing out errors made by students and demonstrating their consequences) was more effective than simple reteaching, but Bunderson & Olsen (1983) found no difference between error-specific remediation and reteaching. More studies are needed in order to understand the factors which lead to successful remediation. The three studies discussed in this article compared error-specific or model-based remediation (MBR) with reteaching in algebra. MBR bases its remediation on the model inferred for an individual student before reteaching the correct procedure. Reteaching simply shows students the correct procedure without addressing specific errors. The results show that MBR and reteaching are both more effective than no tutoring; however, MBR is not clearly more effective than reteaching. The results are discussed in terms of stability of errors, their relevance to educational practice and to intelligent tutoring systems (ITS). Although the studies were carried out using human tutors, the results suggest that for the purpose of remediation in the algebra domain, when taught procedurally, “classical” computer-assisted instruction (CAI) would be as effective as an ITS.
Article
Thesis (Ph. D.)--Carnegie-Mellon University, 1988. Includes bibliographical references (leaves 129-131). Microfiche. s
Transfer of skills between inconsistent editors. Human-Computer Interaction The chunking of goal hierarchies: A generalized model of practice Learning by chunkhrg: A production system model of practice The transfer of cognitive skills
  • Pg Poison
  • E Muncher
  • D E Kieras
  • P S Rosenbloom
  • A D &newell
  • P K&r
  • R Langley
  • Neches
Poison, PG., Muncher, E., & Kieras, D.E. (ii press). Transfer of skills between inconsistent editors. Human-Computer Interaction. Rosenbloom, P.S, &Newell, A. (1986). The chunking of goal hierarchies: A generalized model of practice. In R.S. Michalski, J.G. Carbonell. & T.M. Mitchell (Eds.), Machine Learn-ing (Vol. II, pp. 247-288). Los Altos, CA: Morgan Kaufmann. Rosenbloom, P., & Newell, A. (1987). Learning by chunkhrg: A production system model of practice. In D. K&r, P. Langley, & R. Neches, (Eds.), Production System Models of Learning and Development. Cambridge, MA: MIT Press. Singley, K., & Anderson, J.R. (1989). The transfer of cognitive skills. Cambridge, MA: Harvard Press.
wfects of immediacy of feedback on learning proof skills
  • M W Lewis
  • J R Anderson
Lewis, M.W., & Anderson, J.R. (1989). wfects of immediacy of feedback on learning proof skills. Manuscript in preparation.
Transfer of programming skills in novice LISP learners. Unpublished doctoral dissertatioan, Carnegie-Mellon University Production system models ofpractice e$fects. Unpublished doctoral dis-sertation
  • K Kessler
Kessler, K. (1988). Transfer of programming skills in novice LISP learners. Unpublished doctoral dissertatioan, Carnegie-Mellon University, Pittsburgh, PA. Lewis, C.H. (1978). Production system models ofpractice e$fects. Unpublished doctoral dis-sertation. University of Michigan, Ann Arbor.
Leaming to program recursion
  • P L Pirolli
  • J R &anderson
Pirolli, P.L., &Anderson, J.R. (1984). Leaming to program recursion. In Sixth Annual Cog-nitive Science Meetings (pp. 277-280). Boulder, CO.
Automatic programming. In Artificial Intelligence Handbook
  • A Barr
  • E A Feigenbaum
Barr, A., & Feigenbaum, E. A. (1982). Automatic programming. In Artificial Intelligence Handbook. Los Altos, CA: Kaufmann.
Impact of feedback during complex skill acquisition Frequency and practice effects on the composition on knowledge in LISP evaluation (pp. 236-259) Cognitive aspects of human-computer interaction
  • J E Mckendree
  • Pa Mckendree
  • J E &anderson
McKendree, J.E. (1986). Impact of feedback during complex skill acquisition. Unpublished doctoral dissertation, Carnegie-Mellon University, Pittsburgh, PA. McKendree, J.E., &Anderson, J.R. (1987). Frequency and practice effects on the composition on knowledge in LISP evaluation (pp. 236-259). In J.M. Carroll (Ed.), Cognitive aspects of human-computer interaction. Cambridge, MA: MIT Press.
Foundationsof Cyclopeanperception
  • B Julesa
Julesa. B. (1971). Foundationsof Cyclopeanperception. Chicago: University of Chicago Press.
Production system models ofpractice e$fects
  • C H Lewis
Lewis, C.H. (1978). Production system models ofpractice e$fects. Unpublished doctoral dissertation. University of Michigan, Ann Arbor.
  • J R Anderson
  • A T Corbett
  • B J Breiser
Anderson, J.R., Corbett, A.T., BReiser, B.J. (1987). ErrentialLISP. Reading, MA: Addison- Wesley.
Impact of feedback during complex skill acquisition. Unpublished doctoral dissertation
  • J E Mckendree
McKendree, J.E. (1986). Impact of feedback during complex skill acquisition. Unpublished doctoral dissertation, Carnegie-Mellon University, Pittsburgh, PA.
  • J R Anderson
  • A T Corbett
  • B J Breiser
Anderson, J.R., Corbett, A.T., BReiser, B.J. (1987). ErrentialLISP. Reading, MA: Addison-Wesley.
Intelligent tutoring systems: At the crossroads of artificial intelligence and education
  • A T Corbett
  • J R Anderson
  • E J Patterson
Corbett, A.T., Anderson, J.R., & Patterson, E.J. (in press). Student modeling and tutoring flexibility in the LISP tutor. In C. Frasson & G. Gauthier (Eds.), Intelligent tutoring systems: At the crossroads of artificial intelligence and education. Nonvood, NJ: Ablex.
A cognitive model and computer tutor for programming recursion
  • P Piiolli
Piiolli, P. (1986). A cognitive model and computer tutor for programming recursion. Human-Computer Interaction, 2, 319-355.