Kurt Vanlehn

Arizona State University | ASU · School of Computing, Informatics, and Decision Systems Engineering

An algebraic model uses a set of algebraic equations to describe a situation. Constructing such models is a fundamental skill, but many students still lack the skill, even after taking several algebra courses in high school and college. For such students, we developed instruction that taught students to decompose the to-be-modelled situation into s...

An Intelligent Orchestration System, such as our FACT [1], should act like an automated teaching assistant that helps teachers provide relevant, timely help. To do so, it needs to know what the students are doing and thus who needs help more than the others. This is especially important when students work in small groups and the teacher’s ability t...

An algebraic model uses a set of algebraic equations to describe a situation. Constructing such models is a fundamental skill, but many students still lack the skill, even after taking several algebra courses in high school and college. For underachieving college students, we developed a tutoring system that taught students to decompose the to-be-m...

An algebraic model uses a set of algebra equations to precisely describe a situation. Constructing such models is a fundamental skill required by US standards for both math and science. It is usually taught with algebra word problems. However, many students still lack the skill, even after taking several algebra courses in high school and college....

Our system classifies audio from microphones worn by the teacher in order to determine (1) whether the teacher is addressing the whole class or talking to individuals or groups of students. In the latter case, it determines (2) whether the teacher is giving formative feedback, giving corrective feedback, chatting socially, or addressing administrat...

Research in the field of collaboration shows that students do not spontaneously collaborate with each other. A system that can measure collaboration in real time could be useful by, for example, helping the teacher locate a group requiring guidance. To address this challenge, my research focuses on building and comparing collaboration detectors for...

Collaboration is a 21st Century skill as well as an effective method for learning, so detection of collaboration is important for both assessment and instruction. Speech-based collaboration detection can be quite accurate but collecting the speech of students in classrooms can raise privacy issues. An alternative is to send only whether or not the...

FACT (Formative Assessment with Computational Technology) is an intelligent orchestration system. That is, because it helps the teacher manage the workflow of a complicated set of activities in the classroom, it is an orchestration system. Because it conducts tasks-specific and domain-specific analyses of the students’ mathematical products and the...

Despite their drawback, multiple-choice questions are an enduring feature in instruction because they can be answered more rapidly than open response questions and they are easily scored. However, it can be difficult to generate good incorrect choices (called “distractors”). We designed an algorithm to generate distractors from a semantic network f...

Mathematics is often taught by explaining an idea, then giving students practice in applying it. Tutoring systems can increase the effectiveness of this method by monitoring the students’ practice and giving feedback. However, math can also be taught by having students work collaboratively on problems that lead them to discover the idea. Here, teac...

Most approaches to student modeling assume that students’ knowledge can be represented by a large set of knowledge components that are learned independently. Knowledge components typically represent fairly small pieces of knowledge. This seems to conflict with the literature on problem solving which suggests that expert knowledge is composed of lar...

Artificial intelligence (AI) holds great promise for improving classroom orchestration—the teacher’s management of a classroom workflow that mixes small group, individual, and whole class activities. Although we have developed an orchestration system, named FACT, that uses AI, we were concerned that usability issues might decrease its effectiveness...

We are developing an intelligent orchestration system named FACT (Formative Assessment using Computational Technology). Orchestration refers the teacher’s management of a face-to-face classroom workflow that mixes small group, individual and whole class activities. FACT is composed of an unintelligent Media system and an intelligent Analysis system...

Background The Office of Naval Research (ONR) organized a STEM Challenge initiative to explore how intelligent tutoring systems (ITSs) can be developed in a reasonable amount of time to help students learn STEM topics. This competitive initiative sponsored four teams that separately developed systems that covered topics in mathematics, electronics,...

Effective collaboration between student peers is not spontaneous. A system that can measure collaboration in real-time may be useful, as it could alert an instructor to pairs that need help in collaborating effectively. We tested whether superficial measures of speech and user interface actions would suffice for measuring collaboration. Pairs of st...

Background: The Office of Naval Research (ONR) organized a STEM challenge initiative to explore how intelligent tutoring systems (ITS) can be developed in a reasonable amount of time to help students learn STEM topics. This competitive initiative sponsored four teams that separately developed systems that covered topics in mathematics, electronics,...

Science instructors need questions for use in exams, homework assignments, class discussions, reviews, and other instructional activities. Textbooks never have enough questions, so instructors must find them from other sources or generate their own questions. In order to supply biology instructors with questions for college students in introductory...

This project aimed to improve students’ learning and task performance using a non-cognitive learning companion in the context of both a tutor and a meta-tutor. The tutor taught students how to construct models of dynamic systems and the meta-tutor taught students a learning strategy. The non-cognitive learning companion was designed to increase stu...

The paper describes a biology tutoring system with adaptive question selection. Questions were selected for presentation to the student based on their utilities, which were estimated from the chance that the student’s competence would increase if the questions were asked. Competence was represented by the probability of mastery of a set of biology...

Formative Assessment is difficult to apply in real-world classrooms due to the requirement for extensive interaction between students and teachers. We have constructed a distributed system called FACT for in-class use that facilitates the use of popular Classroom Challenges (CCs) developed by the Mathematics Assessment Project. FACT lets students w...

This paper describes Dragoon, a simple intelligent tutoring system which teaches the construction of models of dynamic systems. Modelling is one of seven practices dictated in two new sets of educational standards in the U.S.A., and Dragoon is one of the first systems for teaching model construction for dynamic systems. Dragoon can be classified as...

A common hypothesis is that students will more deeply understand dynamic systems and other complex phenomena if they construct computational models of them. Attempts to demonstrate the advantages of model construction have been stymied by the long time required for students to acquire skill in model construction. In order to make model construction...

Constructing models of dynamic systems is an important skill in both mathematics and science instruction. However, it has proved difficult to teach. Dragoon is an intelligent tutoring system intended to quickly and effectively teach this important skill. This paper describes Dragoon and an evaluation of it. The evaluation randomly assigned students...

An orchestration system helps the teacher conduct a face-to-face class that includes small group work as well as activities by individuals and the whole class. The FACT (Formative Assessment with Computational Technology) system is an orchestration system designed to support formative assessment and collaborative learning. Students use tablets to e...

Although the Andes project produced many results over its 18 years of activity, this commentary focuses on its contributions to understanding how a goal-free user interface impacts the overall design and performance of a step-based tutoring system. Whereas a goal-aligned user interface displays relevant goals as blank boxes or empty locations that...

This commentary suggests a generalization of the conception of the behavior of tutoring systems, which the target article characterized as having an outer loop that was executed once per task and an inner loop that was executed once per step of the task. A more general conception sees these two loops as instances of regulative loops, which repeated...

The authors investigated whether some advantages of tutoring over other instructional methods are due to microadaptation, or, tutors basing their actions on assessments of tutees they develop during tutoring. In a 2 × 2 between-subjects experiment, independent variables were shared experience (tutors either worked with the same or a different tutee...

The objective of articulating sustainability visions through modeling is to enhance the outcomes and process of visioning in order to successfully move the system toward a desired state. Models emphasize approaches to develop visions that are viable and resilient and are crafted to adhere to sustainability principles. This approach is largely assem...

Intelligent Tutoring Systems (ITSs) constitute an alternative to expert human tutors, providing direct customized instruction and feedback to students. ITSs could positively impact education if adopted on a large scale, but doing that requires tools to enable their mass production. This circumstance is the key motivation for this work. We present a...

It is often assumed that one-on-one dialogue with a tutor, which involves micro-steps, is more effective than conventional step-based tutoring. Although earlier research often has not supported this hypothesis, it may be because tutors often are not good at making micro-step decisions. In this paper, we compare a micro-step based NL-tutoring system...

The Affective Meta-Tutoring system is comprised of (1) a tutor that teaches system dynamics modeling, (2) a meta-tutor that teaches good strategies for learning how to model from the tutor, and (3) an affective learning companion that encourages students to use the learning strategy that the meta-tutor teaches. The affective learning companion’s me...

Modelling is an important skill to acquire, but it is not an easy one for students to learn. Existing instructional technology has had limited success in teaching modelling. We have applied a recently developed technology, meta-tutoring, to address the important problem of teaching model construction. More specifically, we have developed and evalua...

Research on expertise suggests that a critical aspect of expert understanding is knowledge of the relations between domain principles and problem features. We investigated two instructional pathways hypothesized to facilitate students’ learning of these relations when studying worked examples. The first path is through self-explaining how worked ex...

Modeling is becoming increasingly important both as a way to learn science and mathematics, and as a useful cognitive skill. Although many learning activities qualify as “modeling”, this article focuses on activities where (1) students construct a model rather than explore a given model, (2) the model is expressed in a formal language rather than d...

Research in affective computing and educational technology has shown the potential of affective interventions to increase student’s self-concept and motivation while learning. Our project aims to investigate whether the use of affective interventions in a meta-cognitive tutor can help students achieve deeper modeling of dynamic systems by being per...

User modeling in AIED has been extended in the past decades to include affective and motivational aspects of learner's interaction in intelligent tutoring systems. An issue in such systems is researchers' ability to understand and detect students' cognitive and meta-cognitive processes while they learn. In order to study those factors, various dete...

In this paper we present Carmel-Tools, a new set of authoring tools for over- coming the knowledge engineering bottleneck for building tutorial dialogue systems that can accept natural language text input from students. Carmel-Tools provides the facilities for speeding up and simplifying the task of creating domain specic knowl- edge sources for se...

Transfer is typically thought of as requiring individuals to “see” what is the same in the deep structure between a new target problem and a previously encountered source problem, even though the surface features may be dissimilar. We propose that experts can “see” the deep structure by considering the first-order interactions of the explicit surfa...

This chapter begins with an extensive examination of the various ways that adaptive expertise can be measured. Most of them have fairly well-known theoretical explanations, which are reviewed briefly. On the other hand, theoretical explanations are not easily found for one particularly valuable manifestation of adaptive expertise: acceleration of f...

Intelligent Tutoring Systems are software applications capable of complementing and enhancing the learning process by providing direct customized instruction and feedback to students in various disciplines. Although Intelligent Tutoring Systems could differ widely in their attached knowledge bases and user interfaces (including interaction mechanis...

This article is a review of experiments comparing the effectiveness of human tutoring, computer tutoring, and no tutoring. “No tutoring” refers to instruction that teaches the same content without tutoring. The computer tutoring systems were divided by their granularity of the user interface interaction into answer-based, step-based, and substep-ba...

Cognitive science principles should have implications for the design of effective learning environments. The self-explanation principle was chosen for the current work because it has developed significantly over the last 20 years. Early formulations hypothesized that self-explanation facilitated inference generation to supply missing information ab...

For many forms of e-learning environments, the system’s behavior can be viewed as a sequential decision process wherein, at each discrete step, the system is responsible for selecting the next action to take. Pedagogical strategies are policies to decide the next system action when there are multiple ones available. In this project we present a Rei...

Students who exploit properties of an instructional system to make progress while avoiding learning are said to be “gaming” the system. In order to investigate what causes gaming and how it impacts students, we analyzed log data from two Intelligent Tutoring Systems (ITS). The primary analyses focused on six college physics classes using the Andes...

Pedagogical strategies are policies for a tutor to decide the next action when there are multiple actions available. When the content is controlled to be the same across experimental conditions, there has been little evidence that tutorial decisions have an impact on students' learning. In this paper, we applied Reinforcement Learning (RL) to induc...

In this paper, we proposed a new cognitive modeling approach: Instructional Factors Analysis Model (IFM). It belongs to a class of Knowledge-Component- based cognitive models. More specifically, IFM is targeted for modeling student's performance when multiple types of instructional interventions are involved and some of them may not generate a dire...

Meta-tutoring applies the basic policies of interactive tutoring to get students to adopt effective meta-cognitive strategies. Unfortunately, when the meta-tutor is removed, students often revert to using ineffective strategies. This paper is an early report on the progress of the Affective Meta-Tutoring (AMT) project, which will use an affective l...

The level up procedure is a method for evaluating the learning gains of educational software, and tutoring systems in particular, that includes some form of embedded assessment. The instruction is arranged in levels that take only a few minutes to master, and students level up when the software indicates they have achieved mastery. This paper repor...

The Andes physics tutoring system is an experiment in student freedom. It allows students to solve a physics problem in virtually any legal way. This means that Andes must recognize an extremely large number of possible steps occurring in an extraordinarily large number of possible orders. Such freedom raises several research questions. (1) How can...

Certain learners are less sensitive to learning environments and can always learn; while others are more sensitive to variations in learning environments and may fail to learn (Cronbach & Snow 1977). We refer to the former as High learners as opposed to the latter, which we designate as Low learners. One important goal of any learning environment i...

Pedagogical tutorial tactics are policies for a tutor to decide the next action when there are multiple actions available. When the contents were controlled so as to be the same, little evidence has shown that tutorial decisions would impact students’ learning. In this paper, we applied Reinforcement Learning (RL) to induce two sets of tutorial tac...

While high interactivity is one of the key characteristics of one-on-one human tutoring, a great deal of controversy surrounds the issue of whether interactivity is indeed the key feature of tutorial dialogue that impacts students' learning. In this paper we investigate three interaction hypotheses: a widely-believed monotonic interactivity hypothe...

A long standing challenge for intelligent tutoring system (ITS) designers and educators alike is how to encourage students to take pleasure and interest in learning activities. In this paper, we present findings from a user study involving students interacting with an ITS, focusing on when students express excitement, what we dub “yes!” moments. Th...

Effective pedagogical strategies are important for e-learning environments. While it is assumed that an effective learning environment should craft and adapt its actions to the user’s needs, it is often not clear how to do so. In this paper, we used a Natural Language Tutoring System named Cordillera and applied Reinforcement Learning (RL) to induc...

Authoring the domain knowledge of an intelligent tutoring system (ITS) is a well-known problem, and an often-mentioned approach is to use authors who are domain experts. Unfortunately, this approach requires that potential authors learn to write and debug knowledge written in a formal knowledge representation language. If authors were able to use n...

We present results from an analysis of students’ shallow behaviors, i.e., gaming, during their interaction with an Intelligent Tutoring System (ITS). The analysis is based on six college classes using the Andes ITS for homework and test preparation. Our findings show that student features are a better predictor of gaming than problem features, and...

Self-explaining is a domain-independent learning strategy that generally leads to a robust understanding of the domain material. However, there are two potential explanations for its effectiveness. First, self-explanation generates additional content that does not exist in the instructional materials. Second, when compared to comprehension, generat...

Self-explaining is a domain-independent learning strategy that generally leads to a robust understanding of the domain material. However, there are two potential explanations for its effectiveness. First, self-explanation generates additional content that does not exist in the instructional materials. Second, when compared to comprehension, generat...

Self-explaining is a domain-independent learning strategy that generally leads to a robust understanding of the domain material. However, there are two potential explanations for its effectiveness. First, self-explanation generates additional content that does not exist in the instructional materials. Second, when compared to comprehension, generat...

This chapter describes two experiments that test the felicity-conditions hypothesis that people learn best if a task is taught one subprocedure per lesson. In these experiments, children were taught multiplication skills by a human tutor. Although there was a slight trend that presenting one topic per lesson led to fewer errors than presenting two...

Collaboratively observing tutoring is a promising method for observational learning (also referred to as vicarious learning). This method was tested in the Pittsburgh Science of Learning Center’s Physics LearnLab, where students were introduced to physics topics by observing videos while problem solving in Andes, a physics tutoring system. Students...

Self-explaining is a beneficial learning strategy for studying worked-out examples because it either supplies missing information through the generation of inferences or because it provides a mechanism for repairing flawed mental models. Although self-explanation is generated with the purpose of helping the individual, is it also helpful to produce...

Professionals such as medical doctors, aeroplane pilots, lawyers, and technical specialists find that some of their peers have reached high levels of achievement that are difficult to measure objectively. In order to understand to what extent it is possible to learn from these expert performers for the purpose of helping others improve their perfor...

Face-to-face (FTF) human-human,tutoring has ranked among,the most effective forms of instruction. However, because computer-mediated (CM) tutoring is becoming increasingly common, it is instructive to evaluate its effectiveness relative to face-to-face tutoring. Does the lack of spoken, face-to-face interaction affect learning gains and motivation?...

The 14th International Conference on Artificial Intelligence in Education (AIED2009) is being held July 6--10 2009 in Brighton, UK. AIED2009 is part of an ongoing series of biennial international conferences for top quality research in intelligent systems ...

While high interactivity has been one of the main characteristics of one- on-one human tutoring, a great deal of controversy surrounds the issue of whether interactivity is indeed the key feature of tutorial dialogue that impacts students' learning results. There are two commonly held hypotheses regarding the issue: a widely-believed monotonic inte...

Cognitive science principles should have implications for the design of effective learning environments. The self-explanation principle was chosen for the current project because it has developed significantly over the past few years. Early formulations suggested that self-explanation facilitated inference generation to supply missing information a...

VanLehn argued that an essential feature of many intelligent tutoring systems (ITSs) is that they provide feedback and hints on every step of a multi-step solution.But if step-based feedback and hints alone suffice for strong learning gains, as Anderson et al. conjecture ([1]), then perhaps a lightweight tutoring system that employ only feedback an...

Authoring the knowledge base for an intelligent tutoring system (ITS) is difficult and time consuming. In many ITS, the knowledge base is used for solving problems, so authoring it is an instance of the notoriously difficult knowledge acquisition problem of expert systems. General tools for knowledge acquisition have shown only limited success, whi...

Learning outcomes from intelligent tutoring systems (ITSs) tend to be quite strong, usually in the neighborhood of one standard deviation. How- ever, most ITS designers use the learning outcomes from expert human tutoring as the gold standard (i.e., two standard deviations). What can be done, with the current state of the art, to increase learning...

One important goal of Intelligent Tutoring Systems (ITSs) is to bring students up to the same level of mastery. We showed that an ITS teaching a domain-independent problem-solving strategy indeed closed the gap between High and Low learners, not only in the domain where it was taught (probability) but also in a second domain where it was not taught...

Collaboration is an important problem-solving skill; however, novice collaboration generally benefits from some kind of support. One possibility for supporting productive conversations between collaborators is to encourage pairs of students to provide explanations for their problem-solving steps. To test this possibility, we contrasted individuals...

Although cognitive science has discovered several methods for increasing the learning of complex skills, such as physics problem solving, detailed examination of verbal protocols suggests there is still room for improvement. Basically, students do not always apply the meta-cognitive strategies that the instruction invites. For instance, when prompt...

Previous research in cognitive science has shown that analogical comparison and self- explanation are two powerful learning activities that can improve conceptual learning in laboratory settings. The current work examines whether these results generalize to students learning physics in a classroom setting. Students were randomly assigned to one of...

Face-to-face tutoring by an expert human tutor is widely thought to be more effective than intelligent tutoring systems (ITS), which are in turn thought to be more effective than computer-aided instruction (CAI), computer-based training (CBT), etc. The latter tutoring systems have students work out complex solutions on paper, then enter their answe...