Steve Ritter’s research while affiliated with Carnegie Learning and other places

Understanding effective human tutors’ strategies is one approach to discovering effective tutorial strategies. These strategies are described in terms of actions that tutors take while interacting with learners. To this end, we analyze in this paper dialoguebased interactions between professional tutors and tutees. There are two challenges when exploring patterns in such dialogue based tutorial interactions. First, we need to map utterances, by the tutor and by the tutee, into actions. To address this challenge, we rely on the language-as-action theory according to which when we say something we do something. A second challenge is detecting effective tutorial sessions using objective measurements of learning. To tackle this challenge we align tutorial conversations with pre- and post- measures of student mastery obtained from an intelligent tutoring system with which the students interacted before and after interacting with the human tutor. We present performance results of the automated tools that we developed to map tutor-tutee utterances onto dialogue acts and dialogue modes. We also report the most interesting emerging patterns in terms of tutor and tutees’ actions. These patterns could inform our understanding of the tutoring process and the development of intelligent tutoring systems.

Nearly every adaptive learning system aims to present students with materials personalized to their level of understanding (Enyedy, 2014). Typically, such adaptation follows some form of mastery learning (Bloom, 1968), in which students are asked to master one topic before proceeding to the next topic. Mastery learning programs have a long history of success (Guskey and Gates, 1986; Kulik, Kulik & Bangert-Drowns, 1990) and have been shown to be superior to alternative instructional approaches. Although there is evidence for the effectiveness of mastery learning when it is well supported by teachers, mastery learning's effectiveness is crucially dependent on the ability and willingness of teachers to implement it properly. In particular, school environments impose time constraints and set goals for curriculum coverage that may encourage teachers to deviate from mastery-based instruction. In this paper we examine mastery learning as implemented in Carnegie Learning's Cognitive Tutor. Like in all real-world systems, teachers and students have the ability to violate mastery learning guidance. We investigate patterns associated with violating and following mastery learning over the course of the full school year at the class and student level. We find that violations of mastery learning are associated with poorer student performance, especially among struggling students, and that this result is likely attributable to such violations of mastery learning.

Time pressure helps students practice efficient strategies. We report strong effects from using games to promote fluency in mathematics.

Traditional experimental paradigms have focused on executing experiments in a lab setting and eventually moving successful findings to larger experiments in the field. However, data from field experiments can also be used to inform new lab experiments. Now, with the advent of large student populations using internet-based learning software, online experiments can serve as a third setting for experimental data collection. In this paper, we introduce the Super Experiment Framework (SEF), which describes how internet-scale experiments can inform and be informed by classroom and lab experiments. We apply the framework to a research project implementing learning games for mathematics that is collecting hundreds of thousands of data trials weekly. We show that the framework allows findings from the lab-scale, classroom-scale and internet-scale experiments to inform each other in a rapid complementary feedback loop.

Intelligent tutoring systems work falls into three waves. The first wave involves basic research on technical implementation, including authoring systems and tutoring architectures. Second wave work takes this technological development beyond the laboratory. This work involves deep analysis of domain knowledge and empirical validation of systems. The emerging “third wave” takes advantage of widespread use of systems to refine and improve their effectiveness. Work in this area includes data mining and end-user authoring. Although many types of systems have followed this evolution, intelligent tutoring systems are uniquely positioned among educational software to take advantage of the third wave. The architecture and authoring work from the first wave and the ability to incorporate domain knowledge and test pedagogical approaches in the second wave make us well positioned to ride this third wave. In this talk, I will describe Carnegie Learning’s experience in riding these waves. We have taken intelligent tutoring systems for mathematics originally developed at Carnegie Mellon to scale with over 500,000 users per year, and are now riding the third wave to leverage this user base and improve the effectiveness and utility of our systems.

We describe our efforts to build new learning environments that incorporate tutoring elements into pre-existing software packages. Two systems are described, one which provides tutoring support in the Geometer's Sketchpad and the other which supports students using Microsoft Excel. Although the implementation of these two systems was somewhat different, they share many basic components. An analysis of their similarities and differences allows us to move towards a set of standards for tutor agents that interact with complex tools. By constructing learning environments in this manner, we can leverage the power of existing workplace software and educational microworlds to create more powerful learning environments.