Steve Ritter’s research while affiliated with Carnegie Learning and other places

What is this page?


This page lists works of an author who doesn't have a ResearchGate profile or hasn't added the works to their profile yet. It is automatically generated from public (personal) data to further our legitimate goal of comprehensive and accurate scientific recordkeeping. If you are this author and want this page removed, please let us know.

Publications (17)


Rewriting Content with GPT-4 to Support Emerging Readers in Adaptive Mathematics Software
  • Article

July 2024

·

13 Reads

·

2 Citations

International Journal of Artificial Intelligence in Education

·

Husni Almoubayyed

·

Logan De Ley

·

[...]

·

Steve Ritter

Large language models (LLMs) offer an opportunity to make large-scale changes to educational content that would otherwise be too costly to implement. The work here highlights how LLMs (in particular GPT-4) can be prompted to revise educational math content ready for large scale deployment in real-world learning environments. We tested the ability of LLMs to improve the readability of math word problems and then looked at how these readability improvements impacted learners, especially those identified as emerging readers. Working with math word problems in the context of an intelligent tutoring system (i.e., MATHia by Carnegie Learning, Inc), we developed an automated process that can rewrite thousands of problems in a fraction of the time required for manual revision. GPT-4 was able to produce revisions with improved scores on common readability metrics. However, when we examined student learning outcomes, the problems revised by GPT-4 showed mixed results. In general, students were more likely to achieve mastery of the concepts when working with problems revised by GPT-4 as compared to the original, non-revised problems, but this benefit was not consistent across all content areas. Further complicating this finding, students had higher error rates on GPT-4 revised problems in some content areas and lower error rates in others. These findings highlight the potential of LLMs for making large-scale improvements to math word problems but also the importance of additional nuanced study to understand how the readability of math word problems affects learning.






Towards the Future of AI-Augmented Human Tutoring in Math Learning
  • Chapter
  • Full-text available

June 2023

·

926 Reads

·

7 Citations

Communications in Computer and Information Science

One of the primary obstacles to improving middle school math achievement is lack of equitable access to high-quality learning opportunities. Human delivery of high-dosage tutoring can bring significant learning gains, but students, particularly economically disadvantaged students, have limited access to well-trained tutors. Augmenting human tutor abilities through the use of artificial intelligence (AI) technology is one way to scale up access to tutors without compromising learning quality. This workshop aims to highlight the challenges and opportunities of AI-in-the-loop math tutoring and encourage discourse in the AIED community to develop human-AI hybrid tutoring and teaching systems. We invite papers that provide clearer understanding and support the progress of human and AI-assisted personalized learning technologies. The structure of this full-day hybrid workshop will include presentations of accepted papers, small or whole group discussion, and a panel discussion focusing on common themes related to research and application, key takeaways, and findings imperative to increasing middle school math learning.KeywordsTutoringPersonalized learningAI-assisted tutoring

Download

Rewriting Math Word Problems to Improve Learning Outcomes for Emerging Readers: A Randomized Field Trial in Carnegie Learning’s MATHia

June 2023

·

147 Reads

·

5 Citations

Communications in Computer and Information Science

We present a randomized field trial delivered in Carnegie Learning’s MATHia’s intelligent tutoring system to 12,374 learners intended to test whether rewriting content in “word problems” improves student mathematics performance within this content, especially among students who are emerging as English language readers. In addition to describing facets of word problems targeted for rewriting and the design of the experiment, we present an artificial intelligence-driven approach to evaluating the effectiveness of the rewrite intervention for emerging readers. Data about students’ reading ability is generally neither collected nor available to MATHia’s developers. Instead, we rely on a recently developed neural network predictive model that infers whether students will likely be in this target sub-population. We present the results of the intervention on a variety of performance metrics in MATHia and compare performance of the intervention group to the entire user base of MATHia, as well as by comparing likely emerging readers to those who are not inferred to be emerging readers. We conclude with areas for future work using more comprehensive models of learners.Keywordsmachine learningA/B testingintelligent tutoring systemsreading abilitymiddle school mathematics




Figure 1. Dialogue mode profiles of top versus bottom 25% sessions, respectively. 
An Analysis of Human Tutors’ Actions in Tutorial Dialogues

May 2017

·

345 Reads

·

5 Citations

Understanding effective human tutors’ strategies is one approach to discovering effective tutorial strategies. These strategies are described in terms of actions that tutors take while interacting with learners. To this end, we analyze in this paper dialoguebased interactions between professional tutors and tutees. There are two challenges when exploring patterns in such dialogue based tutorial interactions. First, we need to map utterances, by the tutor and by the tutee, into actions. To address this challenge, we rely on the language-as-action theory according to which when we say something we do something. A second challenge is detecting effective tutorial sessions using objective measurements of learning. To tackle this challenge we align tutorial conversations with pre- and post- measures of student mastery obtained from an intelligent tutoring system with which the students interacted before and after interacting with the human tutor. We present performance results of the automated tools that we developed to map tutor-tutee utterances onto dialogue acts and dialogue modes. We also report the most interesting emerging patterns in terms of tutor and tutees’ actions. These patterns could inform our understanding of the tutoring process and the development of intelligent tutoring systems.


Citations (11)


... The third strand focuses on enhancing adaptive digital learning environments with LLMs. Norberg et al. (2024) utilize GPT-4 to revise educational math content with the aim of enhancing readability for deployment in an adaptive learning system. The researchers found that GPT-4 could improve readability metrics, such as word frequency, sentence complexity, and semantic similarity. ...

Reference:

The Use of Large Language Models in Education
Rewriting Content with GPT-4 to Support Emerging Readers in Adaptive Mathematics Software
  • Citing Article
  • July 2024

International Journal of Artificial Intelligence in Education

... Studies on Bayesian Knowledge Tracing and carelessness detectors have shown promising results, with performance being relatively equal across demographic groups (Zambrano et al., 2024). However, traditional bias metrics may not be suitable for educational settings due to hierarchical dependencies in classrooms, necessitating adapted measurements using hierarchical linear models (Belitz et al., 2024). To address these challenges, researchers recommend focusing on solidifying understanding of concrete impacts, moving from unknown to known bias, and transitioning from fairness to equity (Baker & Hawn, 2021). ...

Hierarchical Dependencies in Classroom Settings Influence Algorithmic Bias Metrics
  • Citing Conference Paper
  • March 2024

... The expression "Augmented Humans" [4] refers to the adoption of methods and technology that enhance physical, cognitive, or sensory skills beyond what is common for humans. This paradigm shift is transforming various aspects of daily life and industries, such as education [5] and healthcare [6], by providing immersive learning environments, virtual training simulations, and new forms of entertainment. ...

Towards the Future of AI-Augmented Human Tutoring in Math Learning

Communications in Computer and Information Science

... For example, Carnegie Learning's MATHia (https://www.carnegielearning.com/) uses AI to provide real-time feedback and personalized instruction in mathematics, identifying areas where students struggle and offering targeted interventions to enhance understanding and retention (Almoubayyed et al., 2023). ...

Rewriting Math Word Problems to Improve Learning Outcomes for Emerging Readers: A Randomized Field Trial in Carnegie Learning’s MATHia
  • Citing Chapter
  • June 2023

Communications in Computer and Information Science

... A case in point is that of Carnegie Learning in the case of mathematics education. For instance, the personalization of mathematics instruction for every student through artificial intelligence, such as while designing the MATHia system, would be realized because it provides real-time feedback and adjusts the difficulty of the problems based on performance [48], [49]. For example, AutoTutor was designed for conversational learning as a tutor for a human conversational partner using natural language processing. ...

Instruction-Embedded Assessment for Reading Ability in Adaptive Mathematics Software
  • Citing Conference Paper
  • March 2023

... The same group of researchers explored the role of Hidden Markov Models (HMMs), a generative model, and Conditional Random Fields (CRFs), a discriminative model, in classifying speech acts in one to one human tutorial sessions [13]. They demonstrated that the CRF model with features constructed from the first three tokens and last token of previous, next and current utterances, length of current utterance, and other surface features such as bigrams and the speech acts of context utterances performed better than HMM models. ...

An Analysis of Human Tutors’ Actions in Tutorial Dialogues

... Digital learning software for secondary education has been developed for decades [2,3,9,18] and has seen widespread adoption, with millions of students using it globally across different platforms (e.g., [19,22,24,27,32]). While a substantial body of research has focused on how such software should be constructed to most effectively support students' learning (e.g., [1,15,21,25,31]), surprisingly little is known about its long-term usage once integrated into classrooms (but see [4,14]). ...

How Mastery Learning Works at Scale
  • Citing Conference Paper
  • April 2016

... Learning engineering is an emerging field that uses evidence to inform educational design. For example, researchers developed and disseminated different versions of a game's script [12] and various conditions of difficulty and support [13] to large audiences to determine which versions produced desirable outcomes and inform design theory. ...

The Rise of the Super Experiment

... On the other hand, such experiments facilitate the logging of detailed ecological data and patterns that do emerge are likely to be more robust to the variability of real classrooms. Student interface actions are logged as selection-action-input triples, representing the element of the interface with which students interact, the action students took, and the input to the action (Ritter & Koedinger, 1997). For example, entering 25 in a table would be represented as (selection = cell A1, action = enterValue, input = "25"). ...

An architecture for plug-in tutoring agents
  • Citing Article

... These architectures generally provide a programming language that allows users to model human cognitive processes and behaviors. However, these languages almost always are specified at a primitive level, similar to assembly code (Cohen, et al., 2005, Ritter andKoedinger, 1995). This makes it very time consuming to develop the models and usually the models are over-specified. ...

Towards lightweight tutoring agents
  • Citing Conference Paper
  • January 1995