ArticlePDF Available

The effectiveness of intelligent tutoring systems on K‐12 students' reading comprehension: A meta‐analysis

Wiley
British Journal of Educational Technology
Authors:
Zhihong Xu at Texas A&M University
Zhihong Xu
Kausalai Kay Wijekumar at Texas A&M University
Kausalai Kay Wijekumar
Gilbert Ramirez at Texas A&M University
Gilbert Ramirez
Xueyan Hu at China Three Gorges University
Xueyan Hu
Show all 5 authorsHide
Download full-text PDF
Read full-text
Download citation

Abstract and Figures

This meta‐analysis examined the effectiveness of improving reading comprehension for students in K‐12 classrooms using intelligent tutoring systems (ITSs), a computer‐based learning environment that provides customizable and immediate feedback to the learner. Nineteen studies from 13 publications incorporating approximately 10 000 students were included in the final analysis; using robust variance estimation to account for statistical dependencies, the 19 studies yielded 88 effect size estimates. The meta‐analysis indicated that the overall random effect size of ITSs on reading comprehension was 0.60 (using a mix of standardized and researcher‐designed measures) with a 95% confidence interval 0.36 to 0.85 (p < 0.001). This review confirms previous studies comparing ITSs to human tutoring: ITSs produced a small effect size when compared to human tutoring (0.20, 0.02–0.38, p = 0.036, n = 21). All comparisons to human tutoring used standardized measures. This review also found that ITSs produced a larger effect size on reading comprehension when compared to traditional instruction (0.86) for mixed measures and (0.26) for standardized measures. These findings may be of interest to practitioners and policy makers seeking to improve reading comprehension using consistent and accessible ITSs. Recommendations for researchers include conducting studies to understand the difference between traditional and updated versions of ITSs and employing valid and reliable standardized tests and researcher‐designed measures.
Figures - uploaded by Zhihong Xu
Author content
All figure content in this area was uploaded by Zhihong Xu
Content may be subject to copyright.
Content uploaded by Zhihong Xu
Author content
All content in this area was uploaded by Zhihong Xu on Aug 15, 2019
Content may be subject to copyright.
A preview of the PDF is not available
A preview of this full-text is provided by Wiley.
Learn more
Content available from British Journal of Educational Technology 
This content is subject to copyright. Terms and conditions apply.
... ITSs capabilities to provide personalized feedback and instruction are often associated with improved learning experiences. Several meta analyses provide empirical evidence that ITSs might enhance learning outcomes across a wide range of subjects and educational levels [Xu et al. 2019, Fang et al. 2019, Steenbergen-Hu and Cooper 2014. Moreover, there is evidence that ITS is an educational technology that stands out compared to other digital tools [Hillmayr et al. 2020]. ...
... The ability to generate exercise lists, assess student solutions, and receive immediate feedback was highlighted as particularly useful. This finding supports the notion that ITS can enhance the teaching process by providing timely feedback and allowing for more dynamic and responsive lesson planning [Xu et al. 2019]. Hence, expanding prior research to the context of ITS unplugged applied to mathematics education. ...
MathAIde in the Classroom: A Qualitative Analysis of Teachers' Perspectives of Intelligent Tutoring Systems Unplugged
Conference Paper
Full-text available
  • Nov 2024
Intelligent Tutoring Systems (ITS) have shown significant promise in enhancing math education by providing personalized and adaptive learning experiences. However, their adoption in resource-constrained environments is limited by the need for dedicated computing devices for each student. To address this issue, the concept of ITS unplugged has emerged, which helps deliver ITS benefits in resource-constrained settings, for instance, without the necessity for individual computers. However, past research has not investigated how ITS unplugged contributes to mathematics education when deployed in real classrooms. This paper presents a qualitative evaluation of MathAIde, an ITS unplugged designed to support numeracy education, where three teachers used MathAIde in 12 lessons, involving 49 students, and their experiences were captured through semi-structured interviews. Thematic analysis revealed that MathAIde facilitated lesson planning and execution, provided valuable feedback and learning analytics, but faced challenges such as technical issues and the need for more adaptive content. This study contributes empirical evidence on the impact of ITS unplugged in real classrooms, offering insights for future development and adoption of such technologies to promote equitable access to ITS benefits in education.
View
... AI tools and applications are increasingly employed to enhance the learning experience, streamline administrative tasks, and provide personalized educational support [1,2] . In the realm of English as a Foreign Language (EFL) instruction, AI-assisted tools, including intelligent tutoring systems, automated feedback mechanisms, and adaptive learning platforms, have not only transformed student engagement but also redefined the role of teachers in the classroom [3][4][5] . ...
Digital Echoes: Crafting Chinese EFL Teacher Identity in the Era of AI-Enhanced Instruction—A Qualitative Exploration
Article
  • Oct 2024
This study examines the impact of artificial intelligence (AI) integration on the professional identities of Chinese tertiary English as a Foreign Language (EFL) teachers. It explores how these educators perceive and adapt to the incorporation of AI tools into their teaching practices, focusing on both the opportunities and challenges encountered. Through qualitative analysis of semi-structured interviews with 16 tertiary EFL teachers, the study identifies key factors influencing their professional identities, including institutional support, professional development, peer influence, personal attitudes towards technology, and ethical considerations. The findings indicate that AI integration enhances personalized learning and reduces the burden of administrative tasks. However, it also challenges teachers to reshape their roles and manage both technological and ethical complexities effectively. The study highlights the necessity for ongoing professional development and robust institutional support to effectively integrate AI in educational settings, ensuring it enhances rather than undermines the teaching and learning process.
View
Decimal Point: A Decade of Learning Science Findings with a Digital Learning Game
Chapter
  • Dec 2024
The McLearn Lab at Carnegie Mellon University (CMU) first designed and developed the artificial intelligence (AI) in education learning game, Decimal Point, in 2013 and 2014 to support middle school children learning decimals and decimal operations. Over a period of 10 years, the McLearn Lab has run a series of classroom experiments with the game, involving over 1,500 elementary and middle school students. In these studies, we have explored a variety of game-based learning and learning science principles and issues, such as whether the game leads to better learning—demonstrated learning gains from a pretest to a posttest and/or a delayed posttest—than a more traditional online instructional approach; whether giving students more agency leads to more learning and enjoyment; whether students benefit from hints and error messages provided during game play; and what types of prompted self-explanation lead to the best learning and enjoyment outcomes. A fascinating finding also emerged during the variety of experiments we conducted: the game consistently led to a gender effect in which girls learned more from the game than boys. In this chapter I will discuss the current state of digital learning games, how we designed and developed Decimal Point, the technology it is built upon—including AI techniques—and the key results of the various experiments we’ve conducted over the years. I conclude by discussing the important game-based learning take-aways from our studies, what we have learned about using a digital learning game as a research platform for exploring learning science principles and issues; and exciting future directions for this line of research.
View
View
View
Parent Teaching Using the Enhanced Moved by Reading to Accelerate Comprehension in English Intelligent Tutoring System to Teach Question-Asking During Shared Book Reading in Latino Families
Article
  • Nov 2024
  • LANG SPEECH HEAR SER
Purpose The Parent–Enhanced Moved by Reading to Accelerate Comprehension in English (Parent EMBRACE) program offers a bilingual parent-training literacy intervention for Latino families. Within the context of shared book reading, the application leverages both the home language and technology to increase parent question-asking during shared reading. Research goals were to (a) examine the potential of the Parent EMBRACE tutoring system at teaching parents to increase the quantity and variety of their question-asking during shared book reading, (b) examine changes to parents' reading attitudes or motivation, and (c) examine whether children's reading attitude is correlated with parent interactions. Method Twenty-one participants were randomized into three conditions: a digital storybook (DS) group ( n = 7), an interactive storybook (EMBRACE) group ( n = 6), and a parent-teaching interactive storybook (Parent EMBRACE) group ( n = 8). Participants received iPads with digital storybooks for use during the intervention (in which the parent-teaching group received prompts from the app to ask questions while reading). Shared book reading assessments before and after the intervention involved hard-copy books, and behaviors were analyzed using video-recorded reading sessions before and after the intervention. Group differences were explored using descriptive analysis. Reading attitude and motivation were measured through pre- and post-intervention surveys. The relationship between parent interactions and reading attitudes was explored through regression. Results Results indicate that after the intervention, four out of seven parents in the parent-teaching interactive storybook group asked more questions to their children. Parents' reading attitudes and motivations did not significantly change. There was a nonlinear relationship with parent interactions and children's reading attitude. Conclusion Overall, the Parent EMBRACE tool shows feasibility and warrants further study on its efficacy as a linguistically responsive literacy-based language intervention for Latino parents to develop shared book reading strategies.
View
ESL 4.0: A Practical Guide to Benefits, Challenges, and Best Practices for AI-Enhanced Learning
Chapter
  • Oct 2024
The interest of implementing cutting-edge technology has always been potent in education. Artificial intelligence (AI) has the potential of being a ubiquitous powerful tool that promises personalized learning, adaptive curriculum, and enhanced engagement. This chapter investigates the growing attention for integrating AI in education, with a specific focus on its application in the English as a second language (ESL) classroom. Therefore, through the lens of established ESL pedagogical principles, the authors explore the potential benefits and inherent challenges of implementing AI tools into ESL classrooms, offering practical strategies and activities that highlight AI's strengths while mitigating its limitations.
View
Enhancing or Hindering? AI's Role in Sparking Creativity in Language Teaching: Insights from Private High School EFL Teachers
Article
Full-text available
  • Sep 2024
The integration of artificial intelligence in education presents both opportunities and challenges, particularly in the field of language teaching. This study aims to investigate Turkish EFL teachers' perceptions of AI's role in fostering or hindering creativity in language teaching. Through semi-structured interviews with 10 EFL teachers from private high schools, the research explores the potential benefits and challenges of AI integration. Key findings reveal that AI enhances student engagement, provides personalized learning experiences, and offers timely feedback. However, concerns about insufficient training, technical issues, and over-reliance on AI potentially undermining fundamental skills were also expressed. While AI tools support improvements in language skills and foster creative thinking, there is apprehension about standardization and the risk of diminishing originality. The study underscores the necessity for effective teacher training and a balanced approach to AI integration, ensuring it complements rather than replaces traditional teaching methods. These insights provide valuable guidance for educators, policymakers, and technology developers to optimize AI use in EFL education while fostering a dynamic and creative learning environment. Future research should explore innovative ways to integrate AI into language teaching without compromising creativity.
View
View
Investigating the Efficacy of ChatGPT-3.5 for Tutoring in Chinese Elementary Education Settings
Article
  • Jan 2024
The potential of artificial intelligence in transforming education has received considerable attention. This study aims to explore the potential of large language models (LLMs) in assisting students with studying and passing standardized exams while many people think it is hype situation. Using primary education as an example, this research investigates whether ChatGPT-3.5 can achieve satisfactory performance on the Chinese Primary School Exams (CPSE) and whether it can be used as a teaching aid, or tutor. We designed an experimental framework and constructed a benchmark that comprises 4800 questions collected from 48 tasks in Chinese Elementary Education Settings. Through automatic and manual evaluations, we observed that ChatGPT-3.5's pass rate was below the required level of accuracy for most tasks, and the correctness of ChatGPT-3.5's answer interpretation was unsatisfactory. These results revealed a discrepancy between the findings and our initial expectations. However, the comparative experiments between ChatGPT-3.5 and ChatGPT-4 indicated significant improvements in model performance, demonstrating the potential of using LLMs as a teaching aid. This paper also investigates the use of the trans-prompting strategy to reduce the impact of language bias and enhance question understanding. We present a comparison of the models' performance and the improvement under the trans-lingual problem decomposition prompting mechanism. Finally, we discuss the challenges associated with the appropriate application of AI-driven language models, along with future directions and limitations in the field of AI for education.
View
Effects of two types of nanoparticles on the cure, rheological, and mechanical properties of rubber nanocomposites based on the NBR/PVC blends
Article
Full-text available
The effects of graphene nanoplatelet (GNP) and organoclay montmorillonite (OMMT) on the curing, relaxation, and the mechanical properties of acrylonitrile‐butadiene rubber/polyvinyl chloride (NBR/PVC) nanocomposites were investigated. Scorch and cure times of nanocomposites reinforced with GNP increased, while these parameters for nanocomposites reinforced with OMMT decreased. Maximum torque and cure rate increased after adding the nanoparticles. In the relaxation curves, with increasing nanoparticle and acrylonitrile (ACN) content, both the slopes of short‐ and long‐term regions, raised. Cured compounds have the significantly higher initial and final modulus, the greater elastic slope, and lower viscous slope than the uncured ones. The results of the relaxation test showed that the relaxation percentage in the uncured samples was much higher than the cured ones. Also, with the addition of both GNP and OMMT, this parameter had an upward trend. DGS/DGL parameter increased with the addition of nano‐reinforcements in the uncured state while decreasing in the cured state. For example, the blend containing N3 reinforced via GNP showed an 11% increase in the uncured state, whereas 14% reduction in the cured state. Tensile strength, elastic modulus, and hardness were improved by incorporating nanoparticles, while the effect of GNP was more intense. The similar trend observed in samples with NBR type of higher ACN content. © 2019 Wiley Periodicals, Inc. J. Appl. Polym. Sci. 2019, 136, 47550.
View
The Effectiveness of Educational Technology Applications on Adult English Language Learners’ Writing Quality: A Meta Analysis
Article
Full-text available
  • Aug 2018
Learning to write well can be a significant contributor to English language proficiency. The large population of English language learners (ELLs) has highlighted the need for educational technology as an important support for the different approaches to writing instruction for ELLs. In this meta-analysis, we examined the effectiveness of educational technology applications on the writing quality of adult ELLs. Twenty-one independent studies from 16 publications were included in this review. Our findings confirmed that technology applications produce a large effect size (.93 and 1.28, fixed effect model and random effect model respectively) as compared to non-technology instructional methods. We conducted sub-group analyses with six substantive and methodological factors: type of technology, genre of writing, program duration, program intensity, measures outcome, and research design, and followed the analyses with research and pedagogical implications.
View
An analysis of the ecological components within a text structure intervention
Article
Full-text available
The Component Model of Reading expanded upon the Simple View of Reading by adding an ecological and psychological component. Elements of the ecological component include teacher knowledge, information provided in textbooks, and teacher instructional practices. In this study, the authors examined the extent of teacher knowledge about text structure, the extent to which textbooks focused on text structure related skills and strategies as well as the percentage each skill and strategy was covered in lessons and teacher instructional practices. Such analysis shows that although text structure interventions may have positive effects on student reading comprehension, there are multiple elements of the ecological component that may be counteracting the benefits of the intervention. First, teachers have a limited knowledge of the five common text structures. Second, textbooks systematically minimize text structure instruction and only cover comprehension skills and strategies sporadically throughout a year-long curriculum. Third, teacher learning of text structures and change in practice was moderated by these ecological factors including textbook scheduled instruction and administrator support.
View
Learning from human tutoring
Article
Full-text available
  • Jan 2001
Human one-to-one tutoring has been shown to be a very effective form of instruction. Three contrasting hypotheses, a tutor-centered one, a student-centered one, and an interactive one could all potentially explain the effectiveness of tutoring. To test these hypotheses, analyses focused not only on the effectiveness of the tutors' moves, but also on the effectiveness of the students' construction on learning, as well as their interaction. The interaction hypothesis is further tested in the second study by manipulating the kind of tutoring tactics tutors were permitted to use. In order to promote a more interactive style of dialogue, rather than a didactic style, tutors were suppressed from giving explanations and feedback. Instead, tutors were encouraged to prompt the students. Surprisingly, students learned just as effectively even when tutors were suppressed from giving explanations and feedback. Their learning in the interactive style of tutoring is attributed to construction from deeper and a greater amount of scaffolding episodes, as well as their greater effort to take control of their own learning by reading more. What they learned from reading was limited, however, by their reading abilities.
View
Web-Based Text Structure Strategy Instruction Improves Seventh Graders’ Content Area Reading Comprehension
Article
Full-text available
  • Mar 2017
Reading comprehension in the content areas is a challenge for many middle grade students. Text structure-based instruction has yielded positive outcomes in reading comprehension at all grade levels in small and large studies. The text structure strategy delivered via the web, called Intelligent Tutoring System for the Text Structure Strategy (ITSS), has proven successful in large-scale studies at 4th and 5th grades and a smaller study at 7th grade. Text structure-based instruction focuses on selection and encoding of strategic memory. This strategic memory proves to be an effective springboard for many comprehension-based activities such as summarizing, inferring, elaborating, and applying. This was the first large-scale randomized controlled efficacy study on the web-based delivery of the text structure strategy to 7th-grade students. 108 classrooms from rural and suburban schools were randomly assigned to ITSS or control and pretests and posttests were administered at the beginning and end of the school year. Multilevel data analyses were conducted on standardized and researcher designed measures of reading comprehension. Results showed that ITSS classrooms outperformed the control classrooms on all measures with the highest effects reported for number of ideas included in the main idea. Results have practical implications for classroom practices.
View
The Effects of Text Structure Instruction on Expository Reading Comprehension: A Meta-Analysis
Article
Full-text available
  • Jul 2016
In this meta-analysis of 45 studies involving students in Grades 2–12, the authors present evidence on the effects of text structure instruction on the expository reading comprehension of students. The meta-analysis was deigned to answer 2 sets of questions. The first set of questions examined the effectiveness of text structure instruction on proximal measures of comprehension, including examination of potential moderators and effectiveness for students with or at-risk for disabilities. The second set of questions examined the effectiveness on transfer measures of the effectiveness of the intervention across temporal contexts (maintenance), near-contexts (untaught text structures), and far-contexts (general reading comprehension). Overall, the results indicated that text structure instruction improves expository reading comprehension, but the effects were tempered when text structure instruction was compared with stronger comparison groups. The findings also identified 2 moderators that led to increased effect sizes (teaching more text structures and including writing in the instruction). Text structure instruction was also found to be effective across all 3 levels of transfer, although the effects for far-transfer are small and lack consistency. Recommendations include conducting more research to understand the nuances of potential interactions between various instructional approaches and student populations.
View
Effectiveness of Intelligent Tutoring Systems: A Meta-Analytic Review
Article
Full-text available
  • Apr 2015
This review describes a meta-analysis of findings from 50 controlled evaluations of intelligent computer tutoring systems. The median effect of intelligent tutoring in the 50 evaluations was to raise test scores 0.66 standard deviations over conventional levels, or from the 50th to the 75th percentile. However, the amount of improvement found in an evaluation depended to a great extent on whether improvement was measured on locally developed or standardized tests, suggesting that alignment of test and instructional objectives is a critical determinant of evaluation results. The review also describes findings from two groups of evaluations that did not meet all of the selection requirements for the meta-analysis: six evaluations with nonconventional control groups and four with flawed implementations of intelligent tutoring systems. Intelligent tutoring effects in these evaluations were small, suggesting that evaluation results are also affected by the nature of control treatments and the adequacy of program implementations.
View
View
View
Computer-Guided Oral Reading versus Independent Practice: Comparison of Sustained Silent Reading to an Automated Reading Tutor That Listens
Article
  • Sep 2013
A 7-month study of 178 students in grades 1-4 at two Blue Ribbon schools compared two daily 20-minute treatments. Eighty-eight students used the 2000-2001 version of Project LISTEN's Reading Tutor (www.cs.cmu.edu/similar to listen) in 10-computer labs, averaging 19 hours over the course of the year. The Reading Tutor served as a computerized implementation of the National Reading Panel's recommended guided oral reading instruction (NRP, 2000). The Reading Tutor listened to students read aloud, giving spoken and graphical help when it noticed them click for help, make a mistake, or get stuck. Students using the Reading Tutor averaged significantly higher gains across measures of reading ability, especially those involving word level skills (word identification, blending words, and spelling) than their matched classmates who spent that time doing Sustained Silent Reading (SSR) in their classrooms. Additionally, these students trended toward higher gains in fluency and reacting comprehension. Overall, use of the Reading Tutor resulted in the types of improvement that would be expected from guided oral reading, but with the benefit of scalability, a problem for human-guided oral reading practice.
View