Conference Paper

Analyzing Fine-Grained Skill Models Using Bayesian and Mixed Effects Methods.

Conference: Artificial Intelligence in Education, Building Technology Rich Learning Contexts That Work, Proceedings of the 13th International Conference on Artificial Intelligence in Education, AIED 2007, July 9-13, 2007, Los Angeles, California, USA
Source: DBLP

ABSTRACT Two modelling methods were employed to answer the same research question of how accurate the various grained WPI 1, 5, 39 and 106 skill models are at assessing student knowledge in the ASSISTment online tutoring system and predicting their performance on the 2005 state MCAS test. One method, used by the second author, is mixed effect statistical modelling. The first author evaluated the problem with a Bayesian networks machine learning approach. We compare the two results to identify benefits and drawbacks of either method and to find out if the two results agree. We report that both methods showed compelling similarity in results especially with regard to residuals on the test. Our analysis of these residuals and our online skills allows us to better understand our model and conclude with recommendations for improving the tutoring system, as well as implications for state testing programs.

  • Source
  • Source
    [Show abstract] [Hide abstract]
    ABSTRACT: The ASSISTment system is an online benchmark testing system that tutors as it tests. The system has been implemented for the content of the 8th grade Mathematics portion of the Mas- sachusetts Comprehensive Assessment System (MCAS) exams, has been developed and tested in Massachusetts middle schools, and is being adapted for use in other states such as Pennsylvania. Two main statistical goals for the ASSISTment system are to predict end-of-year MCAS scores, and to provide regular, periodic feedback to teachers on how students are doing, what to teach next, etc. In this chapter we focus on the first goal and consid er 10 prediction models: how they reflect di fferent models for student proficiency, how they account for st udent learning over time, and how well they predict MCAS scores. We conclude that a combination of measures, including response accuracy (right/wrong) measures that account for problem diffi culty, response effi ciency, and help-seeking behavior, produce the best prediction models. In addition, our investigations of prediction models reveal patterns of learning over time that should be captured in feedback reports for teachers.
  • [Show abstract] [Hide abstract]
    ABSTRACT: A standing question in the field of Intelligent Tutoring Systems and User Modeling is what is the appropriate level of model granularity (how many skills to model) and how is that granularity derived? In this paper we will explore models with varying levels of skill generality (1, 5, 39 and 106 skill models) and measure the accuracy of these models by predicting student performance within our tutoring system called the ASSISTment System as well as their performance on a high-stakes statewide standardized test. We employ the use of Bayesian networks to model user knowledge and prediction of student responses. Our results show that the finer the granularity of the skill model, the better we can predict student performance within the tutor. However, for the standardized test data we received, it was the 39 skill model that performed the best. We view this as support for fine-grained skill models despite the finest grain model not predicting the standardized test scores most effectively.

Full-text (2 Sources)

Available from
May 31, 2014