-
Psychometrika 04/2012; 76(3):504-506. · 1.77 Impact Factor
-
[show abstract]
[hide abstract]
ABSTRACT: Over the past thirty years, obtaining diagnostic information from examinees’ item responses has become an increasingly important
feature of educational and psychological testing. The objective can be achieved by sequentially selecting multidimensional
items to fit the class of latent traits being assessed, and therefore Multidimensional Computerized Adaptive Testing (MCAT)
is one reasonable approach to such task. This study conducts a rigorous investigation on the relationships among four promising
item selection methods: D-optimality, KL information index, continuous entropy, and mutual information. Some theoretical connections among the methods are demonstrated to show how information about the unknown vector θ can be gained from different perspectives. Two simulation studies were carried out to compare the performance of the four
methods. The simulation results showed that mutual information not only improved the overall estimation accuracy but also yielded the smallest conditional mean squared error in most region
ofθ. In the end, the overlap rates were calculated to empirically show the similarity and difference among the four methods.
KeywordsKullback–Leibler information–Fisher information–mutual information–multidimensional computerized adaptive test–continuous entropy
Psychometrika 04/2012; 76(3):363-384. · 1.77 Impact Factor
-
[show abstract]
[hide abstract]
ABSTRACT: The item response times (RTs) collected from computerized testing represent an underutilized source of information about items and examinees. In addition to knowing the examinees' responses to each item, we can investigate the amount of time examinees spend on each item. In this paper, we propose a semi-parametric model for RTs, the linear transformation model with a latent speed covariate, which combines the flexibility of non-parametric modelling and the brevity as well as interpretability of parametric modelling. In this new model, the RTs, after some non-parametric monotone transformation, become a linear model with latent speed as covariate plus an error term. The distribution of the error term implicitly defines the relationship between the RT and examinees' latent speeds; whereas the non-parametric transformation is able to describe various shapes of RT distributions. The linear transformation model represents a rich family of models that includes the Cox proportional hazards model, the Box-Cox normal model, and many other models as special cases. This new model is embedded in a hierarchical framework so that both RTs and responses are modelled simultaneously. A two-stage estimation method is proposed. In the first stage, the Markov chain Monte Carlo method is employed to estimate the parametric part of the model. In the second stage, an estimating equation method with a recursive algorithm is adopted to estimate the non-parametric transformation. Applicability of the new model is demonstrated with a simulation study and a real data application. Finally, methods to evaluate the model fit are suggested.
British Journal of Mathematical and Statistical Psychology 04/2012; · 1.31 Impact Factor
-
[show abstract]
[hide abstract]
ABSTRACT: Computerized adaptive testing (CAT) was originally proposed to measure θ, usually a latent trait, with greater precision by sequentially selecting items according to the student's responses to previously administered items. Although the application of CAT is promising for many educational testing programs, most of the current CAT systems were not designed to provide diagnostic information. This article discusses item selection strategies specifically tailored for cognitive diagnostic tests. Our goal is to identify an effective item selection algorithm that not only estimates θ efficiently, but also classifies the student's knowledge status α accurately. A single-stage item selection method with a dual purpose will be introduced. The main idea is to treat diagnostic criteria as constraints: Using the maximum priority index method to meet these constraints, the CAT system is able to generate cognitive diagnostic feedback in a fairly straightforward fashion. Different priority functions are proposed. Some of them are based on certain information measures, such as Kullback-Leibler information, and others utilize only the information provided by the Q-matrix. An extensive simulation study is conducted, and the results indicate that the information-based method not only yields higher classification rates for cognitive diagnosis, but also achieves more accurate θ estimation. Other constraint controls, such as item exposure rates, are also considered for all the competing methods.
Behavior Research Methods 08/2011; 44(1):95-109. · 2.12 Impact Factor
