On measurement properties of continuation ratio models

Full text

Page 1

PSYCHOMETRIKA--VOL. 66, NO. 4, 487-506
DECEMBER 2001
ON MEASUREMENT PROPERTIES OF CONTINUATION RATIO MODELS
BAS T. HEMKER
CITO NATIONAL INSTITUTE FOR EDUCATIONAL MEASUREMENT
L. ANDRIES VAN DER ARK AND KLAAS SIJTSMA
TILBURG UNIVERSITY
Three classes of polytomous IRT models are distinguished. These classes are the adjacent category
models, the cumulative probability models, and the continuation ratio models. So far, the latter class has
received relatively little attention. The class of continuation ratio models includes logistic models, such
as the sequential model (Tutz, 1990), and nonlogistic models, such as the acceleration model (Samejima,
1995) and the nonparametric sequential model (Hemker, 1996). Four measurement properties are dis-
cussed. These axe monotone likelihood ratio of the total score, stochastic ordering of the latent trait by
the total score, stochastic ordering of the total score by the latent trait, and invaxiant item ordering. These
properties have been investigated previously for the adjacent category models and the cumulative proba-
bility models, and for the continuation ratio models this is done here. It is shown that stochastic ordering
of the total score by the latent trait is implied by all continuation ratio models, while monotone likelihood
ratio of the total score and stochastic ordering on the latent trait by the total score are not implied by any
of the continuation ratio models. Only the sequential rating scale model implies the property of invariant
item ordering. Also, we present a Venn-diagram showing the relationships between all known polytomous
IRT models from all three classes.
Key words: acceleration model, adjacent category models, continuation ratio models, cumulative probabil-
ity models, hierarchical relationships between IRT models, invariant item ordering, monotone likelihood
ratio, polytomous IRT models, sequential model, stochastic ordering.
General Introduction
In the social and behavioral sciences data collected by means of items in tests and ques-
tionnaires are often ordered scores, where a higher score indicates a higher position on a latent
trait such as arithmetic ability, introversion, or attitude towards capital punishment. Examples of
items with ordered scores are used in the "NT2-profiel toets" (CITO, 1999), an ability test for
Dutch as a foreign language. We discuss such an item and its sequential scoring rule here because
it appears to be well suited for the class of continuation ratio IRT models (CRMs; Agresti, 1990,
pp. 319-321; Mellenbergh, 1995; Molenaar, 1983) that is central to this paper. Each item of the
"NT2-profiel toets" consists of a spoken Dutch text that ends with a question about this text, for
example (see also Hemker, 2001),
[translated from Dutch] Suppose, you work at an office. You have to fax a letter for
your boss. You have no experience with the fax machine. You know a colleague who
is able to use the fax machine. What do you ask your colleague? (CITO, 1998, p. 5)
An examinee has to give a verbal response (in Dutch). Examinees are tested individually by an
examiner, who scores each item. The item is scored as follows. In the first step, the content of
the answer is assessed. If the response is incorrect with respect to content (e.g., "Can I use this
fax machine?"), the first step is failed and the result is an item score of 0. Only if the examinee's
response is correct or almost correct (i.e., a request for help or for an explanation of the operation
Requests for reprints should be sent to Bas T. Hemker, Measurement and Research Department, CITO Na-
tional Institute for Educational Measurement, EO. Box 1034, 6801 MG Arnhem, THE NETHERLANDS. E-Mail:
bas.hemker @ citogroep
0033-3123/2001-4/1999-0779-A $00.75/0
@ 2001 The Psychometric Society
487

Page 2

488 PSYCttOMETRIKA
of the fax machine) the first step is passed and the examiner proceeds with the second step. In the
second step the examinee's use of grmmnar is assessed. If the examinee makes more than just a
few insignificant grammatical errors, the second step is failed and the result is an item score of 1.
Only if the examinee's response contains no more than a few unimportant grammatical errors
the second step is passed and the examiner proceeds with the third step. In the third step the
pronunciation of the response is assessed. If the examiner thinks that the average Dutchman will
not be able to understand the response easily, the third step is failed and the result is an item score
of 2. If the examiner thinks that the averse Dutchman can understand the response without too
much difficulty, the third is passed and the result is an item score of 3.
Classes of Polytomous Item Response Models
Continuation Ratio Models
The class of CRMs to be discussed here may be suited particularly for modeling data ob-
tained through a sequential scoring rule as illustrated by the example. CRMs usually have logistic
response functions. Hemker (1996, chap. 6) extended tile class of CRMs to also include nonpara-
metric response functions of which logistic functions are special cases. Before discussing the
general form of CRMs, we first introduce some notation. Let the latent trait be denoted by 0, the
random variable for the score on item j by X j, and realizations by x = 0 ..... m. Furthermore,
all models discussed here assume a unidimensional 0 and locally independent item scores. First,
we define the conditional probability of passing an item step as
P(Xj >_ xlO)
(1)
Mjx(O) = P(Xj >_ xIX j ~ X -- 1; 0) = P(Xj > x - ll0)"
Equation (1) implies that if x = 0 then Mix(O) = 1 for all 0. Equation (1) is the item
step response function (ISRF). The conditional probability of obtaining an item score of x,
P (Xj = x I 0), is decomposed into a product of x terms, Mix (0), and one term, 1 - Mj,x+i(O),
as
x
P(Xj = xlO) = 1-I Mjr(O) [1 - Mj,x+i(O)]
y=0
(2)
(Samejima, 1972, chap. 4). Equation (2) is the category characteristic curve (CCC). Thus, CRMs
formalize sequential scoring by writing the CCC as a product of x ISRFs for the x subtasks
that were successfully solved and the conditional probability of failing subtask x 4- 1 given that
the previous subtasks were mastered. Thus, it is assumed that the steps are executed in a fixed
sequence. Tutz (1990) discussed two parametric CRMs and characterized both as sequential
models,
Adjacent Category Models
If the order in which the steps are presented to the respondent is not fixed, then two other
classes of models for ordered item scores might be used. These two classes use alternative defi-
nitions of the ISRF (e.g., Mellenbergh, 1995; Molenaar, 1983). One class of models is known as
adjacent categox7 models (ACMs). The ISRF of models from this class is defined as
P(Xj = xlO)
(3)
Fix(O) = P(Xj = xlO) 4. P(Xj = x- 110)"
It may be noted that the ISRF of ACMs (3) and the ISRF of CRMs (1) are related by
Fix(O) = Mjx(O) - Mjx(O)Mj,x+l(O)
1 - Mjx(O)Mj,x+i(O)

Page 3

BAS T. HEMKER, L. ANDRIES VAN DER ARK, AND KLAAS SIJTSMA
489
Thissen and Steinberg (1986) called parametric models from the class of ACMs divide-by-total
models and Andrich (1995) called these models Rasch models. Some well-known divide-by-total
models are the rating scale model (Andersen, 1977; Andrich, 1978) and the generalized partial
credit model (Muraki, 1992). The best known of these parametric ACMs is the partial credit
model (Masters, 1982), defined by
exp(0 - 3jx)
(4)
Fjx(O) = 1 + exp(0 - 3jx)'
where 3ix is a location parameter. Hemker, Sijtsma, Molenaar and Junker (1996) introduced a
more general class including a nonparametric model. They called this model the nonparametric
partial credit model, defined by Fix (0) (Equation (3)) nondecreasing in 0.
Cumulative Probability Models
The third class of models is known as cumulative probability models (CPMs). The ISRF of
models from this class is defined as
Gjx(O) = P(Xj > xlO).
(5)
It may be noted that the ISRF of CPMs (4) and the ISRF of CRMs (1) are related by
Gjx(O) = 11 Mjy(O).
y=l
(6)
Thissen and Steinberg (1986) called parametric CPMs difference models, because the CCC is
obtained by the difference of two adjacent ISRFs. Andrich (1995) called these models Thurstone
models. A well-known CPM is the homogeneous case of the graded response model (Samejima,
1969; also, see Samejima, 1997), defined as
Gjx (0) =
exp[cej (0 -- ,~jx)]
1 + exp[cej (0 - "~jx)]'
where c~j denotes the slope parameter and )~jx a location parameter, different from 3jx in (4); see
Masters (1988) for a discussion of the interpretations of 3ix and )~jx. When it is assumed that the
ISRF in (5) is nondecreasing in 0, without defining the ISRF parametrically, the nonparametric
graded response model is obtained (Hemker et al., 1996).
Table 1 summarizes the terminology used to identify the three classes of polytomous IRT
models. Van Engelenburg (1997, chap. 2) argued that with each of the three classes of polytomous
IRT models corresponds a particular type of task, and Akkermans (1998, chap. 3) argued that
with each class corresponds a particular type of scoring rule.
Motivation of This Study
Thissen and Steinberg (1986) discussed a taxonomy for divide-by-total models and diffEr-
ence models. This taxonomy also included models with guessing parameters that are not relevant
for this study and, consequently, are left out of consideration. The taxonomy only pertained to
parametric models. Hemker, Sijtsma, Molenaar, and Junker (1997) discussed a taxonomy that ba-
sically extended the taxonomy of Thissen and Steinberg to include nonparametric models. More-
over, the formal relationships between all models were described by means of a Venn-diagram,
based on stochastic ordering (SO) relations between the latent trait 0 and the unweighted sum of
J item scores, denoted X+. Sijtsma and Hemker (1998) discussed the same classes of models
with respect to the item ordering property known as invariant item ordering (Sijtsma & Junker,
1996).

Page 4

490
PSYCHOMETRIKA
TABLE 1.
An overview of the terminology used to identify classes of IRT models
Definition ISRF
Au~or
P(Xj = xlO )
P(Xj = x V x -- ll0)
P(Xj >_ xlO )
P(Xj >_ xIO )
P(Xj >_ x -- 110)
Molenaar (1983);
parmnetric and
nonparaanetric models
Thissen and Steinberg (1986);
Parametric models only
Andrich (1995);
Parametric models only
Tutz (1990);
Parametric models only
Adjacent Category
Models (ACMs)
Divide-by-Total
Models
Rasch Models
Cumulative
Probability
Models (CPMs)
Difference Models
Thurstone Models
Continuation Ratio
Models (CRMs)
Sequential Models
A missing link in this research is the class of CRMs. Both classes of ACMs and CPMs have
been investigated thoroughly (Andersen, 1977, 1997; Andrich, 1978, 1995; Glas, 1989; Kelder-
man & Rijkes, 1994; Masters, 1982; Masters & Wright, 1997; Muraki, 1990, 1992; Samejima,
1969, 1972; Verhelst, Glas, & Verstralen, 1995) and models from these classes have been applied
to many practical data analysis problems (some recent applications include Alexander & Mur-
phy, 1998; Cooke, Michie, Hart, & IIare, 1999; Gumpel, Wilson, & Shalev, 1998; Maurer, Raju,
& Collins, 1998; and Sijtsma & Verweij, 1999). Although potentially useful, the class of CRMs
thus far has received relatively little attention (the exceptions are Samejima, 1995; Tutz, 1990,
1997; and Verhelst, Glas, & De Vries, 1997). CRMs are attracting more attention nowadays,
given the recent studies by Hemker (1996), Van Engelenburg (1997) and Akkermans (1998),
who compared CRMs with other polytomous IRT models. Thus, it seems reasonable to better
incorporate the class of CRMs into the polytomous IRT framework. A contribution to this is
given in this paper, where we investigate likelihood ratio and SO properties between the latent
trait 0 and the sum score X+, and also the invariant item ordering property. Insight into these
relationships contributes to a better understanding of the relationships of CRMs to ACMs and
CPMs and, moreover, gives indications of the practical usefulness of CRMs.
Introduction to Continuation Ratio Models
We discuss the most general model from the class of CRMs, and then we discuss several
special cases. The most general model is the nonparametric sequential model (Hemker, 1996,
chap. 6), which assumes an order-restricted ISRF, without parametrically defining it. The non-
parametric sequential model assumes a unidimensional 0, locally independent item scores, and a
nondecreasing ISRF, given by (1). Several special cases have been proposed.
Samejima (1995) assumed a semi-parametric ISRF, Mjx (0) = [~jx (0)] ~j , where ~j _> 0 is
the acceleration parameter. The function qQx(O) = P(Xj
and is assumed to be strictly increasing with 0 and 1 as its horizontal asymptotes. The acceleration
model is the parametric version of Mjx (0) = [~jx (0)] ~j. Let C~jx denote the discrimination
parameter and fijx the location parameter associated with category x of item j, and let D be
a scaling constant, usually equal to 1.7 to scale the logistic function to the normal-ogive. The
acceleration model assmnes that
> xIXj > x - 1; 0) is nonparametric
exp[Dc~jx(O -- fijx)] I ~j

Page 5

BAS T. HEMKER, L. ANDRIES VAN DER ARK, AND KLAAS SIJTSMA
491
It may be noted that the acceleration model is not a logistic model for ~j ~ 1. The accelera-
tion parameter contributes to the steepness of the complete ISRFs, whereas C~jx influences the
steepness of a logistic curve in its inflection point: ~j > 1 "pushes down" the entire curve and
~j < 1 "lifts up" the entire curve, where both effects add to the effect of C~jx on the slope of an
ISRF in the inflection point. Figure l(a) gives a graphic example of the acceleration model, and
shows two items each having two ISRFs (solid and dashed curves for different items; parameter
l(a) l(b)
¢o
,¢
c~
.;o -;
F fr °
/f/
/I/
/ I/
/ /
...¢J
0 ;
,'0
Theta
QA
/ ///
L//
o~
;I y
-;o
-; 0 ;
;o
Theta
:(c)
l(d)
¢o
O-
lD
el
~mm
-,0 -; 0 ;
,'0
Theta
.;o
-;
; ;
;o
Theta
l(e) l(f)
o Q 1 ,,,,,
~
////
II//
II// i ~
~ //,I/
-;o -; 'o ; ;o -~o -;
o ;
~'o
Theta Theta
FIGURE 1.
The ISRFs of six parametric CRMs for two items (solid and dashed curve) with three answer categories. Figure 1 (a) is
the Acceleration Model; Figure l(b) is the 2p(jx)-Sequential Model; Figure l(c) is the 2p(j)-Sequential Model; Figure
1 (d) is the 2p(x)-Sequential Model; Figure 1 (e) is the Sequential Rasch Model; Figure 1 (f) is the Sequential Rating Scale
Model.