Adaptive success control in computerized adaptive testing

Abstract

In computerized adaptive testing (CAT) procedures within the framework of probabilistic test theory the difficulty of an item is adjusted to the ability of the respondent, with the aim of maximizing the amount of information generated per item, thereby also increasing test economy and test reasonableness.However, earlier research indicates that respondents might feel over-challenged by a constant success probability of p=0.5 and therefore cannot come to a sufficiently high answer certainty within a reasonable timeframe. Consequently response time per item increases, which – depending on the test material – can outweigh the benefit of administering optimally informative items. Instead of a benefit, the result of using CAT procedures could be a loss of test economy.Based on this problem, an adaptive success control algorithm was designed and tested, adapting the success probability to the working style of the respondent. Persons who need higher answer certainty in order to come to a decision are detected and receive a higher success probability, in order to minimize the test duration (not the number of items as in classical CAT). The method is validated on the re-analysis of data from the Adaptive Matrices Test (AMT, Hornke, Etzel & Rettig, 1999) and by the comparison between an AMT version using classical CAT and an experimental version using Adaptive Success Control.The results are discussed in the light of psychometric and psychological aspects of test quality.

Full text

Page 1

Psychology Science, Volume 48, 2006 (4), p. 436-450



Adaptive success control in computerized adaptive testing
JOACHIM HÄUSLER1
Abstract
In computerized adaptive testing (CAT) procedures within the framework of probabilistic test the-
ory the difficulty of an item is adjusted to the ability of the respondent, with the aim of maximizing the
amount of information generated per item, thereby also increasing test economy and test reasonable-
ness.
However, earlier research indicates that respondents might feel over-challenged by a constant suc-
cess probability of p=0.5 and therefore cannot come to a sufficiently high answer certainty within a
reasonable timeframe. Consequently response time per item increases, which – depending on the test
material – can outweigh the benefit of administering optimally informative items. Instead of a benefit,
the result of using CAT procedures could be a loss of test economy.
Based on this problem, an adaptive success control algorithm was designed and tested, adapting the
success probability to the working style of the respondent. Persons who need higher answer certainty in
order to come to a decision are detected and receive a higher success probability, in order to minimize
the test duration (not the number of items as in classical CAT).
The method is validated on the re-analysis of data from the Adaptive Matrices Test (AMT, Hornke,
Etzel & Rettig, 1999) and by the comparison between an AMT version using classical CAT and an
experimental version using Adaptive Success Control.
The results are discussed in the light of psychometric and psychological aspects of test quality.

Key words: CAT, adaptive testing, test economy, ASC, success control

1 Mag. Joachim Häusler, Dr.G.Schuhfried GmbH, Hyrtlstraße 45, 2340 Mödling, Austria; e-mail: haeusler@
schuhfried.at

Page 2

Adaptive success control in computerized adaptive testing
437
Theoretical framework

One of the main arguments for adaptive test procedures (CAT, Computerized Adaptive
Testing) – which were originally known as Tailored Testing (Lord, 1968) – is the high level
of economy promised for tests constructed in this way (Hornke, 1993). Assuming that a
sufficiently large item pool is available, each subject receives the specific items appropriate
to his or her ability; each item therefore generates the maximum amount of information
about the latent trait.
The result is that a pre-defined level of precision can be achieved with a minimum num-
ber of items – a situation that at first sight appears very economical. Since, however, CAT
can only be used within the framework of a probabilistic test model – usually the 1-PL
model of Rasch (Rasch, 1980) or the 2-PL model of Birnbaum (1968) – items must be pre-
sented in the form of a pure power test. The subject alone is responsible for selecting the
amount of time he or she will invest in working on an item. In consequence the response
time per item may increase significantly and this may offset or even outweigh the saving in
the number of items that need to be presented (Wild, 1989). Approaches have been devised
which attempt to counter this problem with “speed-adapted testing” (Nährer, 1989). This
method incorporates an exploratory phase in which a linear speed-accuracy trade-off is esti-
mated; this is used in the test phase to select items with a very high probability of leading to
answers within the time limits set for individual items. It therefore provides a means of coun-
teracting those effects of time limits, which are disadvantageous for probabilistic test models.
Adaptive tests based on probabilistic speed and power models (Scheiblechner, 1979; Ros-
kam, 1997) would be able to solve this problem more elegantly, but no such tests are as yet
in existence. The Frankfurt Adaptive Concentration Test (FAKT; Moosbrugger & Gold-
hammer, 2006) could be mentioned as an example of an adaptive speed test. However, the
problem described above does not arise with this test, since in FAKT speed and accuracy are
aggregated in an unsystematic way into a single measure of performance. In his context
“unsystematic” refers to the fact that it could not be shown that the combined scoring of
accuracy and speed in fact corresponds to the speed-accuracy trade-off that characterizes this
type of task (cf. Häusler, 2004a). The combined scoring therefore could favour a specific
working style on the impulsive/reflexive dimension; the test would therefore not meet the
criterion of fairness.
Furthermore, CAT is often described by test subjects as over-challenging, too tiring and
unduly demotivating (Andrich, 1995). This is in sharp contrast to the original intention,
which was to use item selection to create a challenging and optimally motivating situation in
which the subject feels neither over- nor under-challenged. The findings of Heckhausen
(1989) shed doubt on whether all subjects prefer a success probability of p=0.5. Heckhausen
postulates inter-individual differences, with success-motivated subjects preferring a success
probability of around 70-80%, while failure-avoidance motivated subjects prefer success
probabilities that are minimally informative (very high or very low). In psychological as-
sessment it does of course make no sense to provide subjects with tasks of the latter type.
According to Atkinson (1964) the subjective expectancy of success or failure is a determin-
ing factor of motivation. This expectancy primarily depends on the difficulty of the item.
According to Koestler & McClelland (1990) a moderate difficulty is to be preferred, where
success can be achieved by investing enough effort.

Page 3

J. Häusler
438
The result of these problems is that adaptive tests, despite the innovative principle behind
them, still do not get the credit they deserve. The present paper aims at initiating a discussion
of how CAT procedures might be modified in order to make them more appropriate.


The method of adaptive success control

The primary goal of Nährers “speed-adapted testing” was a test duration that is calcula-
ble in advance and as far as possible similar for all subjects. By contrast, the method de-
scribed in this paper aims at the optimization of test duration in order to improve test econ-
omy. The testing process can be viewed as an information-gathering task in which informa-
tion on the value of a latent trait in a particular individual is acquired. Concepts from the
field of information theory can therefore be applied.
CAT maximises the amount of information (I) generated by an item.

!
2
( |
+ ξ σ
, )* ( |
P
,)max.
(1)
ji
ji
ijiji
e
IP
e
ξ −σ
ξ −σ
= − ξ σ ==
+
(1)

This is achieved when the person ability parameter
j ξ equals the item difficulty parame-
ter
i σ . In this case – in accordance with the model equation of the Rasch model – the success
ξ −σ
+ ξ σ ==
+
ered is I=0.25. If items are presented with a success probability either above or below p=0.5,
the amount of information generated will be correspondingly less. The relation between
success probability and information generated is described by the information function. The
shape of this function is, however, extremely flat. This means that within reasonable limits
there is very little loss of information when the success probability deviates from p=0.5.
If the success probabilities of a linear test with fixed item sequence are assumed to be
distributed uniformly within the interval p=[0.2;0.8], the optimal adaptive test is only about
13% more informative. If the success probabilities are assumed to have a normal distribution
(m=0.5; s=0.1), the information gain by applying an adaptive test procedure is only around
5%.
Since CAT procedures continue testing until a particular level of reliability has been
reached and since the standard error of measurement SEM is a function of the information (I)
1
SEM
I
items. The number of items is in turn related – via the response time of the individual items –
to the total test duration T.

1
*
*ISEM
probability is
( |,) 0.5
1
ji
ji
ji
e
P
e
ξ −σ
; the corresponding amount of information gath-
collected (
=
) equation (1) can be resolved analytically for the required number of
2
Tt
=
(2)

Page 4

Adaptive success control in computerized adaptive testing
439
The function of test duration is therefore made up of two components, namely the test
length and the mean item response time t . The test length is the number of items, necessary
to reach a given standard error of measurement (SEM) and can be defined by using the in-
formation function. The mean item response time on the other hand is an unknown function
of the success probability. There follows a demonstration of how the type of this unknown
function affects the relation between success probability and the test duration. Figure 1
shows examples of several different forms of relationship and demonstrates that the minimal
test duration is not necessarily achieved by means of items with a success probability of
p=0.5.
A constant relation between success probability and item response time is highly improb-
able. Decision-making models for stimulus pair comparisons (Vickers, 1970) or multiple
choice situations (Häusler, 2004b) postulate that the decision time required increases (e.g.
linearly) as success probability decreases. This can be explained theoretically by viewing the
solution process as a method of accumulating information that is continued until an inter-
individually different required response certainty threshold has been reached. The item is
then answered – but not necessarily correctly. It is therefore entirely plausible that relations




Figure 1:
Examples of the effect of different mathematical relations between success probability
(represented here by the item difficulty σ for a subject of ability ξ=0 based on the model
e
P
e
+
duration T. b0 and b1 are the parameters of the different relations.
In the example on the left there is assumed to be a constant relation (the test duration is
independent of the success probability) between success probability and item response time.
The relation between success probability and total test duration corresponds to the inverse
information function. The test is longest when items that are too easy or too difficult are
used; it is shortest when items with p=0.5 are presented.
In the other two examples there is assumed to be a linear (centre) or exponential (right)
relation between success probability and item response time. Here the shortest overall test
duration is no longer achieved with items with p=0.5, but with more simple items.
equation of the Rasch model
1
ξ−σ
ξ−σ
=
) and mean item response time on the total test

Page 5

J. Häusler
440
between success probability and response time might be modelled in which success prob-
abilities of p=0.5 lead to sub-optimal minimization of test duration. In fact the method cur-
rently used in CAT is only a marginal solution for the unlikely event that there is a constant
relation between success probability and item response time.
With Adaptive Success Control (ASC) not only item difficulty but also success probabil-
ity is adapted to the individual subject. It is therefore a second-order adaptive process. The
success control algorithm operates as an approximation algorithm for estimating the most
economical success probability on the basis of the previous item response times. The adap-
tive algorithm selects an item that offers the subject the previously optimized success prob-
ability. Figure 2 shows the closed-loop systems of adaptive testing and adaptive success
control.
For many subjects ASC would result in little, if any, shortening of test duration. Adap-
tive adjustment of success probability is therefore only necessary and useful with test sub-
jects with particular working styles, or in some cases also with particular test materials. The
ASC algorithm should therefore only be triggered if a subject’s response times over several
items have exceeded a particular threshold.
However, because of the limited data available, actual estimation of the optimal test du-
ration is scarcely feasible. Neither can it be expected that a large amount of the variance of
the test duration can be explained by the relation between success probability and response
time. Since, however, the aim is only the pragmatic improvement of test economy, it is in-
stead sufficient to formulate an appropriate heuristic adaptation method.




Figure 2:
Adaptive Testing (left) and Adaptive Success Control (right) represented as servoloops.
Adaptive Testing is a first-order closed-loop system, in which the selection of items is
determined by the performance achieved (adaptive difficulty control). By contrast, Adaptive
Success Control utilizes an additional loop in order to adapt the desired success probability
to the subject’s working style.