No full-text available
To read the full-text of this research,
you can request a copy directly from the authors.
Merging relative and absolute methods: The IRT-Angoff method for pass mark identification
Medical Teacher
... For example, we have recently merged the Rasch model and the Angoff ratings allowing for a systematic conversion of subjective judgments into an objective measurement scale aiming to provide a deep understanding of item difficulty, considering both empirical data and opinions of standard setters, potentially enhancing the fairness and accuracy of assessments. 7 For more details on these variants, we suggest referring to the Berk guide. 4 Neither pass-mark or pass-mark based decision making are immune from random or systematic errors. ...
Objectives:
To measure intra-standard-setter variability and assess the variations between the pass marks obtained from Angoff ratings, guided by the latent trait theory as the theoretical model.
Methods:
A non-experimental cross-sectional study was conducted to achieve the purpose of the study. Two knowledge-based tests were administered to 358 final-year medical students (223 females and 135 males) as part of their normal summative programme of assessments. The results of judgmental standard-setting using the Angoff method, which is widely used in medical schools, were used to determine intra-standard-setter inconsistency using the three-parameter item response theory (IRT). Permission for this study was granted by the local Research Ethics Committee of the University of Nottingham. To ensure anonymity and confidentiality, all identifiers at the student level were removed before the data were analysed.
Results:
The results of this study confirm that the three-parameter IRT can be used to analyse the results of individual judgmental standard setters. Overall, standard-setters behaved fairly consistently in both tests. The mean Angoff ratings and conditional probability were strongly positively correlated, which is a matter of inter-standard-setter validity.
Conclusions:
We recommend that assessment providers adopt the methodology used in this study to help determine inter and intra-judgmental inconsistencies across standard setters to minimise the number of false positive and false negative decisions.
Although the Angoff procedure is among the most widely used standard setting procedures for tests comprising multiple-choice items, research has shown that subject matter experts have considerable difficulty accurately making the required judgments in the absence of examinee performance data. Some authors have viewed the need to provide performance data as a fatal flaw for the procedure; others have considered it appropriate for experts to integrate performance data into their judgments but have been concerned that experts may rely too heavily on the data. There have, however, been relatively few studies examining how experts use the data. This article reports on two studies that examine how experts modify their judgments after reviewing data. In both studies, data for some items were accurate and data for other items had been manipulated. Judges in both studies substantially modified their judgments whether the data were accurate or not.
This study represents an attempt at incorporating empirical item difficulty data into the Angoff standard-setting procedure without affecting the subjective judgment of the raters. The Rasch-model ability level corresponding to minimal competence was estimated for each of 536 items on the American Association of State Social Work Boards (AASSWB) social work licensure examinations from their empirical calibrations and Angoff ratings. The mean of these estimates for all items on a given examination was taken as the level of minimal competence of the entire examination. This procedure yielded raw passing scores that were 4 to 6 items lower (out of 150) and pass rates that were 7% to 15% higher than those obtained using the "standard" Angoff procedure.