Tracy K, Adler LA, Rotrosen J, Edson R, Lavori P. Interrater reliability issues in multicenter trials, Part I: Theoretical concepts and operational procedures used in Department of Veterans Affairs Cooperative Study #394. Psychopharmacol Bull 33: 53-57
ABSTRACT This article describes a standardized method for establishing and maintaining desired levels of interrater reliability (IRR) in multicenter trials. The procedure involves six steps: distribution of procedural guides, distribution of an introduction tape, initial distribution of patient interviews to rate, training at the study kickoff meeting, ongoing IRR monitoring, and group training throughout the study. This method is being used in a national Veterans Affairs Cooperative Study (CS #394), involving nine sites to examine the treatment effects of vitamin E on tardive dyskinesia. The six-step standardized process allowed for early detection of areas of concern in assessment administration. When comparing intraclass correlation coefficients (ICCs) at different points in the initial training, the Barnes Akathisia Scale and Anchored Brief Psychiatric Rating Scale reliability improved from 0.68 to 0.74 and from 0.54 to 0.87, respectively. After analyzing the ratings collected prior to the start of CS #394, data were collected to conduct the first check on Abnormal Involuntary Movement Scale (AIMS) IRR during enrollment; the estimated ICC for the AIMS had decreased from 0.87 to 0.60. Raters were instructed to re-assess the subjects from the first videotape on the AIMS and received additional training. The re-rating indicated very good reliability, 0.84, IRR was measured once for the Global Assessment of Functioning Scale resulting in an ICC of 0.90. The companion article (Part II: Edson et al. 1997, page 59 of this issue) describes the statistical procedures used to measure IRR.
- SourceAvailable from: David Haak
[Show abstract] [Hide abstract]
- "This information can then be directly imported into the data base, automatically updating existing information for each rater. Maintaining a data base provides quick feedback to all raters regarding their status and ratings—an important part of maintaining interrater reliability (Tracy et al. 1997; Igarashi et al. 1998). "
ABSTRACT: Schizophrenia is a symptomatically heterogeneous disorder characterized by the presence of positive and negative symptoms, and variable impairment in community functioning. Given the diversity of symptom presentations and functioning associated with schizophrenia, one of the key challenges facing the Clinical Antipsychotic Trials of Intervention Effectiveness (CATIE) schizophrenia trial was the selection of efficient assessment measures appropriate to a community-based effectiveness trial. This article describes the rationale for the measurement approach adopted for the trial, provides a brief overview of the selected measures, and describes the process of training assessment raters for a large and geographically dispersed study group.Schizophrenia Bulletin 02/2003; 29(1):33-43. DOI:10.1093/oxfordjournals.schbul.a006989 · 8.45 Impact Factor