Ned L Kirsch’s research while affiliated with University of Michigan and other places

This study was designed to develop validity cutoffs within the Finger Tapping Test (FTT) using demographically adjusted T-scores, and to compare their classification accuracy to existing cutoffs based on raw scores. Given that FTT performance is known to vary with age, sex, and level of education, failure to correct for these demographic variables poses the risk of elevated false positive rates in examinees who, at the level of raw scores, have inherently lower FTT performance (women, older, and less educated individuals). Data were collected from an archival sample of 100 adult outpatients (MAge = 38.8 years, MEducation = 13.7 years, 56% men) consecutively referred for neuropsychological assessment at an academic medical center in the Midwestern USA after sustaining a traumatic brain injury (TBI). Performance validity was psychometrically defined using the Word Memory Test and two validity composites based on five embedded performance validity indicators. Previously published raw score-based validity cutoffs disproportionately sacrificed sensitivity (.13–.33) for specificity (.98–1.00). Worse yet, they were confounded by sex and education. Newly introduced demographically adjusted cutoffs (T ≤ 33 for the dominant hand, T ≤ 37 for both hands) produced high levels of specificity (.89–.98) and acceptable sensitivity (.36–.55) across criterion measures. Equally importantly, they were robust to injury severity and demographic variables. The present findings provide empirical support for a growing trend of demographically adjusted performance validity cutoffs. They provide a practical and epistemologically superior alternative to raw score cutoffs, while also reducing the potential bias against examinees inherently vulnerable to lower raw score level FTT performance.

This study was designed to develop validity cutoffs by utilizing demographically adjusted T-scores on the trail making test (TMT), with the goal of eliminating potential age and education-related biases associated with the use of raw score cutoffs. Failure to correct for the effect of age and education on TMT performance may lead to increased false positive errors for older adults and examinees with lower levels of education. Data were collected from an archival sample of 100 adult outpatients (MAge = 38.8, 56% male; MEd = 13.7) who were clinically referred for neuropsychological assessment at an academic medical center in the Midwestern USA after sustaining a traumatic brain injury (TBI). Performance validity was psychometrically determined using the Word Memory Test and two multivariate validity composites based on five embedded performance validity indicators. Cutoffs on the demographically corrected TMT T-scores had generally superior classification accuracy compared to the raw score cutoffs reported in the literature. As expected, the T-scores also eliminated age and education bias that was observed in the raw score cutoffs. Both T-score and raw score cutoffs were orthogonal to injury severity. Multivariate models of T-score based cutoff failed to improve classification accuracy over univariate T-score cutoffs. The present findings provide support for the use of demographically adjusted validity cutoffs within the TMT. They produced superior classification to raw score-based cutoffs, in addition to eliminating the bias against older adults and examinees with lower levels of education.

This study was designed to replicate an earlier report on the link between low scores on the Grooved Pegboard test (GPB), invalid responding, and elevated self-reported psychiatric symptoms. A fixed battery of neuropsychological tests was administered to 100 consecutively referred outpatients (MAge = 38.3, MEducation = 13.6 years) following traumatic brain injury at a Midwestern academic medical center. Classification accuracy of GPB validity cutoffs was computed against a free-standing PVT and three composite measures of embedded validity indicators. Previously suggested GPB validity cutoffs (T ≤ 29 in either hand) produced good combinations of sensitivity (0.25–0.55) and specificity (0.89–0.98) to psychometrically defined invalid performance. Raising the cutoff to T ≤ 31 resulted in a reasonable trade-off between increased sensitivity (0.36–0.55) and decreased specificity (0.84–0.94). T ≤ 31 in both hands was highly specific (0.93–0.98) to noncredible responding. GPB validity cutoffs were unrelated to psychiatric symptoms or injury severity. Failing PVTs based on forced choice recognition was associated with elevated self-reported depression, somatic concerns, and overall symptomatology. Low scores on the GPB are reliable indicators of noncredible responding. Self-reported emotional distress has a complex relationship with performance validity. Psychogenic interference is a potential mechanism behind PVT failures, and its expression is likely mediated by instrumentation and sampling artifacts. Further research on the topic is clearly needed to advance current understanding of psychogenic interference as a confound in cognitive testing.

Objective The Forced Choice Recognition (FCR) trial of the California Verbal Learning Test—Second Edition (CVLT-II) was designed to serve as a performance validity test (PVT). The present study was designed to compare the classification accuracy of a more liberal alternative (≤15) to the de facto FCR cutoff (≤14). Method The classification accuracy of the two cutoffs was computed in reference to psychometrically defined invalid performance, across various criterion measures, in a sample of 104 adults with TBI clinically referred for neuropsychological assessment. Results The FCR was highly predictive (AUC: .71–.83) of Pass/Fail status on reference PVTs, but unrelated to performance on measures known to be sensitive to TBI. On average, FCR ≤15 correctly identified an additional 6% of invalid response sets compared to FCR ≤14, while maintaining .92 specificity. Patients who failed the FCR reported higher levels of emotional distress. Conclusions Results suggest that even a single error on the FCR is a reliable indicator of invalid responding. Further research is needed to investigate the clinical significance of the relationship between failing the FCR and level of self-reported psychiatric symptoms.

This study examined the clinical utility of the Wechsler Adult Intelligence Scales-Fourth Edition (WAIS-IV) in individuals with complicated mild, moderate or severe TBI. One hundred individuals with TBI (n = 35 complicated mild or moderate TBI; n = 65 severe TBI) and 100 control participants matched on key demographic variables from the WAIS-IV normative dataset completed the WAIS-IV. Univariate analyses indicated that participants with severe TBI had poorer performance than matched controls on all index scores and subtests (except Matrix Reasoning). Individuals with complicated mild/moderate TBI performed more poorly than controls on the Working Memory Index (WMI), Processing Speed Index (PSI), and Full Scale IQ (FSIQ), and on four subtests: the two processing speed subtests (SS, CD), two working memory subtests (AR, LN), and a perceptual reasoning subtest (BD). Participants with severe TBI had significantly lower scores than the complicated mild/moderate TBI on PSI, and on three subtests: the two processing speed subtests (SS and CD), and the new visual puzzles test. Effect sizes for index and subtest scores were generally small-to-moderate for the group with complicated mild/moderate and moderate-to-large for the group with severe TBI. PSI also showed good sensitivity and specificity for classifying individuals with severe TBI versus controls. Findings provide support for the clinical utility of the WAIS-IV in individuals with complicated mild, moderate, and severe TBI.

The Recognition Memory Test (RMT) was compared to the Word Choice Test (WCT) within the same fixed neuropsychological battery administered to a mixed clinical sample of 237 adults to empirically evaluate the psychometric equivalence of these two instruments. On average, there was a 3-point difference in raw scores between the two instruments (M RMT = 44.3, SD RMT = 6.8; M WCT = 47.1, SD WCT = 4.6; p < .001, d = .48). The probability density functions differ substantially at the two ends of the scale but are similarly ≤42. Cross-validation analyses suggest that the RMT cutoff of ≤42 is functionally equivalent to a WCT score of ≤45.

Continuous performance tests (CPT) provide a useful paradigm to assess vigilance and sustained attention. However, few established methods exist to assess the validity of a given response set. The present study examined embedded validity indicators (EVIs) previously found effective at dissociating valid from invalid performance in relation to well-established performance validity tests in 104 adults with TBI referred for neuropsychological testing. Findings suggest that aggregating EVIs increases their signal detection performance. While individual EVIs performed well at their optimal cutoffs, two specific combinations of these five indicators generally produced the best classification accuracy. A CVI-5A ≥3 had a specificity of .92–.95 and a sensitivity of .45–.54. At ≥4 the CVI-5B had a specificity of .94–.97 and sensitivity of .40–.50. The CVI-5s provide a single numerical summary of the cumulative evidence of invalid performance within the CPT-II. Results support the use of a flexible, multivariate approach to performance validity assessment.