Table 7 - uploaded by Laszlo A Erdodi
Content may be subject to copyright.
Source publication
Objective
The Forced Choice Recognition (FCR) trial of the California Verbal Learning Test—Second Edition (CVLT-II) was designed to serve as a performance validity test (PVT). The present study was designed to compare the classification accuracy of a more liberal alternative (≤15) to the de facto FCR cutoff (≤14).
Method
The classification accurac...
Contexts in source publication
Context 1
... SCL-90-R scores were dichotomized around the T ≥ 63 cutoff into "clinical" versus "non-clinical", non-parametric contrasts produced essentially the same results ( Table 7). One comparison (PAR) became non-significant. ...
Citations
... The California Verbal Learning Test-Second Edition (CVLT-II) (Delis et al., 2008) is a test measuring verbal learning and memory. Scores ≤11 on total recognition hits or scores ≤15 on forced-choice recognition indicated noncredible performance, which previous research has supported (Delis et al., 2008;Persinger et al., 2017;Erdodi et al., 2018). ...
Objective
Research has demonstrated that over-reporting and under-reporting, when detected by the MMPI-2/-RF Validity Scales, generalize to responses to other self-report measures. The purpose of this study was to investigate whether the same is true for the Minnesota Multiphasic Personality Inventory–3 (MMPI-3) Validity Scales. We examined the generalizability of over-reporting and under-reporting detected by MMPI-3 Validity Scales to extra-test self-report, performance-based, and performance validity measures.
Method
The sample included 665 majority White, male disability claimants who, in addition to the MMPI-3, were administered several self-report measures, some with embedded symptom validity tests (SVTs), performance-based measures, and performance validity tests (PVTs). Three groups were identified based on MMPI-3 Validity Scale scores as over-reporting (n = 276), under-reporting (n = 100), or scoring within normal limits (WNL; n = 289).
Results
Over-reporting on the MMPI-3 generalized to symptom over-reporting on concurrently administered self-report measures of psychopathology and was associated with evidence of over-reporting from other embedded SVTs. It was also associated with poorer performance on concurrently administered measures of cognitive functioning and PVTs. Under-reporting on the MMPI-3 generalized to symptom minimization on collateral measures of psychopathology. On measures of cognitive functioning, we found no differences between the under-reporting and WNL groups, except for the Wisconsin Card Sorting Test–64 Card Version and Wide Range Achievement Test–Fifth Edition (each with negligible effect sizes).
Conclusions
MMPI-3 Validity Scales can identify possible over- and under-reporting on concurrently administered measures. This can be of particular value when such measures lack validity indicators.
... Patient characteristics and neuropsychological test data were extracted from patient medical records. PVTs with preestablished conservative and liberal cutoff scores included Digit Span Age-Corrected Scaled Score and Reliable Digit Span (Greiffenstein et al., 1994;Erdodi, 2019) from the Wechsler Adult Intelligence Scale-IV; Medical Symptom Validity Test (Howe et al., 2007); Test of Memory Malingering Trial 1 (Denning, 2012;Greve et al., 2006;Jones, 2013;Kulas et al., 2014;Rai & Erdodi, 2021); and either the California Verbal Learning Test, Third Edition, Recognition Hits and Forced Choice Recognition (Erdodi et al., 2018) or Hopkins Verbal Learning Test-Revised Recognition Discriminability and Forced Choice Recognition (Abeare et al., 2021;Cutler et al., 2022). Of note, Trial 1 from the Test of Memory Malingering was used as part of an abbreviated administration protocol. ...
Objective: Severe acute respiratory syndrome coronavirus 2 may result in persistent physical, psychological, and/or cognitive symptoms, termed postacute sequelae of COVID-19 (PASC). A growing literature has documented the presence of cognitive deficits in individuals diagnosed with PASC; however, the use of performance validity tests (PVTs) has varied widely. The present study characterized base rates of PVT failure in patients with PASC. Method: The sample consisted of 91 adults with PASC referred for clinical neuropsychological evaluation between May 2021 and September 2023. Patients were administered a battery of neuropsychological tests, self-report symptom checklists, and two or more PVTs, including the Test of Memory Malingering Trial 1; Medical Symptom Validity Test; Digit Span Age-Corrected Scaled Score and Reliable Digit Span; and either California Verbal Learning Test, Third Edition, Recognition Hits and Forced-Choice Recognition (FCR) or Hopkins Verbal Learning Test–Revised Recognition Discriminability and FCR. Results: Using a threshold of ≥2 PVT failures, nine patients (9.9%) were classified as having invalid performance when using more conservative dichotomous cutoffs for individual PVTs, and 13 patients (14.3%) were classified when using more liberal cutoffs. When using trichotomous cutoffs, 25 patients’ performances (27.5%) were considered borderline, and eight (8.8%) were considered invalid. Conclusions: These findings indicate that PVT failure in patients with PASC is relatively uncommon, but these findings, nonetheless, highlight the importance of including PVTs in clinical and research settings when assessing cognition in those with PASC.
... In our study, the cut score used for this measure was ≤14 as mentioned in the CVLT-II manual (Delis et al., 2000), which suggests that over 90% of the CVLT-II normative sample obtained a perfect score on FCR (16/16), with ≤1% scoring ≤14. In addition, this cut score has demonstrated strong specificity (≥90%) in independent studies (Erdodi et al., 2018;Root et al., 2006). Although sensitivity tends to be weaker at these specificity values, strong specificity was prioritized in accordance with Sherman et al. (2020)'s recommendations. ...
... This suggests that when applied to a similar H/L population, the recommended cutoff of <15 is likely appropriate. Notably, this suggested cutoff has been considered conservative when examined in English-speaking samples in prior studies (Erdodi et al., 2018;Resch et al., 2022). ...
Objective: The use of performance validity tests (PVTs) to assess suboptimal performance has become a recommended feature in neuropsychological evaluations. However, most of the literature has examined the utility of PVTs in primarily non-Hispanic White, monolingual English-speaking samples. Given the continual growth of culturally and linguistically diverse populations, especially Hispanic/Latin Americans (H/Ls), it is necessary to provide an updated review of the available literature involving the use of PVTs in H/L adults. Method: We conducted a systematic review that examined the available literature surrounding the use of performance validity measures in H/L adults. This review identified evidence for the utility of PVTs in assessing invalid performance in H/Ls and compiled psychometric properties related to these measures, specifically rates of specificity and sensitivity when available. Results: We identified six standalone and four embedded performance validity measures that have been examined in H/Ls. The Test of Memory Malingering was identified as the PVT most studied in various H/L heritage groups. Across the studies reviewed, adjustments to cutoff scores are generally recommended for the majority of PVTs discussed. Within H/Ls, specificity rates tend to be bolstered when sociocultural factors, like education, are considered. Conclusions: Recommended cutoff scores across the different PVTs reviewed are not typically suitable for use in heritage groups of H/L adults, especially in groups that include adults with lower levels of education and literacy. Therefore, education may need to be considered when applying cutoffs to these populations.
... Abeare et al., 2019). Noncredible performance also contaminates the normative samples for most cognitive tests: Given the cumulative evidence that cutoffs in the borderline/low average range (3rd to 25th percentile) are specific to invalid responding (Ashendorf et al., 2017;Deloria et al., 2021;Erdodi, Abeare, et al., 2018;Erdodi, Hurtubise, et al., 2018;Pearson, 2009;Shura et al., 2016Shura et al., , 2020Tyson et al., 2023), the logical conclusion is that a significant minority of the individuals who volunteered their data for these massive instrument development projects did not provide an optimal (or even representative) sample of their cognitive abilities (Erdodi & Lichtenstein, 2017). ...
This editorial article introduces a special issue of Psychology & Neuroscience dealing with performance and symptom validity testing (SVTs). We first discuss the importance of assessing the credibility of observed performance on cognitive tasks and of symptoms reported in questionnaires or clinical interviews, both in research and in clinical and forensic settings. We then briefly summarize the content of each article in this special issue and discuss their contribution to this topic. We conclude that practitioners have an increasing number of embedded performance validity tests (PVTs) at their disposal, so current research trends are focused on finding newer and better algorithms for integrating results from multiple PVTs. In contrast, there are significantly fewer SVTs available to practitioners, so researchers in this area currently seem to be focused on developing and validating both embedded and free-standing SVTs.
... CVLT-II scores have been shown to be sensitive to noncredible performance. The present study considered participants who produced scores on total recognition hits ≤ 11 (Persinger et al., 2017) or forced choice recognition ≤ 15 (Erdodi et al., 2018) to indicate noncredible performance. ...
Objective: The present study was the first to investigate the test performance and symptom reports of individuals who engage in both over-reporting (i.e., exaggerating or fabricating symptoms) and under-reporting (i.e., exaggerating positive qualities or denying shortcomings) in the context of a forensic evaluation. We focused on comparing individuals who over- and under-reported (OR+UR) with those who only over-reported (OR-only) on the MMPI-3.
Method: Using a disability claimant sample referred for comprehensive psychological evaluations (n = 848), the present study first determined the rates of possible over-reporting (MMPI-3 F ≥ 75T, Fp ≥ 70T, Fs ≥ 100T, or FBS or RBS ≥ 90T) with (n = 42) and without (n = 332) under-reporting (L ≥ 65T). Next, we examined group mean differences on MMPI-3 substantive scale scores and scores on several additional measures completed by the disability claimant sample during their evaluation.
Results: The small group of individuals identified as both over-reporting and under-reporting (OR+UR) scored meaningfully higher than the OR-only group on several over- and under-reporting symptom validity tests, as well as on measures of emotional and cognitive/somatic complaints, but lower on externalizing measures. The OR+UR group also performed significantly worse than the OR-only group on several performance validity tests and measures of cognitive ability.
Conclusions: The present study indicated that disability claimants who engage in simultaneous over- and under-reporting portray themselves as having greater levels of dysfunction but fewer externalizing tendencies relative to claimants who only over-report; however, these portrayals are likely less accurate reflections of their true functioning.
... This is not standard practice, as typically studies will utilize another robust PVT as a means of calculating sensitivity and specificity values for another PVT such as the TOMM. While an embedded PVT is not as robust as a stand-alone measure, the CVLT-II is still a reasonable embedded test to use when calculating sensitivity and specificity, as its features are consistent with stand-alone PVTs (Erdodi et al., 2018). ...
... Therefore, only an embedded PVT was available for use. An embedded PVT is not as robust as a stand-alone measure; however, the CVLT-II is still a reasonable embedded test to use when calculating sensitivity and specificity, as its features are consistent with standalone PVTs, and was developed with the primary purpose of evaluating response validity (Bauer et al., 2005;Erdodi et al., 2018;Schwartz et al., 2016). ...
The accuracy of neuropsychological assessments relies on participants exhibiting their true abilities during administration. The Test of Memory Malingering (TOMM) is a popular performance validity test used to determine whether an individual is providing honest answers. While the TOMM has proven to be highly sensitive to those who are deliberately exaggerating their symptoms, there is a limited explanation regarding the significance of using 45 as a cutoff score. The present study aims to further investigate this question by examining TOMM scores obtained in a large sample of active-duty military personnel (N = 859, M = 26 years, SD = 6.14, 97.31% males, 72.44% white). Results indicated that no notable discrepancies existed between the frequency of participants who scored a 45 and those who scored slightly below a 45 on the TOMM. The sensitivity and specificity of the TOMM were derived using the forced-choice recognition (FCR) scores obtained by participants on the California Verbal Learning Test, Second Edition (CVLT-II). The sensitivity for each trial of the TOMM was 0.84, 0.55, and 0.63, respectively; the specificity for each trial of the TOMM was 0.69, 0.93, and 0.92, respectively. Because sensitivity and specificity rates are both of importance in this study, balanced accuracy scores were also reported. Results suggested that various alternative cutoff scores produced a more accurate classification compared to the traditional cutoff of 45. Further analyses using Fisher's exact test also indicated that there were no significant performance differences on the FCR of the CVLT-II between individuals who received a 44 and individuals who received a 45 on the TOMM. The current study provides evidence on why the traditional cutoff may not be the most effective score. Future research should consider employing alternative methods which do not rely on a single score.
... FCR has been deemed highly specific and moderately sensitive to invalid responses in nonimpaired individuals (Schwartz et al., 2016), and has reasonable concurrent validity with other measures, such as the Test of Memory Malingering . It has also been employed in a brain injured sample, with results suggesting that even a single error on FCR is sufficient to identify invalid responses (Erdodi et al., 2018). The sole piece of the literature identified about the CVLT-II-SF FCR in dementia suggests a similar conclusion. ...
... In the literature, there have been several reports of appropriate cut scores for performance validity determination using FCR. For example, Erdodi et al. (2018) reported a potentially useful cutoff of one error on the CVLT-II FCR in a TBI sample, finding it unrelated to TBI-sensitive measures. A similar finding was reported by Fogel et al. (2013) in their dementia sample. ...
... They reported a potential cutoff of 8/9 for the CVLT-II-SF FCR as a measure of performance validity in a dementia sample, while acknowledging the significant relationship between performance validity test performance and cognition that has been replicated here and elsewhere (e.g., Burton et al., 2015;Dean et al., 2009). Given that the present mean CVLT-II-SF FCR value for those diagnosed with AD was 7.96 (SD ¼ 0.12), it may not be appropriate to ascertain poor performance validity based on existing suggestions for single-error cutoff scores (Erdodi et al., 2018;Fogel et al., 2013) in dementia. It is still possible that the proposed literature-based cutoff of one error is adequate for discerning performance validity (aligning with Schwartz et al. 2016) when combined with additional measures of performance validity and clinical judgment. ...
Performance validity tests are susceptible to false positives from genuine cognitive impairment (e.g., dementia); this has not been explored with the short form of the California Verbal Learning Test II (CVLT-II-SF). In a memory clinic sample, we examined whether CVLT-II-SF Forced Choice Recognition (FCR) scores differed across diagnostic groups, and how the severity of impairment [Clinical Dementia Rating Sum of Boxes (CDR-SOB) or Mini-Mental State Examination (MMSE)] modulated test performance. Three diagnostic groups were identified: subjective cognitive impairment (SCI; n = 85), amnestic mild cognitive impairment (a-MCI; n = 17), and dementia due to Alzheimer's Disease (AD; n = 50). Significant group differences in FCR were observed using one-way ANOVA; post-hoc analysis indicated the AD group performed significantly worse than the other groups. Using multiple regression, FCR performance was modeled as a function of the diagnostic group, severity (MMSE or CDR-SOB), and their interaction. Results yielded significant main effects for MMSE and diagnostic group, with a significant interaction. CDR-SOB analyses were non-significant. Increases in impairment disproportionately impacted FCR performance for persons with AD, adding caution to research-based cutoffs for performance validity in dementia. Caution is warranted when assessing performance validity in dementia populations. Future research should examine whether CVLT-II-SF-FCR is appropriately specific for best-practice testing batteries for dementia.
... Examples of such forced choice PVTs include the Test of Memory Malingering (TOMM; Tombaugh, 1996), Word Memory Test (WMT; Green, 2003), and the California Verbal Learning Test, Second Edition (CVLT-II) Forced Choice Recognition (Delis, Kramer, Kaplan, & Ober, 2000). A strength of forced choice PVTs is that even moderately to severely impaired individuals, such as those with moderate to severe traumatic brain injury (TBI), intellectual disability, and mild to moderate dementia, make few errors (Dean, Victor, Boone, Philpott, & Hess, 2009;Erdodi et al., 2018;Marshall & Happe, 2007). ...
... The results of this study lend support to the utility of a novel forced choice PVT for RBANS, the RFC. Consistent with past findings regarding CVLT-II Forced Choice performance (e.g., Erdodi et al., 2018), RFC errors were uncommon in our sample of non-litigating older adult patients ranging in cognitive status from intact to severely impaired. Given that participants in this study were presumed not to be intentionally biasing their performance, our findings suggest that the RFC may be less susceptible to false positive errors (indicating non-credible performance) even among individuals with more severe neurological conditions compared with the EI and ES, two previously established RBANS PVTs. ...
Objective:
We aimed to assess the utility of a novel and easy-to-administer performance validity test (PVT), the Repeatable Battery for the Assessment of Neuropsychological Status (RBANS) Forced Choice recognition trial (RFC), compared with previously developed RBANS PVTs.
Method:
We compared the RFC with the RBANS Effort Index (EI) and Effort Scale (ES) in a sample of 62 non-litigating older adults (mean age = 74 years, 52% female) with varying levels of cognitive impairment.
Results:
A significantly greater proportion of the sample met EI criteria for non-credible performance (EI > 2; 31%) compared with RFC criteria (RFC < 9; 15%). Among participants with Major Neurocognitive Disorder, 60% met EI criteria for non-credible performance, 32% met ES criteria (ES > 12), and 24% met RFC criteria.
Conclusions:
The RFC may have greater specificity than other RBANS PVTs among individuals with more severe cognitive impairment. Further research is needed to establish the classification accuracy of the RFC for assessing performance validity.
... The cognitive impairment group obtained a mean score of 43.9 (5.3) for trial 1 and a mean score of 48.6 (3.1) for trial 2, suggesting that performance on the TOMM is very resistant to different types of severe cognitive impairment (Tombaugh, 1996). While it is traditional to use the cutoff score of <45 on trial 2 as suggested in the manual, this study also examined alternative cutoffs for trial 2 based on existing literature (e.g., Erdodi et al., 2018;Martin et al., 2020) in addition to examining trial 1 data as part of secondary analyses. ...
... Table 5 contains the number of individuals who performed at various cutoffs on the CVLT-FC at three time points. Cutoff scores were determined using existing literature (e.g., Erdodi et al., 2018;Schwartz et al., 2016). One cutoff was based on a systematic review conducted by Schwartz and colleagues (2016) in which they applied a cutoff of 14 on the forced choice trial (sensitivity 50% and specificity 93%). ...
... One cutoff was based on a systematic review conducted by Schwartz and colleagues (2016) in which they applied a cutoff of 14 on the forced choice trial (sensitivity 50% and specificity 93%). Another, more stringent cutoff, was based on a study conducted with a group of mixed TBI individuals (75% mTBI) (Erdodi et al., 2018). In that study, they used a cutoff of 15 on the forced choice trial (sensitivity 56% and specificity 92%). ...
Objective
Assessing performance validity is imperative in both clinical and research contexts as data interpretation presupposes adequate participation from examinees. Performance validity tests (PVTs) are utilized to identify instances in which results cannot be interpreted at face value. This study explored the hit rates for two frequently used PVTs in a research sample of individuals with and without histories of bipolar disorder (BD).
Method
As part of an ongoing longitudinal study of individuals with BD, we examined the performance of 736 individuals with BD and 255 individuals with no history of mental health disorder on the Test of Memory Malingering (TOMM) and the California Verbal Learning Test forced choice trial (CVLT-FC) at three time points.
Results
Undiagnosed individuals demonstrated 100% pass rate on PVTs and individuals with BD passed over 98% of the time. A mixed effects model adjusting for relevant demographic variables revealed no significant difference in TOMM scores between the groups, a = .07, SE = .07, p = .31. On the CVLT-FC, no clinically significant differences were observed ( ps < .001).
Conclusions
Perfect PVT scores were obtained by the majority of individuals, with no differences in failure rates between groups. The tests have approximately >98% specificity in BD and 100% specificity among non-diagnosed individuals. Further, nearly 90% of individuals with BD obtained perfect scores on both measures, a trend observed at each time point.
