What counts as evidence for working memory training? Problems with correlated gains and dichotomization

University of Maryland, College Park, MD, USA, .
Psychonomic Bulletin & Review (Impact Factor: 2.99). 12/2013; 21(3). DOI: 10.3758/s13423-013-0560-7
Source: PubMed

ABSTRACT The question of whether computerized cognitive training leads to generalized improvements of intellectual abilities has been a popular, yet contentious, topic within both the psychological and neurocognitive literatures. Evidence for the effective transfer of cognitive training to nontrained measures of cognitive abilities is mixed, with some studies showing apparent successful transfer, while others have failed to obtain this effect. At the same time, several authors have made claims about both successful and unsuccessful transfer effects on the basis of a form of responder analysis, an analysis technique that shows that those who gain the most on training show the greatest gains on transfer tasks. Through a series of Monte Carlo experiments and mathematical analyses, we demonstrate that the apparent transfer effects observed through responder analysis are illusory and are independent of the effectiveness of cognitive training. We argue that responder analysis can be used neither to support nor to refute hypotheses related to whether cognitive training is a useful intervention to obtain generalized cognitive benefits. We end by discussing several proposed alternative analysis techniques that incorporate training gain scores and argue that none of these methods are appropriate for testing hypotheses regarding the effectiveness of cognitive training.

  • Source
    [Show abstract] [Hide abstract]
    ABSTRACT: There is a replication crisis in science, to which psychological research has not been immune: many effects have proven uncomfortably difficult to reproduce. Although the reliability of data is a serious concern, we argue that there is a deeper and more insidious problem in the field: the persistent and dramatic misinterpretation of empirical results that replicate easily and consistently. Using a series of four highly-studied "textbook" examples from different research domains (eyewitness memory, deductive reasoning, social psychology, and child welfare), we show how simple unrecognized incompatibilities among dependent measures, analysis tools, and the properties of the data can lead to fundamental interpretive errors. These errors, which are not reduced by additional data collection, may lead to misguided research efforts and policy recommendations. We conclude with a set of recommended strategies and research tools to reduce the probability of these persistent and largely unrecognized errors. The use of receiver operating characteristic (ROC) curves is highlighted as one such recommendation.
    Psychonomic Bulletin & Review 11/2014; DOI:10.3758/s13423-014-0759-2 · 2.99 Impact Factor
  • Source
    Frontiers in Systems Neuroscience 09/2014; 8. DOI:10.3389/fnsys.2014.00161
  • Source
    [Show abstract] [Hide abstract]
    ABSTRACT: As a general rule, the repeated administration of tests measuring a given cognitive ability in the same participants reveals increased scores. This brings to life the well-known practice effect and it must be taken into account in research aimed at the proper assessment of changes after the completion of cognitive training programs. Here we focus in one specific research question: Are changes in test scores accounted for by the tapped underlying cognitive construct/factor? The evaluation of the factor of interest by several measures is required for that purpose. 477 university students completed twice a battery of four heterogeneous standardized intelligence tests within a time lapse of four weeks. Between the pre-test and the post-test sessions, some participants completed eighteen practice sessions based on memory span tasks, other participants completed eighteen practice sessions based on processing speed tasks, and a third group of participants did nothing between testing sessions. The three groups showed remarkable changes in test scores from the pre-test to the post-test intelligence session. However, results from multi-group longitudinal latent variable analyses revealed that the identified latent factor tapped by the specific intelligence measures fails to account for the observed changes.
    Intelligence 03/2015; 50:93-99. DOI:10.1016/j.intell.2015.02.004 · 2.67 Impact Factor


Available from
May 23, 2014