David R. Shanks’s research while affiliated with University College London and other places

This study investigates the efficacy of AI-assisted evaluation of open science practices in brain sciences, comparing ChatGPT 4 and Claude 3.5 Sonnet against human expert assessment. We analysed 100 randomly selected journal articles across various brain science disciplines using a 6-item transparency checklist. Three human experts and two AI chatbots independently evaluated the articles. Results showed strong correlations between human and AI chatbot overall ratings. Both chatbots demonstrated high concordance with humans in assessing code sharing, materials availability, preregistration, and sample size rationales. However, they struggled with accurately identifying the presence of data availability statements and assessing public accessibility of shared data. These findings suggest that AI chatbots can effectively support the evaluation of some open science practices and potentially expedite the assessment process in academic research. However, their limitations in certain areas highlight the continued importance of human oversight in ensuring comprehensive and accurate evaluations of scientific transparency.

This study investigates the efficacy of AI-assisted evaluation of open science practices in brain sciences, comparing ChatGPT 4 and Claude 3.5 Sonnet against human expert assessment. We analysed 100 randomly selected journal articles across various brain science disciplines using a 6-item transparency checklist. Three human experts and two AI chatbots independently evaluated the articles. Results showed strong correlations between human and AI chatbot overall ratings. Both chatbots demonstrated high concordance with humans in assessing code sharing, materials availability, preregistration, and sample size rationales. However, they struggled with accurately identifying the presence of data availability statements and assessing public accessibility of shared data. These findings suggest that AI chatbots can effectively support the evaluation of some open science practices and potentially expedite the assessment process in academic research. However, their limitations in certain areas highlight the continued importance of human oversight in ensuring comprehensive and accurate evaluations of scientific transparency.

Research on unconscious processing has been a valuable source of evidence in psycholinguistics for shedding light on the cognitive architecture of language. The automaticity of syntactic processing, in particular, has long been debated. One strategy to establish this automaticity involves detecting significant syntactic priming effects in tasks that limit conscious awareness of the stimuli. Criteria for assessing unconscious priming include the visibility (d’) of masked words not differing significantly from zero and no positive correlation between visibility and priming. However, such outcomes could also arise for strictly methodological reasons, such as low statistical power in visibility tests or low reliability of dependent measures. In this study, we aimed to address these potential limitations. Through meta-analysis and Bayesian re-analysis,we find evidence of low statistical power and of participants having above-chance awareness of ‘subliminal’ words. Moreover, we conducted reliability analyses on a dataset from Berkovitch and Dehaene (2019), finding that low reliability in both syntactic priming and visibility tasks may better explain the absence of a significant correlation. Overall, these findings cast doubt on the validity of previous conclusions regarding the automaticity of syntactic processing based on masked priming effects.The results underscore the importance of revisiting the methods employed when exploring unconscious processing in future psycholinguistic research.

Interleaving (intermixing exemplars from different categories) is more effective in promoting inductive learning than blocking (massing exemplars from a given category together). Yet learners typically prefer blocking over interleaving during self-regulated inductive learning, highlighting the need to develop effective interventions to overcome this metacognitive illusion and promote learners’ practical use of the interleaving strategy. Drawing on a sample of university students, three experiments examined the effects of an instructional intervention on (a) correction of metacognitive fallacies regarding the superiority of blocking over interleaving for inductive learning, (b) adoption of the interleaving strategy during self-regulated learning when learners are allowed to make study choices exemplar-by-exemplar, (c) classification performance, and (d) transfer of category learning across diverse domains. Experiments 1 and 2 showed that instructions about the benefits of interleaving over blocking improved metacognitive awareness of the efficacy of interleaving and enhanced self-usage of the interleaving strategy during learning of new categories. However, this intervention had negligible influence on interleaving distance and did not improve classification performance. Experiment 3 found that informing learners about the benefits of extensive interleaving, as compared to minimal interleaving or no interleaving, successfully increased interleaving distance and boosted classification performance, and the intervention effects transferred to learning categories in a different domain. These findings support the practical use of the instructional intervention in promoting self-usage of the interleaving strategy and highlight the important role of enlarging interleaving distance in facilitating inductive learning.

Retrieval practice is a powerful method for consolidating long-term learning. When learning takes place over an extended period, how should tests be scheduled to obtain the maximal benefit? In an end-test schedule, all material is studied prior to a large practice test on all studied material, whereas in an interim test schedule, learning is divided into multiple study/test cycles in which each test is smaller and only assesses material from the preceding study block. Past investigations have generally found a difference between these schedules during practice but not during a final assessment, although they may have been underpowered. Five experiments confirmed that final assessment performance was better in students taught using interim than end tests in list (Experiments 1, 2, and 5) and paired associate (Experiments 3 and 4) learning, with a meta-analysis of all available studies (k = 19) yielding a small- to medium-sized effect, g = 0.25, 95% confidence interval [0.09, 0.42]. Experiment 5 finds that the higher level of practice retrieval success in interim tests contributes to the grain size effect, but the effect is eliminated if these tests are too easy. Additional analyses also suggest that the forward testing effect, in which tests promote subsequent learning, may be a major cause of the grain size effect. The practical and theoretical implications of these demonstrations of robust grain size effects are discussed.

Cronbach (1957) famously noted the divergence between the experimental and psychometric traditions in psychology and called for a unification, but many domains of cognitive experimental psychology continue to pay minimal heed to basic psychometric principles. The present article considers the lack of attention devoted to the reliability of measures extracted in a popular visual search task for studying putatively unconscious mental processes, contextual cuing, and the inferential fallacies that this neglect can cause. Two experiments (total N = 200) demonstrated that the reliability of contextual cuing and awareness measures can be increased by three manipulations designed to increase between-participant variability in search performance. At the same time, the data were subjected to a multiverse analysis, which found that specific data preprocessing pipelines result in more reliable estimates. Nevertheless, the reliability estimates remained too low for drawing firm conclusions from standard statistical techniques. Interpreting results from analyses based on individual differences, such as the typical low correlations between implicit and explicit measures, will be challenging so long as the underlying measures have poor reliability.

A widely adopted approach in research on unconscious perception and cognition involves contrasting behavioral or neural responses to stimuli that have been presented to participants (e.g., old items in a memory test) against those that have not (e.g., new items), and which participants do not discriminate in their conscious reports. We demonstrate that such contrasts do not license inferences about unconscious processing, for two reasons. One is Kelley’s Paradox, a statistical phenomenon caused by regression to the mean. In the inevitable presence of measurement error, true awareness of the contrasted stimuli is not equal. The second is a consequence, within the framework of Signal Detection Theory, of unequal skewness in the strengths of target and nontarget items. The fallacious reasoning that underlies the employment of this contrast methodology is illustrated through both computational simulations and formal analysis, and its prevalence is documented in a narrative literature review. Additionally, a recognition memory experiment is reported which tests and confirms a prediction of our analysis of the contrast methodology and corroborates the susceptibility of this method to artifacts attributable to Kelley’s Paradox and strength skewness. This work challenges the validity of conclusions drawn from this popular analytic approach.

Gender bias has been documented in many aspects of Science, Technology, Engineering and Mathematics (STEM) careers, yet efforts to identify the underlying causes have been inconclusive. To what extent do cognitive biases, including unequal receptiveness in women and men to evidence of gender bias, contribute to gender bias in STEM? We investigated receptiveness in a STEM context among members of the general public, by undertaking a high-powered (total N = 1171) replication, including three experiments (2 pre-registered) of the prominent study by Handley et al. [22]. It was hypothesized that men would evaluate a research summary reporting evidence of gender bias less favourably than women but that there would be no difference between men and women’s evaluations of research summaries unrelated to gender bias. The results revealed no effect of the assessor’s gender on receptiveness to scientific evidence of gender bias. The different results compared to those of Handley et al. [22] suggest either that the gender bias they detected has diminished in the past decade or that their findings are a false positive. The present research adds to a growing body of evidence suggesting that some influential studies on cognitive ‘markers’ of gender bias warrant re-examination.