No file available
To read the file of this research,
you can request a copy directly from the authors.
Who, What, Where: Tracking the development of COVID-19 related PsyArXiv preprints
Preprints and early-stage research may not have been peer reviewed yet.
Abstract
Given the need for a rapid and supposed critical response from behavioural sciences during times of crisis, this study aimed to track the development of COVID-19 psychology-related preprints. We tracked the first 211 COVID related preprints on the repository PsyArXiv. Specifically, we tracked who was submitting preprints, what the preprints were investigating, and whether the preprints lead to publications and their impact (measured by Google Scholar citations). We then followed up with the preprints about a year later to determine the number of preprints that lead to publication and the number of citations they received. The results showed that males from western countries submitted most preprints. Fifty-one per cent of preprints used a survey design, and the most common topic for covid-19 related preprints was mental health. Eighty-three per cent of preprints did not meet credible open science measures. 54% of the sampled preprints had been published in peer-reviewed journals, with a median time between preprint upload and publication of 105 days. Metascience preprints were more likely to be published, and preprints with reviews had lower citation rates. Overall, the results demonstrate that some of the structural problems in research are still in play despite global efforts to mobilise research efforts during the pandemic.
... Since the start of the pandemic, the number of preprints on COVID-19 has been steadily rising, with over 55,000 preprints to date (as of June 27, 2022; see also Figure 1). In the early stages of the pandemic, studies have shown that COVID-19 preprints were typically less well-written in terms of readability and spelling correctness (17), and that most did not meet standards for reproducibility and research integrity (18)(19)(20). A number of preprints also contained extremely serious issues, such as ethical and privacy concerns, data manipulation, and flawed designs (21). ...
... Alternatively, the authors' decisions could be the cause, as some may avoid the publication process altogether (31). Others may intentionally use preprints to release replications, and null results that are difficult to publish (27), or works in progress which may be less well-written and inadequate at sharing data/code (17)(18)(19)(20). Many preprints actually report their results in a balanced way so as not to 'oversell' their findings (31), and there is growing evidence of high concordance between findings published in preprints and in peer-reviewed journals (25,(32)(33)(34)(35)(36)(37)(38)(39). ...
Background : The quality of COVID-19 preprints should be considered with great care, as their contents can influence public policy. Efforts to improve preprint quality have mostly focused on introducing quick peer review, but surprisingly little has been done to calibrate the public’s evaluation of preprints and their contents. The PRECHECK project aimed to generate a tool to teach and guide scientifically literate non-experts to critically evaluate preprints, on COVID-19 and beyond.
Methods : To create a checklist, we applied a four-step procedure consisting of an initial internal review, an external review by a pool of experts (methodologists, meta-researchers/experts on preprints, journal editors, and science journalists), a final internal review, and an implementation stage. For the external review step, experts rated the relevance of each element of the checklist on five-point Likert scales, and provided written feedback. After each internal review round, we applied the checklist on a set of high-quality preprints from an online list of milestone research works on COVID-19 and low-quality preprints, which were eventually retracted, to verify whether the checklist can discriminate between the two categories.
Results : At the external review step, 26 of the 54 contacted experts responded. The final checklist contained four elements (Research question, study type, transparency and integrity, and limitations), with ‘superficial’ and ‘deep’ levels for evaluation. When using both levels of evaluation, the checklist was effective at discriminating high- from low-quality preprints. Its usability was confirmed in workshops with our target audience: Bachelors students in Psychology and Medicine, and science journalists.
Conclusions : We created a simple, easy-to-use tool for helping scientifically literate non-experts navigate preprints with a critical mind. We believe that our checklist has great potential to help guide decisions about the quality of preprints on COVID-19 in our target audience and that this extends beyond COVID-19.
ResearchGate has not been able to resolve any references for this publication.