ArticlePDF Available

Burden of proof: combating inaccurate citation in biomedical literature

  • George Insitute for Global Health
Burden of proof: combating inaccurate citation in biomedical literature
Inappropriate, misleading, missing, and inaccurate citations pervade the biomedical literature. Nicholas
Peoples and colleagues argue that new strategies can better enable scientific references to function
as an accurate web of knowledge
Nicholas Peoples, 1Truls Østbye, 2Lijing L Yan3
Key messages
Up to 25% of all citations in the general scientific
literature are inaccurate and mislead physicians,
academics, and policy makers
The emergence of artificial intelligence (AI) powered
large language models such as ChatGPT has the
potential to both enable and mitigate inaccurate
citation on a scale not previously possible
Researchers need new strategies to ensure that
scientific references function as an accurate web of
We make the case that peer reviewed journals
consider adopting a required statement on the
integrity of cited literature, using the adoption of
required conflict of interest statements as a proof of
Even without a name, it is a devil we all know: an
article cites a source that does not support the
statement in question, or, more commonly, the initial
reference sends the reader down a rabbit hole of
references, the bottom of which is difficult to find
and interpret. This causes two problems. Firstly, it
may propagate data that are false, misinterpreted, or
both, spurring academic urban legendsthat
become circulated as truth.1This delays true results
from reaching the literature and allows incorrect
ideas to masquerade as facts. Second, it undermines
respect for the process of literature review, effacing
the foundation of good scientific inquiry into a mere
box ticking exercise. This cheapens the value of
background and discussion sections in scholarly
articles and encourages trainees and young
investigators to practise sloppy research.
These errors might be especially problematic for
doctors and the general public, who are not focused
on the scientific study of a narrow research topic and
thus are less prone to identify rhetorically misleading
statements or outright factual errors.2Leung and
colleagues, for example, document clear patterns of
inaccurate citation that misrepresent the conclusions
of a single paragraph statement in the New England
Journal of Medicine on the safety of opiate use.3 4 They
argue that these misrepresentations might have
contributed to the North American opioid crisis by
helping to shape a narrative that allayed prescribers
concerns about the risk of addiction associated with
long term opioid therapy.3
Recent estimates indicate citation error rates of 11-15%
in the biomedical literature2 5 and up to 25% in the
general science literature.6In a review of 4912
citations, 38.4% of these errors were citing
non-existent findings, 15.4% were incorrect
interpretations of findings, and 20% were chains of
inaccurate citations copied forward from paper to
paper.5This indicates that mis-citation is widespread.
A surgical study7was found to be misquoted by 40%
of the articles that cited it,8 creating an unsupported
but widely accepted guideline for how an orthopaedic
procedure should be performed. This shows that
mis-citation might also deeply mischaracterise
individual scientific works. Finally, to understand
how an entire scientific belief system might evolve,
a 2009 study systematically mapped out the full
citation chain for a particular scientific claim related
to amyloid β. Among its findings was the marked
expansion of the belief system by papers presenting
no data addressing it; and forms of invention such
as the conversion of hypothesis into fact through
citation alone. 9So, improper citation can even
credibly distort the scientific consensus.
Rekdal offers granular insight into this process in his
excellent analysis of the iron content in spinach
myth (fig 1).1In a 1981 article entitled Fake!,10 Terry
Hamblin believed he was debunking an erroneous
claim about the iron content of spinach but was
unknowingly using incorrect information himself.
Others then cited and transformed his ideas into even
greater inaccuracies.1 11 Rekdal convincingly makes
the case that even later authors, such as Larsson in
1995,12 borrowed the conclusions of articles that cited
Hamblin without actually consulting the 1981 paper
directly, further distorting the truth. In the end,
Hamblins accidental rumour was only debunked in
2010 (some 30 years later),13 and he tried in vain to
extinguish it to up until his death in 2012.1 14
1the bmj | BMJ 2023;383:e076441 | doi: 10.1136/bmj-2023-076441
1Baylor College of Medicine, Houston,
2Family Medicine and Community
Health, Duke University, Durham, NC,
3Global Health Research Center, Duke
Kunshan University, Kunshan, Jiangsu
Province, China
Correspondence to:
Cite this as:
Published: 06 November 2023
Library. Protected by copyright. on 6 November 2023 at Houston Academy of Medicine-Texas Medical Center first published as 10.1136/bmj-2023-076441 on 6 November 2023. Downloaded from
Fig 1 | Creation, propagation, and debunking of an academic urban legend
This all makes for a poor report card, especially given that erroneous
citation has been rampant since at least 1931 (and, judging by Shulls
exasperated remarks, with little progress in the time since).15 But
modern figures risk becoming underestimates: emerging artificial
intelligence (AI) powered large language models now enable
inaccurate citation with an efficiency and scale not previously
possible. On a broad scale, academia is clearly grappling with how
to reconcile technological progress and traditional research ethics
with concerns that AI may hallucinatesources or fabricate
data.16 17 So why do we turn a blind eye if such fabrications are
human, so long as they are neatly packaged into scholarly citations
that look and sound the part?18
Types of inadequate citation
Just as there is no universally accepted termmisquotation8versus
quotation errors,2erroneous citation15 versus inadequate citation,12
and so onthere is no universally accepted classification scheme.
Steven Greenberg developed a vocabulary for the citation
distortions he encountered in his study of amyloid β, which offers
one useful framework, but it primarily describes mechanisms and
consequences of select forms of poor citation without exploring
underlying causes, which also matter.9Others discuss common
citation errorsbut do not report the methodology for how they
arrived at these categories.19 We propose that rigorous, systematic
categorisation, a taxonomy and nomenclature of inadequate
citation, or both, are important next steps. Box 1 provides a concise
overview of some common documented types of mis-citation.
Box 1: Selected examples of inadequate citation practices
Biasedpreferentially citing certain sources, perhaps those of
colleagues, because the author is simply more familiar with them,
regardless of whether they are the best supporting works for the claim
at hand19
Coercion: pre-submissionsenior or principal investigator incorrectly
adds a reference to a work created by a trainee (such as a PhD
student), who does not feel empowered to decline or challenge this
Coercion: within submissionduring peer review, a reviewer or editor
instructs the author to cite publications co-authored by the reviewer
or editor20
Editinga work is initially cited correctly, but as the draft is edited
and sentences are eliminated or consolidated, perhaps by multiple
authors, a correctly cited study inadvertently becomes misquoted19
Hallucinatoryciting a source or conclusion that was convincingly
fabricated by artificial intelligence without confirming existence of
the source or veracity of the claims16 17
Lazy authoradding citations without reading the work in full (or
at all)21 -23
Misinterpretationwrongly interpreting the source2 19
Missingmaking a statement without citing the source19
Non-contextualciting a part of the conclusion, misrepresenting the
results9 19
Plagiarismrepresenting otherswork as ones own by omitting
Secondaryciting a secondary source (such as a review article) when
a primary source exists19 24
Self-citationinappropriately citing ones own work without sufficient
the bmj | BMJ 2023;383:e076441 | doi: 10.1136/bmj-2023-0764412
Library. Protected by copyright. on 6 November 2023 at Houston Academy of Medicine-Texas Medical Center first published as 10.1136/bmj-2023-076441 on 6 November 2023. Downloaded from
TemporalCiting a work that was read, but some time ago, such that
the author inaccurately recollects the findings25
Strategies to mitigate mis-citation
There are few checks and balances on erroneous citation. The first
line of defence (pre-submission quality control) is a standard
expectation of academic integrity. In the age of publish or perish,
however, competing interests, such as enormous pressure on
investigators to exhibit constant productivity, can make it tempting
to cut corners in pursuit of expediency.6 21 24 Market forces are in
tune to this. Consensus, for example, is a new, AI powered search
engine designed specifically for academics (https://consen- Given a search query, it will produce a list of 5-10
peer reviewed papers with a short synopsis of each. The more widely
used ChatGPT has been found to fabricate both facts and legitimate
sounding scholarly sourcesto provide a convincingrather than
factualanswer to a prompt.16 17 Although these tools are
multi-purpose, powerful, and certainly promising adjuvants for
literature review, they also exponentially enable the ease and scale
of producing inaccurate citations.
The second line of defence is peer review, which currently serves
as the major safety net to catch mis-citations after submission.
History has shown, however, that this can also be inefficient,
inconsistent, and insufficient.6 26 -28 (And, as others point out, peer
review may even encourage mis-citation.)20 Moreover, the ultimate
responsibility for auditing cited literature should not rest with peer
reviewers, but with those who selected the literature to support
their claims.
The third line of defence is post-publication review, which is often
either inordinately difficult29 30 or even directly opposed.31 Although
non-replicable primary results have been caught and overturned
after publication, there is little drive to do the same for inaccurate
citations. This effectively allows them to live on into eternity.
Although we are not the first to highlight these problems,15 32 we
argue that prevailing strategies are insufficient.
New tools in our toolkit
Just as AI might exacerbate erroneous citation, we suggest it could
also be part of the solution. It might, for example, eventually be
possible for AI programmes to be integrated into manuscript
submission portals to confirm that all cited works actually exist.
Even more useful, however, will be AI modalities developed to
assess citations for accuracy and to flag potential discrepancies for
review. This would not replace human review, but something with
reasonable sensitivity and high specificity could provide an initial
screen that alerts reviewers to instances of potential mis-citation
without creating additional work.
Another idea is for journals to ask authors to attest that an article
contains no inaccurate citations (box 2), much in the same way that
they are expected to make a statement about conflicts of interest.
Box 2: Example works cited statement
The author(s) certify that all works cited were read in full by at least one
author at the time of writing this manuscript; are necessary to support
the intellectual foundation of the work; were added without undue
coercion; and do not reflect inappropriate self-citation. We affirm that
this document cites primary sources whenever possible, transparently
discloses the source of all secondary information presented, and
accurately represents both the objective findings and earnest spirit of
all works cited.
These ideals are implicit in any submission to a peer reviewed
journal, so making them explicit presents no addition burden. The
key question, then, is whether such declarations are useful. To
answer this, we can compare older to more recent literature as
journals have progressively adopted more rigorous standards.
Specifically, conflict of interest (COI) statements are a useful analogy
to our proposed works cited statement, because when there are no
competing interests to declare, they function as an attestation. These
statements became increasingly common as concerns grew about
the influence of money and corporate interests in research. Estimates
for the proportion of journals requiring a COI statement vary by
discipline but consistently show a positive trend. Some commonly
cited examples include 16% (220 of 1367 highly ranked scientific
and biomedical journals) in 1997,33 33% (28 of 84 journals from 12
scientific disciplines) in 2007,34 89.7% (358 of 399 high impact
biomedical journals) in 2011,35 and 96% (224 of 227 public and
occupational health journals) in 2016.36 It comes as no surprise,
then, that none of 47 trials on febrile neutropenia published from
1981 to 2000 contained a COI statement.37 (Additionally, only 29
reported that informed consent was obtained and only 22 reported
approval of a research ethics committee.) To see how this compares
with more recent literature, we reviewed a random sample of 100
research papers published in 2022 in the New England Journal of
Medicine, the Journal of the American Medical Association, and The
BMJ, finding that 100% of articles included COI statements (and
likewise, statements on institutional review board approval and
informed consent, when applicable) (see supplementary file on
A COI statement encourages transparency because inadequate
disclosure carries consequences. Similarly, a works cited
statement(box 2) might encourage diligence as it implies that
references will be scrutinised, and mis-citation will be penalised.
Assuming that most authors operate on good intent (striving for
their work to be accurate) or self-interest (striving to avoid delays
in publication), or both, the act of formal attestation might inspire
greater diligence in crafting a reference list or double checking it
for accuracy before submission. Even with inadequate disclosure,
however, statements still offer important functionality. A 2014 phase
3 trial, for example, was publicly redacted when it was discovered
that the authors lied in their COI statement.38 Here, the COI
statement created a clear mismatch between the authorswords
and deeds. This alerted a vigilant reader, provided the journal with
irrefutable justification for corrective action, and, ultimately, halted
the dissemination of untrustworthy information. Similarly, a works
cited statement found to have inadequate or inaccurate disclosure
might raise a red flag and lead reviewers or readers to scrutinise a
work more closely, potentially to catch other important flaws. If the
authors formal assurances for something as basic as a reference
list cannot be taken at face value, what else might they have been
mistaken or misleading about?
COI statements enable new inquiries into research bias. Landmark
studies have shown, for example, a strong association between
pharmaceutical industry funding and likelihood of reporting
significant results.39 40 Another study analysed 767 clinical trials
and found a strong association between failure to disclose informed
consent and poor methodological quality.41 A works cited statement,
then, would enable researchers to ask similar questions about
inaccurate citations, such as whether studies with an inaccurate
works cited statement are associated with poor methodological
quality or inflated results. Fact checking citation accuracy can be
labour intensive, but there is a critical mass of people who do this
sort of work already, reflected in the growing literature on
3the bmj | BMJ 2023;383:e076441 | doi: 10.1136/bmj-2023-076441
Library. Protected by copyright. on 6 November 2023 at Houston Academy of Medicine-Texas Medical Center first published as 10.1136/bmj-2023-076441 on 6 November 2023. Downloaded from
So, declarations do seem to have beneficial value. They are not a
perfect, systemic failsafe, but they do provide authors, editors, peer
reviewers, and journal readers with additional opportunities to
promote essential quality control. Similar cases can be made for
the required declaration of institutional review board approval,
author contributions, informed consent, and emerging declarations
of the role of AI. As thousands of erroneous citations now raise
legitimate questions about inaccuracies in published research, there
is a clear case for adopting similar safeguards in this arena as well.
To make such a declaration meaningful, however, it must be
As a final strategy, we propose that journals add two internal
questions for reviewers: were any improper citations noted during
the review of this paper? And did you recommend the authors to
cite any studies in which you are a coauthor or otherwise have a
vested interest? If the answer to either question is yes, reviewers
must provide specific details, which are sent to the editor. We
propose that a submitted study with erroneous citations should be
withdrawn from consideration if the errors are pervasive,
mischaracterise the background or methods, inappropriately shape
interpretation of the results, or otherwise betray a serious lack of
expected due diligence. For manuscripts with less egregious
mis-citation, we recommend that reviewers and editors still adopt
a low threshold to mandate revisions, as there is no good reason
to allow . . . inexact and non-verifiable referencing to pervade
scientific literature.6We further propose that an editor should
closely scrutinise situations in which a reviewer has recommended
citation of their own works.20 If already published when the errors
are caught, authors should be asked to amend the work (without
additional fees29), which is eminently possible in the digital era.
Critically, we extend investigators the benefit of the doubt: many
inaccurate citations are simply the result of honest mistakes. Thus,
this higher standard is primarily meant to deter carelessness and
promote good practice. As is the case broadly in medicine, primary
prevention is often the best policy.
Looking ahead
Mis-citation has heretofore been inadequately tackled. By
acknowledging the fault lines in current practice, prioritising the
development of a rigorous and standardised classification scheme
for inadequate citation, and codifying accurate literature review
into a routine pre-submission declaration, we can strive to better
enshrine integrity into medical scholarship. We encourage journals
to consider adopting a pre-submission declaration. It has the
potential to deter inappropriate manuscript submissions and
facilitate correction after publication, with little added cost or
inconvenience. Over time, the higher standard might also instil
greater respect among the next generation of young investigators
for this fundamental pillar of scientific inquiry. Most importantly,
if it yields improvement in overall citation quality, this will help the
scientific literature better function as an accurate web of knowledge.
Works cited statement: The authors certify that all works cited were read in full by at least one author
at the time of writing this manuscript; are necessary to support the intellectual foundation of the work;
were added without undue coercion; and do not reflect inappropriate self-citation. We affirm that this
document cites primary sources whenever possible, transparently discloses the source of all secondary
information presented, and accurately represents both the objective findings and earnest spirit of all
works cited.
Funding: This work received no funding or financial support of any kind.
Contributors and sources: This work was conceived by NP, who wrote the first draft and acts as the
guarantor for this work. LLY and TO provided critical review that helped shape the key intellectual
output of this work. NP holds an MSc from Duke University in the US and is currently a 4th year MD
student at Baylor College of Medicine and a Schwarzman Scholar at Tsinghua University. LLY is head
of non-communicable disease research and tenured faculty at Duke Kunshan University in China. She
has published over 150 studies in global health, many involving complex multi-country randomised
controlled trials. TO is a physician and epidemiologist. He has published over 690 peer reviewed studies
and holds professorships at Duke University, Duke Kunshan University, and Duke-NUS University in
Singapore. This work is the product of three experienced, professionally and culturally diverse
researchers who disdain corner cutting in research and want to advocate for integrity and quality in
the biomedical sciences.
Patient involvement: No patients were involved in the creation of this manuscript.
Competing interests: We have read and understood BMJ policy on declaration of interests and have
no interests to declare.
We thank Vianna Quach, Dianne Wade, and Alexandra Alvarez for expert editorial feedback on early
drafts of this work.
1Rekdal OB. Academic urban legends.
Soc Stud Sci
doi: 10.1177/0306312714535679 pmid: 25272616
2Mogull SA. Accuracy of cited factsin medical research articles: A review of study methodology
and recalculation of quotation error rate.
PLoS One
doi: 10.1371/journal.pone.0184727 pmid: 28910404
3Leung PTM, Macdonald EM, Stanbrook MB, Dhalla IA, Juurlink DNA. 1980 letter on the risk of
opioid addiction.
N Engl J Med
2017;376:-5. doi: 10.1056/NEJMc1700150 pmid: 28564561
4Porter J, Jick H. Addiction rare in patients treated with narcotics.
N Engl J Med
doi: 10.1056/NEJM198001103020221 pmid: 7350425
5Pavlovic V, Weissgerber T, Stanisavljevic D, etal. How accurate are citations of frequently cited
papers in biomedical literature?
Clin Sci (Lond)
doi: 10.1042/CS20201573 pmid: 33599711
6Smith N, JrCumberledge A. Quotation errors in general science journals.
Proc R Soc Lond A
7Knirk JL, Jupiter JB. Intra-articular fractures of the distal end of the radius in young adults.
J Bone
Joint Surg Am
1986;68:-59. doi: 10.2106/00004623-198668050-00003 pmid: 3722221
8Porrino JA, JrTan V, Daluiski A. Misquotation of a commonly referenced hand surgery study.
Hand Surg Am
2008;33:-7. doi: 10.1016/j.jhsa.2007.10.007 pmid: 18261657
9Greenberg SA.How citation distortions create unfounded authority: analysis of a citation network.
2009;339:. doi: 10.1136/bmj.b2680 pmid: 19622839
10 Hamblin TJ. Fake.
Br Med J (Clin Res Ed)
doi: 10.1136/bmj.283.6307.1671 pmid: 6797607
11 Skrabanek P, McCormick J. Follies and Fallacies in Medicine, 3rd edn. Tarragon Press, 1998.
12 Larsson KS. The dissemination of false data through inadequate citation.
J Intern Med
1995;238:-50. doi: 10.1111/j.1365-2796.1995.tb01222.x pmid: 7595184
13 Sutton M. How the spinach, Popeye and iron decimal point error myth was finally busted.
14 Hamblin TJ. SpinachI was right for the wrong reason. Mutations of Mortality. 2010. http://mu-
15 Shull CA. Erroneous citations and titles of scientific papers.
16 Alkaissi H, McFarlane SI. Artificial hallucinations in ChatGPT: implications in scientific writing.
2023;15:e35179. doi: 10.7759/cureus.35179 pmid: 36811129
17 Bohannon M. Lawyer used ChatGPT in court-and cited fake cases. A judge is considering sanctions.
Forbes Magazine. 2023.
18 Ingelfinger FJ. Seduction by citation.
N Engl J Med
doi: 10.1056/NEJM197611042951911. pmid: 972662
19 Agarwal A, Arafa M, Avidor-Reiss T, Hamoda TAA, Shah R. Citation errors in scientific research
and publications: causes, consequences, and remedies.
World J Mens Health
doi: 10.5534/wjmh.230001 pmid: 37118953
20 Wilhite AW, Fong EA. Scientific publications. Coercive citation in academic publishing.
2012;335:-3. doi: 10.1126/science.1212540 pmid: 22301307
21 Gavras H. Inappropriate attribution: the lazy author syndrome.
Am J Hypertens
doi: 10.1016/S0895-7061(02)02989-8 pmid: 12722775
22 Simkin MV,Roychowdhury VP. Stochastic modeling of citation slips.
doi: 10.1007/s11192-005-0028-2.
23 Simkin MV, Roychowdhury VP. Read before you cite!
Complex Syst
24 Teixeira MC,Thomaz SM, Michelan TS, etal. Incorrect citations give unfair credit to review authors
in ecology journals.
PLoS One
2013;8:e81871. doi: 10.1371/journal.pone.0081871 pmid: 24349143
25 Roig M. Avoiding plagiarism, self-plagiarism, and other questionable writing practices: a guide to
ethical writing. Office of Research Integrity. 2015.
26 Smith R. Peer review: a flawed process at the heart of science and journals.
J R Soc Med
2006;99:-82. doi: 10.1177/014107680609900414 pmid: 16574968
27 Schroter S, Black N, Evans S, Carpenter J, Godlee F, Smith R. Effects of training on quality of
peer review: randomised controlled trial.
doi: 10.1136/bmj.38023.700775.AE pmid: 14996698
28 Godlee F, Gale CR, Martyn CN. Effect on the quality of peer review of blinding reviewers and
asking them to sign their reports: a randomized controlled trial.
doi: 10.1001/jama.280.3.237 pmid: 9676667
the bmj | BMJ 2023;383:e076441 | doi: 10.1136/bmj-2023-0764414
Library. Protected by copyright. on 6 November 2023 at Houston Academy of Medicine-Texas Medical Center first published as 10.1136/bmj-2023-076441 on 6 November 2023. Downloaded from
29 Allison DB, Brown AW, George BJ, Kaiser KA. Reproducibility: a tragedy of errors.
2016;530:-9. doi: 10.1038/530027a pmid: 26842041
30 Hosseini M, Eve MP, Gordijn B, Neylon C. MyCites: a proposal to mark and report inaccurate
citations in scholarly publications.
Res Integr Peer Rev
doi: 10.1186/s41073-020-00099-8 pmid: 32968546
31 Cooper DJ,Gangadharan L, Noussair C. Editorsnote regarding citations of other work.
Exp Econ
2017;20:-7. doi: 10.1007/s10683-016-9497-9.
32 de Lacey G,Record C, Wade J. How accurate are quotations and references in medical journals?
Med J (Clin Res Ed)
1985;291:-6. doi: 10.1136/bmj.291.6499.884. pmid: 3931753
33 Krimsky S, Rothenberg LS. Conflict of interest policies in science and medical journals: editorial
practices and author disclosures.
Sci Eng Ethics
doi: 10.1007/s11948-001-0041-7 pmid: 11349360
34 Ancker JS, Flanagin A. A comparison of conflict of interest policies at peer-reviewed journals in
different scientific disciplines.
Sci Eng Ethics
doi: 10.1007/s11948-007-9011-z pmid: 17717729
35 Bosch X,Pericas JM, Hernández C, Doti P. Financial, nonfinancial and editorsconflicts of interest
in high-impact biomedical journals.
Eur J Clin Invest
doi: 10.1111/eci.12090 pmid: 23550719
36 Resnik DB, Konecny B, Kissling GE. Conflict of interest and funding disclosure policies of
environmental, occupational, and public health journals.
J Occup Environ Med
doi: 10.1097/JOM.0000000000000910 pmid: 28045794
37 Weingarten MA, Paul M, Leibovici L. Assessing ethics of trials in systematic reviews.
2004;328:-4. doi: 10.1136/bmj.328.7446.1013 pmid: 15105330
38 Notice of Retraction: Pak CS, et al. A Phase III, Randomized, Double-Blind, Matched-Pairs,
Active-Controlled Clinical Trial and Preclinical Animal Study to Compare the Durability, Efficacy
and Safety between Polynucleotide Filler and Hyaluronic Acid Filler in the Correction of Crows
Feet: A New Concept of Regenerative Filler. J Korean Med Sci 2014; 29(Suppl 3): S201-S209.
Korean Med Sci
2016;31:-330. doi: 10.3346/jkms.2016.31.2.330. pmid: 26839493
39 Lexchin J, Bero LA, Djulbegovic B, Clark O. Pharmaceutical industry sponsorship and research
outcome and quality: systematic review.
doi: 10.1136/bmj.326.7400.1167 pmid: 12775614
40 Friedman LS, Richter ED. Relationship between conflicts of interest and research results.
J Gen
Intern Med
2004;19:-6. doi: 10.1111/j.1525-1497.2004.30617.x pmid: 14748860
41 Ruiz-Canela M,de Irala-Estevez J, Martínez-González MA, Gómez-Gracia E, Fernández-Crehuet J.
Methodological quality and reporting of ethical requirements in clinical trials.
J Med Ethics
2001;27:-6. doi: 10.1136/jme.27.3.172 pmid: 11417024
42 Travaglia J, Hinchcliff R, Carter D, Billington L, Glennie M, Debono D. Attestation by governing
bodies: literature review. Australian Commission on Safety and Quality in Health Care.
5the bmj | BMJ 2023;383:e076441 | doi: 10.1136/bmj-2023-076441
Library. Protected by copyright. on 6 November 2023 at Houston Academy of Medicine-Texas Medical Center first published as 10.1136/bmj-2023-076441 on 6 November 2023. Downloaded from
Full-text available
While still in its infancy, ChatGPT (Generative Pretrained Transformer), introduced in November 2022, is bound to hugely impact many industries, including healthcare, medical education, biomedical research, and scientific writing. Implications of ChatGPT, that new chatbot introduced by OpenAI on academic writing, is largely unknown. In response to the Journal of Medical Science (Cureus) Turing Test - call for case reports written with the assistance of ChatGPT, we present two cases one of homocystinuria-associated osteoporosis, and the other is on late-onset Pompe disease (LOPD), a rare metabolic disorder. We tested ChatGPT to write about the pathogenesis of these conditions. We documented the positive, negative, and rather troubling aspects of our newly introduced chatbot's performance.
Full-text available
Citations are an important, but often overlooked, part of every scientific paper. They allow the reader to trace the flow of evidence, serving as a gateway to relevant literature. Most scientists are aware of citations errors, but few appreciate the prevalence of these problems. The purpose of this study was to examine how often frequently cited papers in biomedical scientific literature are cited inaccurately. The study included an active participation of first authors of included papers; to first-hand verify the citations accuracy. Findings from feasibility study, where we reviewed 1,540 articles containing 2,526 citations of 14 most cited articles in which the authors were affiliated with the Faculty of Medicine University of Belgrade, were further evaluated for external confirmation in an independent verification set of articles. Verification set included 4,912 citations identified in 2,995 articles that cited 13 most cited articles published by authors affiliated with the Mayo Clinic Division of Nephrology and Hypertension. A citation was defined as being accurate if the cited article supported or was in accordance with the statement by citing authors. At least one inaccurate citation was found in 11% and 15% of articles in the feasibility study and verification set, respectively, suggesting that inaccurate citations are common in biomedical literature. The most common problem was the citation of nonexistent findings (38.4%), followed by an incorrect interpretation of findings (15.4%). One fifth of inaccurate citations were due to chains of inaccurate citations. Based on these findings, several actions to reduce citation inaccuracies have been proposed.
Full-text available
Due to the incremental nature of scientific discovery, scientific writing requires extensive referencing to the writings of others. The accuracy of this referencing is vital, yet errors do occur. These errors are called ‘quotation errors’. This paper presents the first assessment of quotation errors in high-impact general science journals. A total of 250 random citations were examined. The propositions being cited were compared with the referenced materials to verify whether the propositions could be substantiated by those materials. The study found a total error rate of 25%. This result tracks well with error rates found in similar studies in other academic fields. Additionally, several suggestions are offered that may help to decrease these errors and make similar studies more feasible in the future.
Full-text available
Background: Inaccurate citations are erroneous quotations or instances of paraphrasing of previously published material that mislead readers about the claims of the cited source. They are often unaddressed due to underreporting, the inability of peer reviewers and editors to detect them, and editors' reluctance to publish corrections about them. In this paper, we propose a new tool that could be used to tackle their circulation. Methods: We provide a review of available data about inaccurate citations and analytically explore current ways of reporting and dealing with these inaccuracies. Consequently, we make a distinction between publication (i.e., first occurrence) and circulation (i.e., reuse) of inaccurate citations. Sloppy reading of published items, literature ambiguity and insufficient quality control in the editorial process are identified as factors that contribute to the publication of inaccurate citations. However, reiteration or copy-pasting without checking the validity of citations, paralleled with lack of resources/motivation to report/correct inaccurate citations contribute to their circulation. Results and discussion: We propose the development of an online annotation tool called "MyCites" as means with which to mark and map inaccurate citations. This tool allows ORCID users to annotate citations and alert authors (of the cited and citing articles) and also editors of journals where inaccurate citations are published. Each marked citation would travel with the digital version of the document (persistent identifiers) and be visible on websites that host peer-reviewed articles (journals' websites, Pubmed, etc.). In the future development of MyCites, challenges such as the conditions of correct/incorrect-ness and parties that should adjudicate that, and, the issue of dealing with incorrect reports need to be addressed.
Full-text available
Previous reviews estimated that approximately 20 to 25% of assertions cited from original research articles, or “facts,” are inaccurately quoted in the medical literature. These reviews noted that the original studies were dissimilar and only began to compare the methods of the original studies. The aim of this review is to examine the methods of the original studies and provide a more specific rate of incorrectly cited assertions, or quotation errors, in original research articles published in medical journals. Additionally, the estimate of quotation errors calculated here is based on the ratio of quotation errors to quotations examined (a percent) rather than the more prevalent and weighted metric of quotation errors to the references selected. Overall, this resulted in a lower estimate of the quotation error rate in original medical research articles. A total of 15 studies met the criteria for inclusion in the primary quantitative analysis. Quotation errors were divided into two categories: content ("factual") or source (improper indirect citation) errors. Content errors were further subdivided into major and minor errors depending on the degree that the assertion differed from the original source. The rate of quotation errors recalculated here is 14.5% (10.5% to 18.6% at a 95% confidence interval). These content errors are predominantly, 64.8% (56.1% to 73.5% at a 95% confidence interval), major errors or cited assertions in which the referenced source either fails to substantiate, is unrelated to, or contradicts the assertion. Minor errors, which are an oversimplification, overgeneralization, or trivial inaccuracies, are 35.2% (26.5% to 43.9% at a 95% confidence interval). Additionally, improper secondary (or indirect) citations, which are distinguished from calculations of quotation accuracy, occur at a rate of 10.4% (3.4% to 17.5% at a 95% confidence interval).
Full-text available
Objective: The aim of this study was to analyze conflict of interest (COI) and funding disclosure policies of 224 journals listed in Journal Citation Reports as focusing on environmental, occupational, or public health research. Methods: A survey of journal policies and content analysis. Results: About 96.0% of the policies required COI disclosure, 92.4% required funding disclosure, 75.9% defined COIs, 69.6% provided examples of COIs, 68.8% addressed nonfinancial COIs, 33.9% applied to editors and reviewers, 32.1% required discussion of the role of the funding source, and 1.8% included enforcement mechanisms. Policies were significantly associated with journal impact factor and publisher. Conclusion: Although a high percentage of journals in our sample have COI policies that provide substantial guidance to authors, there is a room for improvement. Journals that have not done so should consider developing enforcement mechanisms and applying COI policies to editors and reviewers.