Avoidable Waste in the Production and Reporting of Research Evidence

Abstract

presented,in meeting abstracts: “Research results should be easily accessible to people

Full text

Viewpoint
86
www.thelancet.com Vol 374 July 4, 2009
Avoidable waste in the production and reporting of
research evidence
Iain Chalmers, Paul Glasziou
Without accessible and usable reports, research cannot
help patients and their clinicians. In a published
Personal View,1 a medical researcher with myeloma
refl ected on the way that the results of four randomised
trials relevant to his condition had still not been
published, years after preliminary fi ndings had been
presented in meeting abstracts:
“Research results should be easily accessible to people
who need to make decisions about their own health…
Why was I forced to make my decision knowing that
information was somewhere but not available? Was
the delay because the results were less exciting than
expected? Or because in the evolving fi eld of myeloma
research there are now new exciting hypotheses (or
drugs) to look at? How far can we tolerate the butterfl y
behaviour of researchers, moving on to the next fl ower
well before the previous one has been fully
exploited?”
This experience is not unusual: a recently updated
systematic review of 79 follow-up studies of research
reported in abstracts estimated the rate of publication of
full reports after 9 years to be only 53%.2
Worldwide, over US$100 billion is invested every year
in supporting biomedical research, which results in an
estimated 1 million research publications per year.
Much of this investment has supported basic research.
For example, over two-thirds of government and
charitable investment in biomedical research in the UK
has been for basic research, with less than 10% for
treatment evaluation. The relative lack of support for
applied research and the bureaucracy that regulates
research involving patients have been powerful
disincentives for those who might otherwise have
become involved in research in treatment evaluation. In
recent years, there has been recognition of the need to
address both of these disincentives. In the UK, the
Cooksey enquiry concluded that government support
for applied research should be increased,3 and the
National Institute for Health Research (NIHR) has
responded rapidly to this policy (its funding for clinical
trials will soon be £80 million a year).4 In the USA, a bill
currently before Congress calls for federal support for
evaluations of treatments independent of industry, and
in Italy and Spain, independent research on the eff ects
of drugs is being supported with revenue from a tax on
pharmaceutical company drug promotion.5
This increased investment in independent treatment
evaluation is laudable. Irrespective of who sponsors
research, this investment should be protected from the
avoidable waste of inadequately producing and
reporting research. We examine the causes and degree
of waste occurring at four successive stages: the choice
of research questions; the quality of research design
and methods; the adequacy of publication practices;
and the quality of reports of research (fi gure).
Choosing the wrong questions for research
An effi cient system of research should address health
problems of importance to populations and the
interventions and outcomes considered important by
patients and clinicians. However, public funding of
research is correlated only modestly with disease
burden, if at all.6–8 Within specifi c health problems there
is little research on the extent to which questions
addressed by researchers match questions of relevance
to patients and clinicians. In an analysis of 334 studies,
only nine compared researchers’ priorities with those
of patients or clinicians.9 The fi ndings of these studies
have revealed some dramatic mismatches. For example,
the research priorities of patients with osteoarthritis of
the knee and the clinicians looking after them favoured
more rigorous evaluation of physiotherapy and surgery,
and assessment of educational and coping strategies.
Only 9% of patients wanted more research on drugs,
yet over 80% of randomised controlled trials in patients
with osteoarthritis of the knee were drug evaluations.10
This interest in non-drug interventions in users of
research results is refl ected in the fact that the vast
majority of the most frequently consulted Cochrane
reviews are about non-drug forms of treatment. The
current emphasis on drugs is not simply a feature of
commercial research: controlled trials funded by the UK
Lancet 2009; 374: 86–89
Published Online
June 15, 2009
DOI:10.1016/S0140-
6736(09)60329-9
James Lind Library, James Lind
Initiative, Oxford, UK
(Sir I Chalmers DSc); and Centre
for Evidence-Based Medicine,
Department of Primary Care,
University of Oxford, Oxford,
UK (Prof P Glasziou RACGP)
Correspondence to:
Sir Iain Chalmers, James Lind
Library, James Lind Initiative,
Summertown Pavilion, Middle
Way, Oxford OX2 7LG, UK
ichalmers@jameslindlibrary.org
Low priority questions
addressed
Important outcomes
not assessed
Clinicians and
patients not involved
in setting research
agendas
Questions relevant
to clinicians and
patients?
Over 50% of studies
designed without
reference to
systematic reviews of
existing evidence
Over 50% of studies
fail to take adequate
steps to reduce
biases—eg,
unconcealed
treatment allocation
Appropriate design
and methods?
Over 50% of studies
never published in full
Biased under-
reporting of studies
with disappointing
results
Accessible
full publication?
Over 30% of trial
interventions not
sufficiently described
Over 50% of planned
study outcomes not
reported
Most new research
not interpreted in the
context of systematic
assessment of other
relevant evidence
Unbiased and
usable report?
Research waste
Figure: Stages of waste in the production and reporting of research evidence relevant to clinicians and patients

Viewpoint
www.thelancet.com Vol 374 July 4, 2009
87
Medical Research Council and British medical research
charities between 1980 and 2002, for example, were
substantially more likely to be drug trials than were trials
commissioned by the National Health Service Research
and Development Programme (now the NIHR).11
Furthermore, the outcomes that researchers have
measured have not always been those that patients regard
as most relevant. The exemplary involvement of patients
by researchers assessing treatments for rheumatoid
arthritis showed that, for most patients, fatigue was the
dominant symptom of concern, rather than pain, as
researchers had assumed.12
Many researchers remain dismissive of suggestions that
patients, carers, and clinicians should help to prioritise
research, but there are some signs of change. For example,
the James Lind Alliance has been established to bring
together patients, carers, and clinicians to prioritise
unresolved questions about the eff ects of treatments. In
the case of asthma, unaddressed uncertainties about
possible adverse eff ects of long-term use of steroids and
other powerful treatments emerged as their principal
shared concern.
Doing studies that are unnecessary, or poorly
designed
New research should not be done unless, at the time it
is initiated, the questions it proposes to address cannot
be answered satisfactorily with existing evidence. Many
researchers do not do this—for example, Cooper and
colleagues13 found that only 11 of 24 responding authors
of trial reports that had been added to existing systematic
reviews were even aware of the relevant reviews when
they designed their new studies. About 2500 systematic
reviews of research are now being published every year,
with roughly a quarter of them in the Cochrane Database
of Systematic Reviews. Systematic reviews are now the
most frequently cited form of clinical research14 (the
citation frequency of the Cochrane Database of Systematic
Reviews ranks seventh among general medical
publications), but there is still a long way to go before
we will know what number and proportion of the many
questions of importance to patients and clinicians can
be answered with systematic reviews of existing
evidence. It has been estimated that at least
10 000 systematic reviews would be required to cover the
issues that have been addressed in over half a million
reports of controlled trials.15
New research is also too often wasteful because of
inadequate attention to other important elements of
study design or conduct. For example, in a sample of
234 clinical trials reported in the major general medical
journals, concealment of treatment allocation was often
inadequate (18%) or unclear (26%).16 In an assessment of
487 primary studies of diagnostic accuracy, 20% used
diff erent reference standards for positive and negative
tests, thus overestimating accuracy, and only 17% used
double-blind reading of tests.17
Failure to publish relevant research promptly,
or at all
Biased under-publication and over-publication of research
are forms of unscientifi c and unethical misconduct about
which the public has become increasingly aware,
particularly because of several exposés of suppressed
evidence about serious adverse eff ects of treatments.18
More generally, studies with results that are disappointing
are less likely to be published promptly,19 more likely to
be published in grey literature, and less likely to proceed
from abstracts to full reports.2 The problem of biased
under-reporting of research results mainly from decisions
taken by research sponsors and researchers, not from
journal editors rejecting submitted reports.20
Over the past decade, biased under-reporting and
over-reporting of research have been increasingly
acknowledged as unacceptable, both on scientifi c and on
ethical grounds. Calls for prospective, public registration
of all clinical trials have been issued by infl uential
organisations—eg, WHO and the World Medical
Association (through the latest revision of the Declaration
of Helsinki) and the International Committee of Medical
Journal Editors—and some progress has been made.
WHO’s International Clinical Trials Registry Platform
has been developed to improve transparency and social
involvement in research, and there has been progress,
especially in the USA, where publication of the results of
research is now required. Although these developments
are welcome, public access to the full results of all
research remains an aspiration, and one that continues
to be resisted by some research sponsors and
researchers.
Biased or unusable reports of research
Although their quality has improved, reports of research
remain much less useful than they should be. Sometimes
this is because of frankly biased reporting—eg, adverse
eff ects of treatments are suppressed, the choice of
primary outcomes is changed between trial protocol and
trial reports,21 and the way data are presented does not
allow comparisons with other, related studies. But even
when trial reports are free of such biases, there are many
respects in which reports could be made more useful to
clinicians, patients, and researchers. We select here just
two of these.
First, if clinicians are to be expected to implement
treatments that have been shown in research to be
useful, they need adequate descriptions of the
interventions assessed, especially when these are
non-drug interventions, such as setting up a stroke unit,
off ering a low fat diet, or giving smoking cessation
advice. Adequate information on interventions is
available in around 60% of reports of clinical trials;22 yet,
by checking references, contacting authors, and doing
additional searches, it is possible to increase to 90% the
proportion of trials for which adequate information
could be made available.22
For more on the
James Lind Alliance see
http://www.lindalliance.org
For more on WHO’s
International Clinical Trials
Registry Platform see
http://www.who.int/ictrp/en/

Viewpoint
88
www.thelancet.com Vol 374 July 4, 2009
Second, unless new evidence is set in the context of
updated systematic reviews, readers cannot judge its
relevance. Yet among the world’s major general medical
journals, The Lancet is alone in requiring reports of new
research to be preceded by and to conclude with reference
to systematic reviews of other relevant evidence. In 2005,
the editors wrote: “…we will require authors of clinical
trials submitted to The Lancet to include a clear summary
of previous research fi ndings, and to explain how their
trial’s fi ndings aff ect this summary. The relation between
existing and new evidence should be illustrated by direct
reference to an existing systematic review and
meta-analysis”.23
This principle will remain challenging while the need
for up-to-date, reliable systematic reviews of research
fi ndings remains insuffi ciently recognised and supported.
Although the Cochrane Collaboration aspires to maintain
its reviews by adding new evidence to them, presenting
more detailed analyses, and correcting any mistakes
identifi ed, many Cochrane reviews are not being updated
in a timely manner, and the organisation is struggling to
deal with this defi ciency. The challenge of keeping
existing systematic reviews up to date has not been solved
by any other organisation in the world.
Conclusions and recommendations
Although some waste in the production and reporting of
research evidence is inevitable and bearable, we were
surprised by the levels of waste suggested in the evidence
we have pieced together. Since research must pass
through all four stages shown in the fi gure, the waste is
cumulative. If the losses estimated in the fi gure apply
more generally, then the roughly 50% loss at stages
2, 3, and 4 would lead to a greater than 85% loss, which
implies that the dividends from tens of billions of dollars
of investment in research are lost every year because of
correctable problems. Although we have mainly used
evidence about the design and reporting of clinical trials,
we believe it is reasonable to assume that the problems
also apply to other types of research.
Because there are problems within each stage of
production and reporting, there is no single, simple
solution. But even modest eff orts to understand and
improve production and reporting of research are likely
to yield substantially increased dividends for patients
and the public. Enough is known to justify some specifi c
suggestions for the attention of the research community,
and for action related to each of the stages of design and
reporting. These recommendations are shown in the
panel. Some elements of these recommendations refl ect
policies already implemented by some research funders
in some countries. For example, the NIHR’s Health
Technology Assessment Programme routinely requires
or commissions systematic reviews before funding
primary studies, publishes all research as web-accessible
monographs, and, since 2006, has made all new
protocols freely available.
Even though there is more to learn about the
“epidemiology” and “treatment” of waste in the
production and reporting of research evidence, we
believe that all of our recommendations are justifi ed on
the basis of the evidence we have cited. Action to address
this waste is needed now because it has human as well
as economic consequences, as illustrated by the
quotation with which this Viewpoint began.1
Contributors
The coauthors contributed equally to the background research and text
of the manuscript.
Confl icts of interest
We declare that we have no confl icts of interest.
Panel: Stages of waste in the production and reporting of research evidence—barriers
(in italics) and recommendations (bullet points)
Questions relevant to clinicians and patients
Poor engagement of end users of research in research questions and design
• Increase involvement of patients and clinicians in shaping research agendas and
specifi c questions
Incentives in fellowships and career paths to do primary research even if of low relevance
• Emphasise initial training in critical appraisal and systematic reviews rather than the
conduct of primary research
Appropriate design and methods
Poor training in research methods and research reporting
• Require training of all clinicians in methodological fl aws and biases in research;
improve training in research methods for those doing research apprenticeships
Lack of methodological input to research design and review of research
• Increase numbers of methodologists in health-care research
Incentives for primary research ignore the need to use and improve on existing research on the
same question
• Research funding bodies should require—and support—grant proposals to build on
systematic reviews of existing evidence
Published research fails to set the study in the context of all previous similar research
• Journal editors should require new studies to be set in the context of systematic
assessments of related studies
Accessible full publication
Non-registration of trials
• Require—by incentives and regulation—registration and publication of protocols for
all clinical trials at inception
Failure of sponsors and authors to submit full reports of completed research
• Support timely open access to full results on completion
Unbiased and usable report
Poor awareness and use by authors and editors of reporting guidelines
• Increase author and journal awareness of and training in reporting guidelines, such as
CONSORT and STARD statements (http://www.equator-network.org)
Many journal reviews focus on expert judgments about contribution to knowledge, rather than
methods and usability
• Supplement peer review of studies with review by methodologists and end users
Space restrictions in journals prevent publication of details of interventions and tests
• Support free access repositories—separate from any publications—so that clinicians
and researchers have details of the treatments, test, or instruments studied
CONSORT=Consolidated Standards of Reporting Trials. STARD=Standards for the Reporting of Diagnostic Accuracy Studies.
For more on the Health
Technology Assessment
Programme see
http://www.ncchta.org/funding/
clinicaltrials/index.shtml

Viewpoint
www.thelancet.com Vol 374 July 4, 2009
89
Acknowledgments
We are grateful to Doug Altman, Luis Gabriel Cuervo, John Kirwan,
Marie Claude Lavoie, Sally Davies, Tom Walley, and an anonymous referee
for helpful comments on an earlier draft of this paper.
References
1 Liberati A. An unfi nished trip through uncertainties. BMJ 2004;
328: 531.
2 Scherer RW, Langenberg P, von Elm E. Full publication of results
initially presented in abstracts. Cochrane Database Syst Rev 2007;
2: MR000005.
3 Cooksey D. A review of UK health research funding. December
2006. http://www.hm-treasury.gov.uk/d/pbr06_cooksey_fi nal_
report_636.pdf (accessed Feb 10, 2009).
4 National Institute for Health Research. Transforming health
research: the fi rst two years. Progress report 2006–2008.
http://www.nihr.ac.uk/fi les/pdfs/nihr%20progress%20report%202
006-2008.pdf (accessed Feb 10, 2009).
5 Garattini S, Chalmers I. Patients and the public deserve big
changes in evaluation of drugs. BMJ 2009; 338: 804–06.
6 Gross CP, Anderson GF, Powe NR. The relation between funding
by the National Institutes of Health and the burden of disease.
N Engl J Med 1999; 340: 1914–15.
7 Stuckler D, King L, Robinson H, McKee M. WHO’s budgetary
allocations and burden of disease: a comparative analysis. Lancet
2008; 372: 1563–69.
8 Perel P, Miranda JJ, Ortiz Z, Casas JP. Relation between the global
burden of disease and randomized clinical trials conducted in Latin
America published in the fi ve leading medical journals. PLoS ONE
2008: 3: e1696.
9 Oliver S, Gray J. A bibliography of research reports about patients’,
clinicians’ and researchers’ priorities for new research. Oxford:
James Lind Alliance, 2006. http://www.lindalliance.org/pdfs/JLA_O
liver%20and%20Gray%20report%2007_01_20.pdf (accessed
Feb 3, 2009).
10 Tallon D, Chard J, Dieppe P. Relation between agendas of the
research community and the research consumer. Lancet 2000;
355: 2037–40.
11 Chalmers I, Rounding C, Lock K. Descriptive survey of
non-commercial randomised controlled trials in the United
Kingdom, 1980–2002. BMJ 2003; 327: 1017.
12 Hewlett S, De Wit M, Richards P, et al. Patients and professionals
as research partners: challenges, practicalities and benefi ts.
Arthritis Rheum 2006; 55: 676–80.
13 Cooper NJ, Jones DR, Sutton AJ. The use of systematic reviews
when designing studies. Clin Trials 2005; 2: 260–64.
14 Patsopoulos NA, Analatos AA, Ioannidis JP. Relative citation impact
of various study designs in the health sciences. JAMA 2005;
293: 2362–66.
15 Mallett S, Clarke M. How many Cochrane reviews are needed to
cover existing evidence on the eff ects of health care interventions?
ACP J Club 2003; 139: A11.
16 Hewitt C, Hahn S, Torgerson DJ, Watson J, Bland JM. Adequacy
and reporting of allocation concealment: review of recent trials
published in four general medical journals. BMJ 2005;
330: 1057–58.
17 Rutjes AW, Reitsma JB, Di Nisio M, Smidt N, van Rijn JC,
Bossuyt PM. Evidence of bias and variation in diagnostic accuracy
studies. CMAJ 2006; 174: 469–76.
18 Chalmers I. From optimism to disillusion about commitment to
transparency in the medico-industrial complex. J R Soc Med 2006;
99: 337–41.
19 Hopewell S, Clarke MJ, Stewart L, Tierney J. Time to publication for
results of clinical trials. Cochrane Database Syst Rev 2007; 2: MR000011.
20 Hopewell S, Dickersin K, Clarke MJ, Oxman AD, Loudon K.
Publication bias in clinical trials. Cochrane Database Syst Rev 2008;
2: MR000006.
21 Dwan K, Altman DG, Arnaiz JA, et al. Systematic review of the
empirical evidence of study publication bias and outcome reporting
bias. PLoS ONE 2008; 3: e3081.
22 Glasziou P, Meats E, Heneghan C, Shepperd S. What is missing
from treatment descriptions in trials and reviews? BMJ 2008;
336: 1472–74.
23 Young C, Horton R. Putting clinical trials into context. Lancet 2005;
366: 107–08.