Negative results: why do they need to be published?

Abstract

This short narrative review article defines 'negative results' and cites several ethical and scientific reasons why such studies should be made publicly available.

Full text

Page 1

Negative results: why do they need to be published?
Peter Sandercock
This short narrative review article defines ‘negative results’
and cites several ethical and scientific reasons why such
studies should be made publicly available.
Key words:clinical trial,methodology,publication bias,stroke
units, systematic reviews, treatment
What do I mean by ‘negative results’? The term applies to
studies conducted both in human and in animal subjects and
encompasses three different types of result:
•truly inconclusive with ‘no evidence of effect’, generally
because the study was too small and inadequately powered
(several of the small studies included in the Cochrane system-
atic review of stroke units are in this category) (1);
•awell-conductedstudy,whichissufficientlylargetoprovide
‘clearevidenceof noeffect’,i.e.thatanyeffectistoosmalltobe
worthwhile pursuing either clinically or in further research
(the Clots in Legs Or Stockings after Stroke (CLOTS) trial
of graded compression stocking for deep vein thrombosis
prevention is a good example) (2); or
•clear evidence of harm when benefit had been expected.
Unfortunately, many such ‘negative’ yet still important
studies in man (3) and in animals (4) remain unpublished.
Why should negative studies be published? The most
important reason is ethics. If human subjects have given
consent to participate in a clinical research study, be it a treat-
ment trial, or an observational study, they have done so in the
clear understanding that the research results will in some way
be of benefit to other people and contribute to scientific
advance. Furthermore, these human subjects have exposed
themselves to risk and inconvenience by participating in the
study, and the justification for doing that ‘good deed’ should
be that the author makes the data publicly available and
ensuresthatitisputtogooduse.Althoughanimalsdonotgive
consent to participate in research, we still have an ethical duty
to make the best use of the data from any animals used in such
research.
There are many scientific reasons for publishing negative or
neutral (i.e.uninformative) studies; chiefly,they contain valu-
able information, which should become part of the scientific
record on the subject under study. Systematic reviews are an
essential part of the research cycle (5). When scientists plan
new research studies (clinical trials or observational studies in
humans or experiments on laboratory animals), the first step
should be a systematic review of the evidence (5). Such a
review may reveal that the question has already been answered
reliably, or it may indicate that a further study is justified. For
example,theUKMedicalResearchCouncilandtheUKHealth
Technology Appraisal Programme require that new applica-
tions for clinical trial funding should have performed (or at
least cite) an up-to-date systematic review of the subject to
ensure that the new research really is justified. If during the
course of a clinical study, a large negative or neutral study is
published, this might require the trial steering committee or
data monitoring committee to pause for thought and consider
whether the study should continue or be modified in some
way. At the end of any clinical study, the results should pref-
erably be presented in context of all the available evidence.
For clinical trials, this is a requirement of the Consolidated
Standards of Reporting Trials (CONSORT) guidelines on
publication of randomized clinical trials (6).
Systematic reviews can put a small but strikingly positive
study into context. For example, when a strikingly positive
small study is viewed within the totality of the evidence avail-
able, it becomes evident that it is a freak ‘lucky’ result arising
from the play of chance,and not a reliable estimate of the true
effect (4,7). The availability of the ‘negative’ studies then
ensures that the one small positive (but outlying) study is
interpreted appropriately.There are numerous examples from
the literature where small clinical trials or small studies of
genetic associations produce striking positive results (often
leading to a high-profile paper in a major journal), which are
then not subsequently replicated when larger, more reliable
(neutral or negative) studies are published (8–10).
To be reliable, systematic reviews need to include all rel-
evant randomized trials. The Stroke Unit Trialists’ Collabora-
tive systematic review of organized inpatient stroke care (1)
very clearly demonstrates the importance of making negative
trial results available for inclusion in systematic reviews.Four-
teen of the 16 trials comparing a stroke unit with general
medical ward care were‘negative’for the effect on death (they
were all underpowered to detect a moderate but clinically
Correspondence: Peter Sandercock, Division of Clinical Neurosciences,
University of Edinburgh, Western General Hospital, Bramwell Dott
Building, Edinburgh EH4 2XU, UK.
Email: peter.sandercock@ed.ac.uk
Twitter: @IST_3
Conflict of interest: None declared.
DOI: 10.1111/j.1747-4949.2011.00723.x
Leading opinion
© 2011 The Author.
International Journal of Stroke © 2011 World Stroke Organization Vol 7, January 2012, 32–33
32

Page 2

important difference), and four were never published.
However, the overall estimate of the effect from all the trials
together is that stroke unit care reduces the odds of death by
17% (95% confidence interval 4–29%) (P = 0·01), a result
which has helped support the introduction of stroke unit care
into practice worldwide. What would have happened if all of
the apparently ‘negative’ studies had remained unpublished?
We might well have‘lost’ one of the most significantly benefi-
cial interventions for stroke!
There are many obstacles to publication of‘neutral or nega-
tive’ studies; the first is the author. A study that does not have
a big fat juicy P-value does not stir the author’s adrenalin
sufficiently to write the paper in the first place. It may not be
in the commercial interest of the sponsors of negative research
for the results to be published. Journals like clear stories with
definite results to increase their sales. Negative studies do not
do much for journal income! There have been reports that
some journals would only publish results from studies that are
statistically significant at the P < 0·05 level. This is clearly
absurd, but I suspect that the practice still continues (perhaps
rather more covertly these days). In the fast-moving world of
the Internet and social media, where only the most strikingly
positive results gain their‘15 minutes of fame’, it is difficult to
ensure that neutral or negative results from well-conducted
research would reach the public domain.
There are a number of possible solutions. Journal editors
could (perhaps) be persuaded to prioritize publishing well-
conducted negative research over poorly conducted positive
research. There are easier alternatives: although the Journal
of Negative Results in Biomedicine (http://www.jnrbm.com/)
provides a rather specific destination, there are now many
open-access publishing journals. Furthermore, several grant-
giving bodies, such as the UK Medical Research Council and
the Wellcome Trust, expect data from research they have
fundedtobepublishedinanopen-accessformat(applications
for research grants now need to incorporate anticipated costs
for such publications). For authors without such funding,
many institutions now make datasets accumulated by their
scientistsfreelyavailableviaopendatarepositories(University
of Edinburgh Datashare site is an example http://datashare.
is.ed.ac.uk/).
In summary, if a study involves informed consent in
humans or the use of whole animals, and has been satisfacto-
rily conducted, it should appear in the publicly available sci-
entific record, irrespective of its overall conclusions.
Acknowledgements
I would like to thank William Whiteley, Charles Warlow, and
Norberto Cabral for helpful comments.
References
1 Stroke Unit Trialists’ Collaboration. Organised inpatient (stroke unit)
care for stroke.Cochrane Database of Systematic Reviews 2007; Issue 4.
Art. No.: CD000197. DOI: 10.1002/14651858.CD000197.pub2.
2 Dennis M, Sandercock PA, Reid J et al. Effectiveness of thigh-length
graduated compression stockings to reduce the risk of deep vein
thrombosis after stroke (CLOTS trial 1): a multicentre, randomised
controlled trial. Lancet 2009; 373:1958–65.
3 Gibson L, Brazzelli M, Thomas B, Sandercock P. A systematic review
of clinical trials of pharmacological interventions for acute ischaemic
stroke (1955–2008) that were completed, but not published in full.
Trials 2010; 11:43. http://www.trialsjournal.com/content/11/1/43
4 Sena ES, van der Worp HB, Bath PMW, Howells DW, Macleod MR.
Publication bias in reports of animal stroke studies leads to major
overstatement of efficacy. PLoS Biol 2010; 8:e1000344.
5 Bath PM, Gray LJ. Systematic reviews as a tool for planning and
interpreting trials. Int J Stroke 2009; 4:23–7.
6 Moher D, Hopewell S, Schulz KF et al. CONSORT 2010 explanation
and elaboration: updated guidelines for reporting parallel group ran-
domised trials. BMJ 2010; 340:c869.
7 CollinsR,MacMahonS.Reliableassessmentof theeffectsof treatment
on mortality and major morbidity,I: clinical trials.[Review] [74 refs].
Lancet 2001; 357:373–80.
8 Ioannidis JPA. Contradicted and initially stronger effects in highly
cited clinical research. JAMA 2005; 294:218–28.
9 Ioannidis JP. Why most discovered true associations are inflated.
Epidemiology 2008; 19:640–8.
10 Ioannidis JPA, Panagiotou OA. Comparison of effect sizes associated
with biomarkers reported in highly cited individual articles and in
subsequent meta-analyses. JAMA 2011; 305:2200–10.
Leading opinion
P. Sandercock
© 2011 The Author.
International Journal of Stroke © 2011 World Stroke Organization Vol 7, January 2012, 32–33
33