ArticlePDF Available

Reporting and Interpretation of Randomized Controlled Trials With Statistically Nonsignificant Results for Primary Outcomes

Authors:

Abstract and Figures

Previous studies indicate that the interpretation of trial results can be distorted by authors of published reports. To identify the nature and frequency of distorted presentation or "spin" (ie, specific reporting strategies, whatever their motive, to highlight that the experimental treatment is beneficial, despite a statistically nonsignificant difference for the primary outcome, or to distract the reader from statistically nonsignificant results) in published reports of randomized controlled trials (RCTs) with statistically nonsignificant results for primary outcomes. March 2007 search of MEDLINE via PubMed using the Cochrane Highly Sensitive Search Strategy to identify reports of RCTs published in December 2006. Articles were included if they were parallel-group RCTs with a clearly identified primary outcome showing statistically nonsignificant results (ie, P > or = .05). Two readers appraised each selected article using a pretested, standardized data abstraction form developed in a pilot test. From the 616 published reports of RCTs examined, 72 were eligible and appraised. The title was reported with spin in 13 articles (18.0%; 95% confidence interval [CI], 10.0%-28.9%). Spin was identified in the Results and Conclusions sections of the abstracts of 27 (37.5%; 95% CI, 26.4%-49.7%) and 42 (58.3%; 95% CI, 46.1%-69.8%) reports, respectively, with the conclusions of 17 (23.6%; 95% CI, 14.4%-35.1%) focusing only on treatment effectiveness. Spin was identified in the main-text Results, Discussion, and Conclusions sections of 21 (29.2%; 95% CI, 19.0%-41.1%), 31 (43.1%; 95% CI, 31.4%-55.3%), and 36 (50.0%; 95% CI, 38.0%-62.0%) reports, respectively. More than 40% of the reports had spin in at least 2 of these sections in the main text. In this representative sample of RCTs published in 2006 with statistically nonsignificant primary outcomes, the reporting and interpretation of findings was frequently inconsistent with the results.
Content may be subject to copyright.
ORIGINAL CONTRIBUTION
Reporting and Interpretation
of Randomized Controlled Trials
With Statistically Nonsignificant Results
for Primary Outcomes
Isabelle Boutron, MD, PhD
Susan Dutton, MSc
Philippe Ravaud, MD, PhD
Douglas G. Altman, DSc
ACCURATE PRESENTATION OF
the results of a randomized
controlled trial (RCT) is the
cornerstone of the dissemi-
nation of the results and their imple-
mentation in clinical practice. The Dec-
laration of Helsinki states that “Authors
have a duty to make publicly available
the results of their research on human
subjects and are accountable for the
completeness and accuracy of their re-
ports.” To help enforce this principle,
trial registration is required,1and re-
porting guidelines are available.2How-
ever, investigators usually have broad
latitude in writing their articles3; they
can choose which data to report and
how to report them.
Consequently, scientific articles are
not simply reports of facts, and au-
thors have many opportunities to con-
sciously or subconsciously shape the
impression of their results for readers,
that is, to add “spin” to their scientific
report.4Spin can be defined as specific
reporting that could distort the inter-
pretation of results and mislead read-
ers.3,5,6 The use of spin in scientific writ-
ing can result from ignorance of the
scientific issue, unconscious bias, or
willful intent to deceive.3Such dis-
torted presentation and interpretation
of trial results in published articles has
been highlighted in letters to editors
criticizing the interpretation of re-
sults7and in methodological reviews
evaluating misleading claims in pub-
lished reports of RCTs8,9 or systematic
reviews.10 However, to our knowl-
edge, the strategies used to create spin
in published articles have never been
systematically assessed.
Author Affiliations: Centre for Statistics in Medicine,
University of Oxford, Oxford, United Kingdom
(Drs Boutron and Altman and Ms Dutton); INSERM,
U738, Paris, France (Drs Boutron and Ravaud); As-
sistance Publique des Hoˆ pitaux de Paris, Hoˆpital Hoˆ-
tel Dieu, Centre d’E
´pide´ miologie Clinique, Paris (Drs
Boutron and Ravaud); and Universite´ Paris Des-
cartes, Faculte´deMe´ decine, Paris (Drs Boutron and
Ravaud).
Corresponding Author: Isabelle Boutron, MD, PhD,
Centre d’E
´pide´miologie Clinique, Hoˆpital Hoˆ tel
Dieu, 1, Place du Parvis Notre-Dame, 75181 Paris
CEDEX 4, France (isabelle.boutron@htd.aphp.fr).
Context Previous studies indicate that the interpretation of trial results can be dis-
torted by authors of published reports.
Objective To identify the nature and frequency of distorted presentation or “spin”
(ie, specific reporting strategies, whatever their motive, to highlight that the experi-
mental treatment is beneficial, despite a statistically nonsignificant difference for the
primary outcome, or to distract the reader from statistically nonsignificant results) in
published reports of randomized controlled trials (RCTs) with statistically nonsignifi-
cant results for primary outcomes.
Data Sources March 2007 search of MEDLINE via PubMed using the Cochrane Highly
Sensitive Search Strategy to identify reports of RCTs published in December 2006.
Study Selection Articles were included if they were parallel-group RCTs with a clearly
identified primary outcome showing statistically nonsignificant results (ie, P.05).
Data Extraction Two readers appraised each selected article using a pretested, stan-
dardized data abstraction form developed in a pilot test.
Results From the 616 published reports of RCTs examined, 72 were eligible and ap-
praised. The title was reported with spin in 13 articles (18.0%; 95% confidence in-
terval [CI], 10.0%-28.9%). Spin was identified in the Results and Conclusions sec-
tions of the abstracts of 27 (37.5%; 95% CI, 26.4%-49.7%) and 42 (58.3%; 95%
CI, 46.1%-69.8%) reports, respectively, with the conclusions of 17 (23.6%; 95% CI,
14.4%-35.1%) focusing only on treatment effectiveness. Spin was identified in the
main-text Results, Discussion, and Conclusions sections of 21 (29.2%; 95% CI, 19.0%-
41.1%), 31 (43.1%; 95% CI, 31.4%-55.3%), and 36 (50.0%; 95% CI, 38.0%-
62.0%) reports, respectively. More than 40% of the reports had spin in at least 2 of
these sections in the main text.
Conclusion In this representative sample of RCTs published in 2006 with statisti-
cally nonsignificant primary outcomes, the reporting and interpretation of findings was
frequently inconsistent with the results.
JAMA. 2010;303(20):2058-2064 www.jama.com
2058 JAMA, May 26, 2010—Vol 303, No. 20 (Reprinted) ©2010 American Medical Association. All rights reserved.
at Yale University on May 26, 2010 www.jama.comDownloaded from
We aimed to identify spin in reports
of parallel-group RCTs with statisti-
cally nonsignificant results for the pri-
mary outcome and to develop a scheme
for classification of spin strategies. We
focused on trials with statistically non-
significant primary outcomes because the
interpretation of these results are more
likely to be affected by a preconceived
notion of effectiveness, resulting in a bi-
ased interpretation.9
METHODS
Selection of Articles
The articles were screened from a rep-
resentative cohort of articles of RCTs
indexed in PubMed. The search strat-
egy and eligibility criteria for this co-
hort have been described elsewhere.11
Randomized controlled trials were de-
fined as prospective studies assessing
health care interventions in human par-
ticipants randomly allocated to study
groups. Reports of cost-effectiveness
studies, reports of diagnostic test ac-
curacy, and non–English-language re-
ports were excluded.
In brief, the Cochrane Highly Sensi-
tive Search Strategy,12 performed in
PubMed to identify primary reports of
RCTs published in December 2006 and
indexed in PubMed by March 22, 2007,
yielded 1735 PubMed citations. After
reading the titles and abstracts of re-
trieved citations, reports of obviously
noneligible trials were excluded, and the
full-text article and any online appendi-
ces were obtained and evaluated for 879
selected citations. Of these, 263 cita-
tions were excluded after the full text was
read; the remaining 616 were included
in this representative sample of RCTs.
From this sample, we selected parallel-
group RCTs with clearly identified pri-
mary outcomes. We excluded equiva-
lence or noninferiority trials, crossover
trials, cluster trials, factorial and split-
body designs, trials with more than 2
groups, and phase 2 trials. Primary out-
comes were those explicitly reported as
such in the published article. If none was
explicitly reported, we considered the
outcomes stated in the sample size esti-
mation; if outcomes were not stated in
the sample size estimation, we took the
outcomes in the primary study objec-
tives, if available. If no primary out-
come was clearly identified (ie, explic-
itly specified in the article, in a sample
size calculation, or in the primary study
objectives), the article was excluded.
One reviewer (I.B.) screened the full-
text articles and determined results for
all primary outcomes according to sta-
tistical significance: results statisti-
cally signficant (ie, P.05), results that
did not reach statistical significance (ie,
P.05), or unclear results. We in-
cluded only trials with nonsignificant
results (ie, P.05) for all primary out-
comes. When no formal statistical
analyses were reported for the pri-
mary outcomes, we attempted to cal-
culate the effect size and confidence in-
terval for the primary outcomes, and the
article was included if the estimated
treatment effect was not statistically sig-
nificant. If we could not calculate the
effect size using the published data, the
article was excluded.
Assessment of Selected Articles
For each selected article, 2 readers (I.B.,
S.D.) independently read the title, ab-
stract, and Methods, Results, Discus-
sion, and Conclusions sections, as well
as online appendices referenced in the
articles, when available. The reviewers
independently appraised the content of
the article using a pretested and stan-
dardized data abstraction form; then they
met to compare results. All discrepan-
cies were discussed to obtain consen-
sus; if needed, the article was discussed
with a third reader (D.G.A.). The repro-
ducibility was moderate, with a of 0.47
(95% confidence interval [CI], 0.27-
0.67) for presence of spin in the ab-
stract Conclusions and of 0.64 (95% CI,
0.47-0.82) for spin in the article Con-
clusions.
General Characteristics
of Selected Articles
For each selected article, we recorded the
funding source (ie, for-profit, non-
profit, or both; not reported, no fund-
ing), 2007 journal impact factor, num-
ber of citations in 2008, the experimental
intervention, comparator, sample size,
and type of primary outcomes (safety,
efficacy, both).
Reporting the Primary Outcomes
in Abstract and Main Text
We checked whether the primary out-
comes were clearly identified in the ab-
stract. We also recorded the reporting
of results for the primary outcomes both
in the abstract and in the article (ie, re-
porting of estimated effect size with or
without precision and reporting of sum-
mary statistics [eg, proportion of event,
mean] for each group with or without
precision).
Definition of Spin
In the context of a trial with statisti-
cally nonsignificant primary out-
comes, spin was defined as use of spe-
cific reporting strategies, from whatever
motive, to highlight that the experi-
mental treatment is beneficial, despite
a statistically nonsignificant differ-
ence for the primary outcome, or to dis-
tract the reader from statistically non-
significant results.
Development
of Classification Scheme
All of the authors participated in the de-
velopment of a classification scheme to
standardize the collection of the strate-
gies used for spin in the selected re-
ports. For this purpose, in a first step, we
reviewed the literature published on this
topic.3,6,13-22 We also contacted by e-mail
all the members of the Cochrane Statis-
tical Method Group and invited them to
send us any examples of published RCTs
with spin, in any medical field, and with
any publication date. Lastly, we re-
viewed a sample of trials with statisti-
cally nonsignificant results published in
general medical journals with high im-
pact factors or in specialist journals.23 The
classification scheme was developed fol-
lowing discussion and agreement among
the authors.
Strategies of Spin
Using the developed classification
scheme, we searched for spin in each
section of the manuscript in our sample,
ie, abstract Results; abstract Conclu-
DISTORTED PRIMARY OUTCOMES PRESENTATION IN RCTS
©2010 American Medical Association. All rights reserved. (Reprinted) JAMA, May 26, 2010—Vol 303, No. 20 2059
at Yale University on May 26, 2010 www.jama.comDownloaded from
sions; and main-text Results, Discus-
sion, and Conclusions (ie, last para-
graph of the manuscript when this
paragraph summarized the results) sec-
tions. We then determined whether
authors had used a spin strategy. The
strategies of spin considered were (1)
a focus on statistically significant re-
sults (within-group comparison, sec-
ondary outcomes, subgroup analyses,
modified population of analyses); (2)
interpreting statistically nonsignifi-
cant results for the primary outcomes
as showing treatment equivalence or
comparable effectiveness; and (3) claim-
ing or emphasizing the beneficial effect
of the treatment despite statistically
nonsignificant results. All other spin
strategies that could not be classified
according to this scheme were system-
atically recorded and secondarily
classified.
Extent of Spin
We determined the extent of spin across
the whole report, defined as the num-
ber of sections with spin in the ab-
stract (spin in the Results section only,
in the Conclusions section only, or in
both sections) and in the main text
(spin in one section other than the Con-
clusions section, in the Conclusions
section only, in 2 sections, or in all 3
sections). The assessment of the ex-
tent of spin is exploratory and should
not be considered a scoring system. This
classification scheme was developed by
consensus among the authors for a
pragmatic purpose: to be able to cap-
ture the diversity of spin in terms of vol-
ume (ie, whether spin concerned only
a small part or most of the article).
Level of Spin in Conclusions
We also classified the level of spin in
the Conclusions sections of the ab-
stract and the main text as follows. High
spin was defined as no uncertainty in
the framing, no recommendations for
further trials, and no acknowledg-
ment of the statistically nonsignifi-
cant results for the primary outcomes;
in addition, when the Conclusions sec-
tion reported recommendations to use
the treatment in clinical practice, we
classified this section as having a high
level of spin. Moderate spin was de-
fined as some uncertainty in the fram-
ing or recommendations for further
trials but no acknowledgment of the sta-
tistically nonsignificant results for the
primary outcomes. Low spin was de-
fined as uncertainty in the framing and
recommendations for further trials or
acknowledgment of the statistically
nonsignificant results for the primary
outcomes. This classification of the level
of spin is exploratory and not vali-
dated and should not be considered a
scoring system. The level of spin was
used to explore the heterogeneity of
spin in the reporting of conclusions.
Statistical Analysis
Medians and interquartile ranges for
continuous variables and number (%)
of articles for categorical variables were
calculated. Statistical analyses were per-
formed using SAS version 9.1 (SAS In-
stitute Inc, Cary, North Carolina).
RESULTS
General Characteristics
of Selected Articles
Of the 616 PubMed citations retrieved,
205 reports of parallel-group RCTs
were identified. Among these reports,
we identified and appraised 72 reports
with statistically nonsignificant results
for the primary outcomes (FIGURE).
Characteristics of the included reports
are presented in TABLE 1. Most reports
evaluated efficacy (n=63 [87.5%; 95%
CI, 77.6%-94.1%]), and half evaluated
pharmacological treatments. The fund-
ing source was for-profit (only or with
a nonprofit source) in one-third of the
reports and was not stated in 27 (37.5%).
Reporting of Primary Outcomes
in Abstract and Main Text
Primary outcomes were clearly identi-
fied in 44 of the 72 report abstracts
(61.1%; 95% CI, 48.9%-72.4%). In 3 ab-
stracts (4.2%; 95% CI, 0.9%-11.7%), a
secondary outcome was reported as
being the primary outcome. Only 9 ab-
stracts (12.5%; 95% CI, 5.9%-22.4%)
reported the effect size and 95% con-
fidence interval, and 28 (38.9%; 95%
CI, 27.6%-51.1%) did not report any
numerical results for primary out-
comes. In only 16 articles (22.2%; 95%
CI, 13.3%-33.6%) did the main text de-
scribe the effect size and its precision
for primary outcomes; in 21 (29.2%;
95% CI, 19.0%-41.1%), the main text
reported only summary statistics for
each group, without precision.
Spin Strategies
The strategies of spin in each article sec-
tion are shown in TABLE 2. The title was
reported with spin in 13 of the 72 ar-
ticles (18.0%; 95% CI, 10.0%-28.9%).
Spin was identified in 27 (37.5%; 95%
CI, 26.4%-49.7%) and 42 (58.3%; 95%
CI, 46.1%-69.8%) of the abstract Re-
sults and Conclusions sections, respec-
tively. We identified spin in 21 (29.2%;
95% CI, 19.0%-41.1%), 31 (43.1%; 95%
CI, 31.4%-55.3%), and 36 (50.0%; 95%
CI, 38.0%-62.0%) of the main-text Re-
Figure. Study Selection
133 Excluded
124 Statistically significant results
(P<.05)
9 Unclear results
263 Excluded based on review of full
text and online appendices
856 Excluded based on review of titles
and abstracts
411 Excluded
122 No primary outcome identified
100 Crossover trial
93 Multiple-group trial
26 Pilot study
16 Split-body design trial
13 Cluster trial
10 Brief communication/letter
10 Factorial trial
10 Noninferiority or equivalence trial
5 Phase 1 trial
3 Phase 2 trial
1 Sequential trial
2 Other
72 RCTs with statistically nonsignificant
results for all primary outcomes
identified and included in analysis
205 Parallel-group RCTs with clearly
identified primary outcomes
included
616 Included in sample
879 Identified for further review
1735 Potentially relevant trials
identified in PubMed search
RCT indicates randomized controlled trial.
DISTORTED PRIMARY OUTCOMES PRESENTATION IN RCTS
2060 JAMA, May 26, 2010—Vol 303, No. 20 (Reprinted) ©2010 American Medical Association. All rights reserved.
at Yale University on May 26, 2010 www.jama.comDownloaded from
sults, Discussion, and Conclusions sec-
tions, respectively.
The strategies of spin were also di-
verse (Table 2). Examples are provided
in eTable 1, available at http://www.jama
.com. In abstracts, spin consisted mainly
of focusing on within-group compari-
son and subgroup analyses in the Re-
sults section. One-quarter of the ab-
stract Conclusions sections focused on
only the beneficial effect of treatment,
claiming equivalence or comparable ef-
fectiveness (n=10 [13.9%; 95% CI,
6.9%-24.1%]), claiming efficacy (n=4
[5.6%; 95% CI, 1.5%-13.6%]), or focus-
ing on only statistically significant re-
sults such as within-group, secondary
outcome, or subgroup analyses (n=3
[4.2%; 95% CI, 0.9%-11.7%]). Further-
more, 9 abstract Conclusions sections
(12.5%; 95% CI, 5.9%-22.4%) acknowl-
edged statistically nonsignificant pri-
mary outcomes but focused on or em-
phasized statistically significant results.
Other specific strategies of spin were
identified. In some reports in which pri-
mary outcomes concerned safety, au-
thors interpreted statistically nonsig-
nificant results as demonstrating lack
of any difference in adverse events. As
an example, the authors of one study
concluded that “we have demon-
strated (for the first time) that [with the
treatment], embryo implantation is un-
altered.” Some reports focused on an
overall within-group comparison as if
the trial planned was a before-after
study, concluding, for example, that
“the mean improvement...wasclini-
cally relevant in both treatment groups.”
Some authors focused on another ob-
jective to distract the reader from the
statistically nonsignificant results, such
as identifying a genetic prognostic fac-
tor of improvement.
Extent of Spin
As shown in TABLE 3, the extent of spin
varied. In total, 49 of the 72 abstracts
(68.1%; 95% CI, 56.0%-78.6%) and 44
main texts (61.1%; 95% CI, 48.9%-
72.4%) were classified as having spin in
at least 1 section. More than 40% of the
articles had spin in at least 2 sections of
the main text. Spin was identified in all
sections of 20 abstracts (27.8%; 95% CI,
17.9%-39.6%) and 14 articles (19.4%;
95% CI, 11.1%-30.5%).
Level of Spin in Conclusions
The level of spin in Conclusions sec-
tions is illustrated in Table 3, and ex-
amples are provided in eTable 2. We
identified spin in more than half of the
Conclusions sections; the level of spin
was high (ie, no uncertainty in the fram-
ing, no recommendations for further
trials, and no acknowledgment of the sta-
tistically nonsignificant results for the
primary outcomes or recommenda-
tions to use the treatment in clinical
practice) in 24 abstracts Conclusions
sections (33.3%; 95% CI, 22.7%-
45.4%) and 19 main-text Conclusions
sections (26.4%; 95% CI, 16.7%-38.1%).
Examples of spin identified are pre-
sented in the eAppendix.
COMMENT
This study appraised the strategies of
spin used in reports of RCTs with sta-
tistically nonsignificant results for pri-
mary outcomes. We evaluated 72 re-
ports selected from all reports of RCTs
published in December 2006.11 Spin
used in the articles and their abstracts
was common, but strategies used for
spin varied. Furthermore, spin seemed
more prevalent in article abstracts than
in the main texts of articles.
Our results are consistent with those
of other related studies showing a posi-
tive relation between financial ties and
favorable conclusions stated in trial re-
ports.24,25 Other studies assessed dis-
crepancies between results and their
interpretation in the Conclusions sec-
tions.10,26 Yank and colleagues10 found
that for-profit funding of meta-analyses
was associated with favorable conclu-
sions but not favorable results. Other
studies have shown that the Discus-
sion sections of articles often lacked a
discussion of limitations.27
Our results add to these previous
methodological reviews10,24-26 in that ours
was a systematic study of the use of in-
appropriate presentation in published
trial reports, for which we propose a clas-
sification of the strategies authors use for
spin in their reports. Furthermore, un-
like other studies10,24-26 that investi-
gated a specific category of journals,
medical area, or category of treatment,
ours took a representative sample.
Table 1. Report Characteristics
Characteristic
No. (%) [95% CI]
(N = 72)
Journal type
Specialty 69 (95.8) [88.3-99.1]
General medical 3 (4.2) [0.9-11.7]
Top 5 medical area
Cardiology 9 (12.5) [5.9-22.4]
Obstetrics/gynecology 9 (12.5) [5.9-22.4]
Surgery 9 (12.5) [5.9-22.4]
Psychiatry/psychology 7 (9.7) [4.0-19.0]
Anesthesia 5 (6.9) [2.3-15.5]
Primary outcome
Safety of treatment 7 (9.7) [4.0-19.0]
Efficacy of treatment 63 (87.5) [77.6-94.1]
Safety and efficacy 1 (1.4) [0.0-7.5]
Other (diagnosis
accuracy)
1 (1.4) [0.0-7.5]
Primary outcome
Binary 25 (34.7) [23.9-46.9]
Other 43 (59.7) [47.5-71.1]
Binary and other 4 (5.6) [1.5-13.6]
Sample size, median
(IQR) [range]
84 (46-206) [4-6848]
Journal impact factor,
median (IQR) [range]
2.9 (2.3-4.5) [0-52.6]
Number of citations in
2008, median
(IQR) [range]
4 (1-7) [0-98]
Experimental treatment
Drug 37 (51.4) [39.3-63.3]
Surgery/procedure 11 (15.3) [7.9-25.7]
Participative
intervention (eg,
rehabilitation,
education)
7 (9.7) [4.0-19.0]
Device 7 (9.7) [4.0-19.0]
Other 10 (13.9) [6.9-24.1]
Comparator(s)
Placebo 19 (26.4) [16.7-38.1]
Usual care 14 (19.4) [11.1-30.5]
Drug 18 (25.0) [15.5-36.6]
Surgery/procedure 7 (9.7) [4.0-19.0]
Participative
intervention (eg,
rehabilitation,
education)
5 (6.9) [2.3-15.5]
Device 4 (5.6) [1.5-13.6]
Other 5 (6.9) [2.3-15.5]
Funding source
None 2 (2.8) [0.3-9.7]
For-profit 16 (22.2) [13.3-33.6]
Nonprofit 19 (26.4) [16.7-38.1]
For-profit and
nonprofit
8 (11.1) [4.9-20.7]
Not reported 27 (37.5) [26.4-49.7]
Abbreviations: CI, confidence interval; IQR, interquartile range.
DISTORTED PRIMARY OUTCOMES PRESENTATION IN RCTS
©2010 American Medical Association. All rights reserved. (Reprinted) JAMA, May 26, 2010—Vol 303, No. 20 2061
at Yale University on May 26, 2010 www.jama.comDownloaded from
We identified many strategies of spin.
The most familiar and common ap-
proach was to focus on statistically sig-
nificant results for other analyses, such
as within-group comparisons, second-
ary outcomes, or subgroup analyses. An-
other common strategy was to interpret
P.05 as demonstrating a similar effect
when the study was not designed to as-
sess equivalence or noninferiority (such
trials require specific design and con-
duct, as well as a larger sample size, than
superiority trials). This dubious inter-
pretation was used only when the com-
parator was an active treatment.
Some authors interpreted the trial re-
sults as being from a before-after study;
they focused on within-group compari-
sons that were statistically significant for
the experimental treatment but not for
the comparator, which they incor-
rectly interpreted as demonstrating the
beneficial effect of the treatment.28 Some
authors reported that they had demon-
strated the beneficial effect of both treat-
ments when the results showed a sta-
tistically significant change from baseline
for each group or for both groups com-
bined.20,29 Some reports of safety trials
provided an inadequate interpretation of
the nonsignificant results by conclud-
ing lack of harm of the experimental
treatment. Other methods relied on
masking the nonsignificant results by fo-
cusing on other objectives. In one re-
port, the authors statistically com-
pared the experimental group, also not
with the comparator in that trial, but
rather with the placebo group of an-
other trial to conclude that the treat-
ment was better than placebo.
Lastly, our results highlight the im-
portant prevalence of spin in the ab-
stract as compared with the main text
of an article. These results have impor-
tant implications, because readers of-
ten base their initial assessment of a trial
on the information reported in an ab-
stract. They may then use this infor-
mation to decide whether to read the
full report, if available. Furthermore, ab-
stracts are freely available, and in some
situations, clinical decisions might be
made on the basis of the abstract alone.30
Our study has several limitations.
First, the assessment of spin necessarily
involved some subjectivity, because the
strategies used for spin were highly vari-
able and interpretation depended on the
context. Interpretation of trial results is
not a straightforward process, and some
disagreement may arise, even among au-
thors.31 We attempted to limit this sub-
jectivity by having 2 reviewers extract the
data independently using a standard-
ized data abstraction form, with any dis-
agreements resolved by consensus. How-
Table 2. Spin in the Title, Abstract, and Main Text of Articles
Spin
No. (%) [95% CI]
Abstract (n = 72) Main Text (n = 72)
Spin in the title NA 13 (18.0) [10.0-28.9]
Spin in Results section 27 (37.5) [26.4-49.7] 21 (29.2) [19.0-41.1]
Focus on statistically significant within-group
comparisona8 (11.1) [4.9-20.7] 10 (13.9) [6.9-24.1]
Focus on statistically significant secondary outcomes 3 (4.2) [0.9-11.7] 0
Focus on statistically significant subgroup analyses 6 (8.3) [3.1-17.3] 4 (5.6) [1.5-13.6]
Focus on statistically significant modified
population of analyses (eg, per-protocol analyses)
1 (1.4) [0.0-7.5] 1 (1.4) [0.0-7.5]
Focus on statistically significant within- and
between-group comparisons for secondary
outcomes
1 (1.4) [0.0-7.5] 0
Other spin only in the Results section 8 (11.1) [4.9-20.7] 6 (8.3) [3.1-17.3]
Spin in the synthesis of the results in Discussion section NA 31 (43.1) [31.4-55.3]
Reporting of statistically nonsignificant outcome
as if the trial were an equivalence trial
NA 6 (8.3) [3.1-17.3]
Focus on statistically significant secondary outcomesbNA 3 (4.2) [0.9-11.7]
Focus on statistically significant subgroup analysescNA 4 (5.6) [1.5-13.6]
Focus on statistically significant modified population of
analyses
NA 1 (1.4) [0.0-7.5]
Focus on overall within-group improvement NA 9 (12.5) [5.9-22.4]
Ruling out adverse event NA 3 (4.2) [0.9-11.7]
Other spin in the synthesis of the results NA 7 (9.7) [4.0-19.0]
Spin in Conclusions section 42 (58.3) [46.1-69.8] 36 (50.0) [38.0-62.0]
Focus only on treatment effectiveness 17 (23.6) [14.4-35.1] 14 (19.4) [11.1-30.5]
Claiming equivalence for statistically nonsignificant
results
10 (13.9) [6.9-24.1] 6 (8.3) [3.1-17.3]
Claiming efficacy with no consideration of the
statistically nonsignificant primary outcome
4 (5.6) [1.5-13.6] 4 (5.6) [1.5-13.6] d
Focusing only on statistically significant results 3 (4.2) [0.9-11.7] 4 (5.6) [1.5-13.6]
Acknowledge statistically nonsignificant results for the
primary outcome but emphasize the beneficial
effect of treatment
3 (4.2) [0.9-11.7] 1 (1.4) [0.0-7.5]
Acknowledge statistically nonsignificant results for the
primary outcome but emphasize other statistically
significant results
6 (8.3) [3.1-17.3] 4 (5.6) [1.5-13.6]
Other spin in Conclusions section 16 (22.2) [13.3-33.6] 21 (29.2) [19.0-41.1]
Conclusion ruling out an adverse event on
statistically nonsignificant results
3 (4.2) [0.9-11.7] 3 (4.2) [0.9-11.7]
Conclusion focusing on within-group assessment
(both treatments are effective/treatment
administered in both groups is effective
(eg, add-on studies)
6 (8.3) [3.1-17.3] 9 (12.5) [5.9-22.4]
Recommendation to use the treatment 3 (4.2) [0.9-11.7] 3 (4.2) [0.9-11.7] e
Focus on another objective 2 (2.8) [0.3-9.7] 3 (4.2) [0.9-11.7]
Comparison with placebo group of another trial 1 (1.4) [0.0-7.5] 0
Statistically nonsignificant subgroup results
reported as beneficial
1 (1.4) [0.0-7.5] 1 (1.4) [0.0-7.5]
Other 0 2 (2.8) [0.3-9.7]
Abbreviations: CI, confidence interval; NA, not applicable.
aAll these articles also reported the statistically nonsignificant results for the primary outcome in the abstract and in the
main text.
bOne article also associated with other spin strategies.
cOne article associated with particular focus on overall within-group analyses.
dOne article associated with other spin (focuses on overall within-group assessment).
eThree articles associated respectively with particular focus on overall within-group analyses, acknowledgment
of negative primary outcome plus focus on positive secondary outcomes, and focus only on statistically significant
results.
DISTORTED PRIMARY OUTCOMES PRESENTATION IN RCTS
2062 JAMA, May 26, 2010—Vol 303, No. 20 (Reprinted) ©2010 American Medical Association. All rights reserved.
at Yale University on May 26, 2010 www.jama.comDownloaded from
ever, to our knowledge, no objective
measure exists for the subjective com-
ponent of interpretation.32 Conse-
quently, to be completely transparent, a
detailed summary of all the examples of
spin we classified is available in the eAp-
pendix.
We dichotomized trial findings as posi-
tive or negative using an arbitrary value
(P=.05) as a significance threshold. How-
ever, we acknowledge that the interpre-
tation of RCTs should not be based solely
on the arbitrary Pvalue of .05 dichotomiz-
ing findings as positive or negative.
We focused on spin only in trials for
which the primary outcomes were clearly
defined and results for the primary out-
comes were not statistically significant.
This focus implies that the strategies iden-
tified may not be applicable to all reports
of RCTs and that other strategies of spin
may not have been identified. Further-
more, when the results of an RCT are not
statistically significant, the risk of spin
may be increased. Trialists and spon-
sors are rarely neutral regarding the
results of their trial. They may have
invested considerable time, energy, and
money in developing the experimental
intervention and expended much effort
in planning and conducting the trial.
Therefore, they may have a strong pre-
conception about the beneficial effect of
the experimental intervention. Further-
more, the results of the trial could have
important implications at different lev-
els, eg, for the publication of the trial
results in terms of delay and type of jour-
nal33; for the use of the experimental treat-
ment in clinical practice; and, conse-
quently, for future career advancement
or profit.34,35 A trial with statistically non-
significant results will thus frequently be
a disappointment and could lead to sub-
conscious or even deliberate intent to
mislead the reader when presenting and
interpreting the trial results.32,36 Few
authors have studied this phenomenon,
but Hewitt and colleagues reviewed a
panel of 17 trial reports with nonsignifi-
cant results published in BMJ. They found
that, despite evidence that the treat-
ment might be ineffective, in 3 trials the
authors seemed to support the experi-
mental intervention.9
We focused on only some categories
of spin, and other forms of spin may not
have been identified. For example, we did
not consider some specific strategies of
spin, such as authors obscuring the risk
associated with the experimental treat-
ment, as reported in the published Vi-
oxx GI Outcomes Research (VIGOR)
study. That report concealed the cardio-
vascular risk by presenting the hazard of
myocardial infarction as if the compara-
tor (ie, naproxen) were the interven-
tion group, concluding on the protec-
tive effect of the comparator (relative risk,
0.2; 95% CI, 0.1-0.7)37 instead of the
harmful effect of the experimental treat-
ment (ie, rofecoxib) (relative risk, 5.00;
95% CI, 1.68-20.13).38
We cannot say to what extent the
spin we identified might have been de-
liberately misleading, the result of lack
of knowledge, or both. Nor are we able
to draw conclusions about the pos-
sible effect of the spin on peer review-
ers’ and readers’ interpretations. Stud-
ies evaluating the effect of framing on
clinical practice have focused on the re-
porting of treatment-effect estimates
and showed inconsistent results.39,40
Our study has identified many dif-
ferent strategies that authors use to pro-
vide a biased interpretation of results
of RCTs with statistically nonsignifi-
cant results for primary outcomes. Peer
and editorial reviewers must be aware
of the different strategies of spin used
to temper the article text. The choice
of analyses reported (statistically sig-
nificant analyses such as subgroup
analyses or within-group analyses) and
the terms used to report and interpret
results are important in a scientific ar-
ticle. Special attention should be paid
to inadequate interpretation of the trial
results, particularly when authors con-
clude on efficacy from secondary out-
comes, subgroup analyses, or within-
group comparisons or when the authors
inadequately interpret lack of differ-
ence as demonstrating equivalence in
terms of safety or efficacy. The publi-
cation process in biomedical research
tends to favor statistically significant re-
sults and to be responsible for “opti-
mism bias” (ie, unwarranted belief in
the efficacy of a new therapy).41 Re-
ports of RCTs with statistically signifi-
cant results for outcomes are pub-
lished more often and more rapidly than
are those of trials with statistically non-
significant results.34,42 Good evidence
exists of selective reporting of statisti-
cally significant results for outcomes in
published articles.33,43-46
Table 3. Extent of Spin in the Abstract and Main Text of Articles and Level of Spin
in Conclusions Sections
Spin
No. (%) [95% CI]
Abstract (n = 72) Main Text (n = 72)
Extent of spin
None 23 (31.9) [21.4-44.0] 28 (38.9) [27.6-51.1]
In 1 section other than
Conclusions section
7 (9.7) [0.4-19.0] 5 (6.9) [2.3-15.5]
In the Conclusions section only 22 (30.6) [20.2-42.5] 9 (12.5) [5.9-22.4]
In 2 sections NA 16 (22.2) [13.3-33.6]
In all sections 20 (27.8) [17.9-39.6] 14 (19.4) [11.1-30.5]
Level of spin in Conclusions section
None 30 (41.7) [30.2-53.9] 36 (50.0) [38.0-62.0]
Lowa9 (12.5) [5.9-22.4] 8 (11.1) [4.9-20.7]
Moderateb9 (12.5) [5.9-22.4] 9 (12.5) [5.9-22.4]
Highc24 (33.3) [22.7-45.4] 19 (26.4) [16.7-38.1]
Abbreviations: CI, confidence interval; NA, not applicable.
aSpin in the Conclusions section with acknowledgment of the statistically nonsignificant results for the primary out-
come or spin in the Conclusions section with no acknowledgment of the statistically nonsignificant results for the
primary outcome but reported with uncertainty and recommendations for further trials.
bSpin in the Conclusions section with no acknowledgment of the statistically nonsignificant results for the primary out-
come but reported with uncertainty or recommendations for further trials.
cSpin in the Conclusions section with no uncertainty, no recommendations for further trials, and no acknowledgment of the
statistically nonsignificant results for the primary outcome or spin with recommendation to use the treatment in clinical practice.
DISTORTED PRIMARY OUTCOMES PRESENTATION IN RCTS
©2010 American Medical Association. All rights reserved. (Reprinted) JAMA, May 26, 2010—Vol 303, No. 20 2063
at Yale University on May 26, 2010 www.jama.comDownloaded from
In conclusion, in this representative
sample of RCTs indexed in PubMed and
published in December 2006 with sta-
tistically nonsignificant primary out-
comes, the reporting and interpretation
of findings was frequently inconsistent
with the results. However, this work is
only a first step, and future research is
needed. Determining which category and
level of spin affect readers’ interpreta-
tion is important. Future research on the
reasons for and the mechanisms of spin
would also be useful. We hope that high-
lighting this issue may lead to more vigi-
lance by peer reviewers and editors to re-
duce the use of these questionable
strategies, which can distort the inter-
pretation of research findings.
Author Contributions: Dr Boutron had full access to
all of the data in the study and takes responsibility for
the integrity of the data and the accuracy of the data
analysis.
Study concept and design: Boutron, Dutton, Ravaud,
Altman.
Acquisition of data: Boutron, Dutton.
Analysis and interpretation of data: Boutron, Dutton,
Ravaud, Altman.
Drafting of the manuscript: Boutron.
Critical revision of the manuscript for important in-
tellectual content: Dutton, Ravaud, Altman.
Statistical analysis: Boutron.
Financial Disclosures: None reported.
Funding/Support: Dr Boutron was supported by a grant
from the Societe Francaise de Rhumatologie (SFR) and
that Lavoisier Program (Ministère des Affaires
e´ trangères et europe´ ennes).
Role of the Sponsors: The SFR and the Lavoisier Pro-
gram (Ministère des Affaires e´ trangères et europ-
e´ ennes) had no role in the design and conduct of the
study; the collection, management, analysis, and in-
terpretation of the data; or the preparation, review,
or approval of the manuscript.
Online-Only Material: eTable 1, eTable 2, and the
eAppendix are available at http://www.jama.com.
Additional Contributions: We are very grateful to
Ly-Mee Yu, Msc (Center for Statistics in Medicine, Uni-
versity of Oxford, Oxford, United Kingdom), for her
important work in developing the database of the rep-
resentative reports of randomized controlled trials in-
dexed in PubMed. Ms Yu received no compensation
for her contributions.
REFERENCES
1. DeAngelis CD, Drazen JM, Frizelle FA, et al; Inter-
national Committee of Medical Journal Editors. Clini-
cal trial registration: a statement from the Interna-
tional Committee of Medical Journal Editors. JAMA.
2004;292(11):1363-1364.
2. Altman DG, Schulz KF, Moher D, et al; CON-
SORT GROUP (Consolidated Standards of Reporting
Trials). The revised CONSORT statement for report-
ing randomized trials: explanation and elaboration. Ann
Intern Med. 2001;134(8):663-694.
3. Fletcher RH, Black B. “Spin” in scientific writing:
scientific mischief and legal jeopardy. Med Law. 2007;
26(3):511-525.
4. Junger D. The rhetoric of research: embrace sci-
entific rhetoric for its power. BMJ. 1995;311(6996):
61.
5. Bailar JC. How to distort the scientific record with-
out actually lying: truth, and arts of science. Eur J Oncol.
2006;11(4):217-224.
6. Marco CA, Larkin GL. Research ethics: ethical is-
sues of data reporting and the quest for authenticity.
Acad Emerg Med. 2000;7(6):691-694.
7. Hro´ bjartsson A, Gøtzsche PC. Powerful spin in the
conclusion of Wampold et al.’s re-analysis of placebo
versus no-treatment trials despite similar results as in
original review. J Clin Psychol. 2007;63(4):373-
377.
8. Jefferson T, Di Pietrantonj C, Debalini MG, Rivetti
A, Demicheli V. Relation of study quality, concor-
dance, take home message, funding, and impact in
studies of influenza vaccines: systematic review. BMJ.
2009;338:b354.
9. Hewitt CE, Mitchell N, Torgerson DJ. Listen to the
data when results are not significant. BMJ. 2008;
336(7634):23-25.
10. Yank V, Rennie D, Bero LA. Financial ties and con-
cordance between results and conclusions in meta-
analyses: retrospective cohort study. BMJ. 2007;
335(7631):1202-1205.
11. Hopewell S, Dutton S, Yu LM, Chan AW, Altman
DG. The quality of reports of randomised trials in 2000
and 2006: comparative study of articles indexed in
PubMed. BMJ. 2010;340:c723.
12. Robinson KA, Dickersin K. Development of a highly
sensitive search strategy for the retrieval of reports of
controlled trials using PubMed. Int J Epidemiol. 2002;
31(1):150-153.
13. Horton R. The rhetoric of research. BMJ. 1995;
310(6985):985-987.
14. Boutron I, Guittet L, Estellat C, Moher D,
Hro´ bjartsson A, Ravaud P. Reporting methods of blind-
ing in randomized trials assessing nonpharmacologi-
cal treatments. PLoS Med. 2007;4(2):e61.
15. Blader JC. Can keeping clinical trial participants
blind to their study treatment adversely affect subse-
quent care? [published online ahead of print March
3, 2005]. Contemp Clin Trials. 2005;26(3):290-
299. doi:10.1016/j.cct.2005.01.003.
16. Al-Marzouki S, Roberts I, Marshall T, Evans S. The
effect of scientific misconduct on the results of clini-
cal trials: a Delphi survey. Contemp Clin Trials. 2005;
26(3):331-337.
17. Gøtzsche PC. Believability of relative risks and odds
ratios in abstracts: cross sectional study. BMJ. 2006;
333(7561):231-234.
18. Jørgensen KJ, Johansen HK, Gøtzsche PC. Flaws
in design, analysis and interpretation of Pfizer’s anti-
fungal trials of voriconazole and uncritical subse-
quent quotations. Trials. 2006;7:3.
19. Hoekstra R, Finch S, Kiers HA, Johnson A. Prob-
ability as certainty: dichotomous thinking and the mis-
use of p values. Psychon Bull Rev. 2006;13(6):
1033-1037.
20. Zinsmeister AR, Connor JT. Ten common statis-
tical errors and how to avoid them. Am J Gastroenterol.
2008;103(2):262-266.
21. Pocock SJ, Ware JH. Translating statistical find-
ings into plain English. Lancet. 2009;373(9679):
1926-1928.
22. Wang R, Lagakos SW, Ware JH, Hunter DJ, Drazen
JM. Statistics in medicine—reporting of subgroup analy-
ses in clinical trials. N Engl J Med. 2007;357(21):
2189-2194.
23. Mathieu S, Boutron I, Moher D, Altman DG,
Ravaud P. Comparison of registered and published pri-
mary outcomes in randomized controlled trials. JAMA.
2009;302(9):977-984.
24. Rattinger G, Bero L. Factors associated with re-
sults and conclusions of trials of thiazolidinediones. PLoS
One. 2009;4(6):e5826.
25. Als-Nielsen B, Chen W, Gluud C, Kjaergard LL.
Association of funding and conclusions in random-
ized drug trials: a reflection of treatment effect or ad-
verse events? JAMA. 2003;290(7):921-928.
26. Jørgensen AW, Hilden J, Gøtzsche PC. Coch-
rane reviews compared with industry supported meta-
analyses and other meta-analyses of the same drugs:
systematic review. BMJ. 2006;333(7572):782.
27. Ioannidis JP. Limitations are not properly acknowl-
edged in the scientific literature. J Clin Epidemiol. 2007;
60(4):324-329.
28. Matthews JN, Altman DG. Interaction 2: com-
pare effect sizes not P values. BMJ. 1996;313(7060):
808.
29. Moyer CA. Between-groups study designs de-
mand between-groups analyses: a response to
Hernandez-Reif, Shor-Posner, Baez, Soto, Mendoza,
Castillo, Quintero, Perez, and Zhang. Evid Based
Complement Alternat Med. 2009;6(1):49-50.
30. Hopewell S, Clarke M, Moher D, et al; CONSORT
Group. CONSORT for reporting randomized controlled
trials in journal and conference abstracts: explanation and
elaboration. PLoS Med. 2008;5(1):e20.
31. Horton R. The hidden research paper. JAMA. 2002;
287(21):2775-2778.
32. Kaptchuk TJ. Effect of interpretive bias on research
evidence. BMJ. 2003;326(7404):1453-1455.
33. Dwan K, Altman DG, Arnaiz JA, et al. Systematic
review of the empirical evidence of study publication
bias and outcome reporting bias. PLoS One. 2008;
3(8):e3081.
34. Rising K, Bacchetti P, Bero L. Reporting bias in drug
trials submitted to the Food and Drug Administra-
tion: review of publication and presentation. PLoS Med.
2008;5(11):e217.
35. Bero L, Oostvogel F, Bacchetti P, Lee K. Factors
associated with findings of published trials of drug-
drug comparisons: why some statins appear more ef-
ficacious than others. PLoS Med. 2007;4(6):e184.
36. Woloshin S, Schwartz LM, Casella SL, Kennedy
AT, Larson RJ. Press releases by academic medical cen-
ters: not so academic? Ann Intern Med. 2009;
150(9):613-618.
37. Bombardier C, Laine L, Reicin A, et al; VIGOR Study
Group. Comparison of upper gastrointestinal toxicity of
rofecoxib and naproxen in patients with rheumatoid
arthritis. N Engl J Med. 2000;343(21):1520-1528.
38. Krumholz HM, Ross JS, Presler AH, Egilman DS.
What have we learnt from Vioxx? BMJ. 2007;
334(7585):120-123.
39. McGettigan P, Sly K, O’Connell D, Hill S, Henry
D. The effects of information framing on the prac-
tices of physicians. J Gen Intern Med. 1999;14
(10):633-642.
40. Bucher HC, Weinbacher M, Gyr K. Influence of
method of reporting study results on decision of phy-
sicians to prescribe drugs to lower cholesterol
concentration. BMJ. 1994;309(6957):761-764.
41. Chalmers I, Matthews R. What are the implica-
tions of optimism bias in clinical research? Lancet. 2006;
367(9509):449-450.
42. Turner EH, Matthews AM, Linardatos E, Tell RA,
Rosenthal R. Selective publication of antidepressant
trials and its influence on apparent efficacy. N Engl J
Med. 2008;358(3):252-260.
43. Chan AW, Altman DG. Identifying outcome re-
porting bias in randomised trials on PubMed: review
of publications and survey of authors. BMJ. 2005;
330(7494):753.
44. Chan AW, Hro´ bjartsson A, Haahr MT, Gøtzsche
PC, Altman DG. Empirical evidence for selective re-
porting of outcomes in randomized trials: compari-
son of protocols to published articles. JAMA. 2004;
291(20):2457-2465.
45. Al-Marzouki S, Roberts I, Evans S, Marshall T. Se-
lective reporting in clinical trials: analysis of trial pro-
tocols accepted by The Lancet. Lancet. 2008;372
(9634):201.
46. Song F, Parekh S, Hooper L, et al. Dissemination
and publication of research findings: an updated re-
view of related biases. Health Technol Assess. 2010;
14(8):iii, ix-xi, 1-193.
DISTORTED PRIMARY OUTCOMES PRESENTATION IN RCTS
2064 JAMA, May 26, 2010—Vol 303, No. 20 (Reprinted) ©2010 American Medical Association. All rights reserved.
at Yale University on May 26, 2010 www.jama.comDownloaded from
... Linguistic spin was defined as the (un)conscious use of language to convince readers of the beneficial effect of the intervention and or to suppress negative results (28)(29)(30). We contacted various authors with expertise on (linguistic) spin by e-mail, aiming to find a database of linguistic spin examples. ...
... To minimize this partial subjectivity in the interpretation of linguistic spin, two authors independently completed the data extraction. Second, we defined linguistic spin as the (un)conscious use of language to convince readers of the beneficial effect of the intervention and or to suppress negative results, which is in line with the definition of spin stated in previous literature (28)(29)(30). However, there may be examples of studies in which linguistic spin has been used to emphasize the disadvantageous effect of the treatment investigated. ...
Article
Full-text available
Objective To evaluate the prevalence, type and extent of linguistic spin in randomized controlled trials (RCTs) exploring interventions in patients with age-related macular degeneration (AMD), as well as to investigate whether study variables were correlated with linguistic spin. Study design and setting PubMed was searched from 2011 to 2020 to identify RCTs including patients with AMD. Two authors independently assessed a total of 96 RCTs. Linear regression analyses were performed to investigate whether linguistic spin was correlated with predefined study variables. Results Linguistic spin was found in 61 of 96 abstracts (63.5%) and in 90 of 96 main texts (93.8%). Use of words pointing out the beneficial effect of a treatment and the use of ‘(statistically) significant/significance' without reporting a P -value or a 95% confidence interval (CI) were the most frequently identified categories of linguistic spin. Sample size was significantly correlated with the total linguistic spin score (95% CI 0.38–5.23, P = 0.02). Conclusion A high prevalence and extent of linguistic spin in RCTs about AMD was found. We highlighted the importance of objective reporting and awareness of linguistic spin among ophthalmologists and other readers.
... Even if the field "RECOMMENDATIONS OF THIS PAPER" corresponds to that, it may be misleading, and the comments are far too short to be of any scholarly value. Some such commentaries may foster "spin", unfairly over-emphasizing the strengths of the paper while misrepresenting its true value to the reader (Boutron et al., 2010). ...
Article
This paper makes a historical assessment of a publishing "experiment" that started in 2020 and ended in 2022 by Academia.edu, a popular academic social network site, that took the form of a peer-reviewed "journal", Academia Letters. We discovered a publicly hidden open access cost, as an article processing charge of US$ 500, some inconsistencies or ambiguities in select editorial policies, the lack of an editorial board, and the absence of an integrity and publishing ethics policy, cumulatively indicating that this publishing model was lacking some basic robust scholarly indices that are typically found in conventional peer-reviewed journals. Despite its short two-year history, about 4500 papers were published in Academia Letters, suggesting that this publishing model was nonetheless attractive and popular. This overview of Academia Letters will allow Academia.edu and other academic publishers to reflect on specifics or weaknesses of this publishing model before using it in the future to ensure trustworthy scholarly communication in the academic community.
... "Spin" was defined as authors highlighting the result of secondary or explanatory analyses despite a non-significant result for primary outcomes. The definition of "spin" did not include a reporting strategy intended to distract the reader from a non-significant result, unlike in previous literature [12]. A.S. and one of the coauthors (N.Y., N.S., M.O., or H.S.) independently assessed the presence of spin and reached a consensus through discussion. ...
Article
Full-text available
Our meta-epidemiological study aimed to describe the prevalence of reporting effect modification only on relative scale outcomes and inappropriate interpretations of the coefficient of interaction terms in nonlinear models on categorical outcomes. Our study targeted articles published in the top 10 high-impact-factor journals between January 1 and December 31, 2021. We included two-arm parallel-group interventional superiority randomized controlled trials to evaluate the effects of modifications on categorical outcomes. The primary outcomes were the prevalence of reporting effect modifications only on relative scale outcomes and that of inappropriately interpreting the coefficient of interaction terms in nonlinear models on categorical outcomes. We included 52 articles, of which 41 (79%) used nonlinear regression to evaluate effect modifications. At least 45/52 articles (87%) reported effect modifications based only on relative scale outcomes, and at least 39/41 (95%) articles inappropriately interpreted the coefficient of interaction terms merely as indices of effect modifications. The quality of evaluating effect modifications in nonlinear models on categorical outcomes was relatively low, even in randomized controlled trials published in medical journals with high impact factors. Researchers should report effect modifications of both absolute and relative scale outcomes and avoid interpreting the coefficients of interaction terms in nonlinear regression analyses.
... The publication of selected results has been associated with industry sponsorship because of the risks of publication, reporting, and selection bias. 31,52 Therefore, the results of this study should be interpreted with caution, and future studies should be conducted to evaluate the effect of sponsorship bias among studies evaluating the effectiveness of PRF. ...
Article
Full-text available
This systematic review evaluated the potential utility of platelet-rich fibrin (PRF) in bone repair in animals. The question is: can the use of PRF in bone defects in healthy rats induce bone repair compared to clot? This systematic review was conducted in accordance with the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (Prisma). The protocol was registered with Prospero (CRD [42020162319]). The literature search involved nine databases, including grey literature. All studies evaluated the bone defects created in rats filled with PRF and clots (control). Biomaterial evaluation was also performed in this study. The risk of bias was assessed using the Systematic Review Center for Laboratory Animal Experimentation (Syrcle) tool for animal studies. A meta-analysis of quantitative data was performed to estimate the effect of PRF on bone repair in rats. Heterogeneity among the studies was assessed using the I2 statistic. The literature search retrieved 685 studies, 10 of which fulfilled the eligibility criteria, and 4 were included in the quantitative assessment. Analysis of the risk of bias revealed that most studies had a high risk of bias in performance and detection. Meta-analysis yielded divergent results and the absence of a statistically significant effect: PRF with control (standardized mean difference 2.54, 95% confidence interval -0.80-5.89; p = 0.14). In general, study heterogeneity was high (I2 ≥ 75.0%). The quality of the studies that influenced the conclusion of the review was based on the PICO, the sources and form of the search, the study selection criteria, the form of evaluation of publication bias, the evaluation of the quality of the studies, and data extraction by two researchers. PRF did not provide significant benefits for bone repair, resulting in unpredictable effects.
... In clinical trials, a report with spin might be framed to convey that a treatment is beneficial, despite little or no objective evidence in support of that conclusion [73]. Such spin is prevalent in scientific publishing and reporting [71,74,75], leading to a substantial distortion of research results. This is particularly concerning for clinical trials, where misled patients might proceed with therapies or interventions that are unsupported by science. ...
Article
Full-text available
When we believe misinformation, we have succumbed to an illusion: our perception or interpretation of the world does not match reality. We often trust misinformation for reasons that are unrelated to an objective, critical interpretation of the available data: Key facts go unnoticed or unreported. Overwhelming information prevents the formulation of alternative explanations. Statements become more believable every time they are repeated. Events are reframed or given “spin” to mislead audiences. In magic shows, illusionists apply similar techniques to convince spectators that false and even seemingly impossible events have happened. Yet, many magicians are “honest liars,” asking audiences to suspend their disbelief only during the performance, for the sole purpose of entertainment. Magic misdirection has been studied in the lab for over a century. Psychological research has sought to understand magic from a scientific perspective and to apply the tools of magic to the understanding of cognitive and perceptual processes. More recently, neuroscientific investigations have also explored the relationship between magic illusions and their underlying brain mechanisms. We propose that the insights gained from such studies can be applied to understanding the prevalence and success of misinformation. Here, we review some of the common factors in how people experience magic during a performance and are subject to misinformation in their daily lives. Considering these factors will be important in reducing misinformation and encouraging critical thinking in society.
Article
Full-text available
In their 4000-year history, abstracts have taken several forms and represented a variety of documents. The scientific journal emerged in the 1600s and gave rise to what would become the modern scientific abstract. Here, I describe the contexts in which abstracts evolved, address the subtexts of opinions about their purpose, and review the texts of 12 kinds of abstracts. For most readers, articles do not exist beyond abstracts. However, the quality of abstracts is often poor. Inaccuracies are common, serious, widespread, and long-standing. Abstracts should inform only the choice of what to read and never what to do.
Article
Full-text available
Objective This study examined the extent to which trials presented at major international medical conferences in 2016 consistently reported their study design, end points and results across conference abstracts, published article abstracts and press releases. Design Cross-sectional analysis of clinical trials presented at 12 major medical conferences in the USA in 2016. Conferences were identified from a list of the largest clinical research meetings aggregated by the Healthcare Convention and Exhibitors Association and were included if their abstracts were publicly available. From these conferences, all late-breaker clinical trials were included, as well as a random selection of all other clinical trials, such that the total sample included up to 25 trial abstracts per conference. Main outcome measures First, it was determined if trials were registered and reported results in an International Committee of Medical Journal Editors-approved clinical trial registry. Second, it was determined if trial results were published in a peer-reviewed journal. Finally, information on trial media coverage and press releases was collected using LexisNexis. For all published trials, the consistency of reporting of the following characteristics was examined, through comparison of the trials’ conference and publication abstracts: primary efficacy endpoint definition, safety endpoint identification, sample size, follow-up period, primary end point effect size and characterisation of trial results. For all published abstracts with press releases, the characterisation of trial results across conference abstracts, press releases and publications was compared. Authors determined consistency of reporting when identical information was presented across abstracts and press releases. Primary analyses were descriptive; secondary analyses included χ ² tests and multiple logistic regression. Results Among 240 clinical trials presented at 12 major medical conferences, 208 (86.7%) were registered, 95 (39.6%) reported summary results in a registry and 177 (73.8%) were published; 82 (34.2%) were covered by the media and 68 (28.3%) had press releases. Among the 177 published trials, 171 (96.6%) reported the definition of primary efficacy endpoints consistently across conference and publication abstracts, whereas 96/128 (75.0%) consistently identified safety endpoints. There were 107/172 (62.2%) trials with consistent sample sizes across conference and publication abstracts, 101/137 (73.7%) that reported their follow-up periods consistently, 92/175 (52.6%) that described their effect sizes consistently and 157/175 (89.7%) that characterised their results consistently. Among the trials that were published and had press releases, 32/32 (100%) characterised their results consistently across conference abstracts, press releases and publication abstracts. No trial characteristics were associated with reporting primary efficacy end points consistently. Conclusions For clinical trials presented at major medical conferences, primary efficacy endpoint definitions were consistently reported and results were consistently characterised across conference abstracts, registry entries and publication abstracts; consistency rates were lower for sample sizes, follow-up periods, and effect size estimates. Registration This study was registered at the Open Science Framework ( https://doi.org/10.17605/OSF.IO/VGXZY ).
Article
Full-text available
Research must be well designed, properly conducted and clearly and transparently reported. Our independent medical research institute wanted a simple, generic tool to assess the quality of the research conducted by its researchers, with the goal of identifying areas that could be improved through targeted educational activities. Unfortunately, none was available, thus we devised our own. Here, we report development of the Quality Output Checklist and Content Assessment (QuOCCA), and its application to publications from our institute's scientists. Following consensus meetings and external review by statistical and methodological experts, 11 items were selected for the final version of the QuOCCA: research transparency (items 1-3), research design and analysis (items 4-6) and research reporting practices (items 7-11). Five pairs of raters assessed all 231 articles published in 2017 and 221 in 2018 by researchers at our institute. Overall, the results were similar between years and revealed limited engagement with several recommended practices highlighted in the QuOCCA. These results will be useful to guide educational initiatives and their effectiveness. The QuOCCA is brief and focuses on broadly applicable and relevant concepts to open, high-quality, reproducible and well-reported science. Thus, the QuOCCA could be used by other biomedical institutions and individual researchers to evaluate research publications, assess changes in research practice over time and guide the discussion about high-quality, open science. Given its generic nature, the QuOCCA may also be useful in other research disciplines.
Chapter
Evidence‐based medicine (EBM) is the integration of best research evidence with clinical expertise and patient values. Practising EBM consists of five steps: (i) formulating a well‐built clinical question generated from a patient encounter; (ii) searching for valid external evidence; (iii) critically appraising that evidence for relevance and validity; (iv) applying the results of that appraisal of evidence back to the patient; and (v) recording the information for future use. An important principle of EBM is that the quality (strength) of evidence is based on a hierarchy of evidence. The order of the hierarchy of evidence is dependent on the type of question being asked. The hierarchy of evidence consists of results of well‐designed experimental studies such as randomized controlled trials (especially if the studies have results of similar magnitude and direction, and if there is homogeneity among studies), results of observational studies, case series, expert opinion and personal experience, in descending order.
Article
Full-text available
To examine the reporting characteristics and methodological details of randomised trials indexed in PubMed in 2000 and 2006 and assess whether the quality of reporting has improved after publication of the Consolidated Standards of Reporting Trials (CONSORT) Statement in 2001. Comparison of two cross sectional investigations. Study sample All primary reports of randomised trials indexed in PubMed in December 2000 (n=519) and December 2006 (n=616), including parallel group, crossover, cluster, factorial, and split body study designs. The proportion of general and methodological items reported, stratified by year and study design. Risk ratios with 95% confidence intervals were calculated to represent changes in reporting between 2000 and 2006. The majority of trials were two arm (379/519 (73%) in 2000 v 468/616 (76%) in 2006) parallel group studies (383/519 (74%) v 477/616 (78%)) published in specialty journals (482/519 (93%) v 555/616 (90%)). In both 2000 and 2006, a median of 80 participants were recruited per trial for parallel group trials. The proportion of articles that reported drug trials decreased between 2000 and 2006 (from 393/519 (76%) to 356/616 (58%)), whereas the proportion of surgery trials increased (51/519 (10%) v 128/616 (21%)). There was an increase between 2000 and 2006 in the proportion of trial reports that included details of the primary outcome (risk ratio (RR) 1.18, 95% CI 1.04 to 1.33), sample size calculation (RR 1.66, 95% CI 1.40 to 1.95), and the methods of random sequence generation (RR 1.62, 95% CI 1.32 to 1.97) and allocation concealment (RR 1.40, 95% CI 1.11 to 1.76). There was no difference in the proportion of trials that provided specific details on who was blinded (RR 0.91, 95% CI 0.75 to 1.10). Reporting of several important aspects of trial methods improved between 2000 and 2006; however, the quality of reporting remains well below an acceptable level. Without complete and transparent reporting of how a trial was designed and conducted, it is difficult for readers to assess its conduct and validity.
Article
Full-text available
To identify and appraise empirical studies on publication and related biases published since 1998; to assess methods to deal with publication and related biases; and to examine, in a random sample of published systematic reviews, measures taken to prevent, reduce and detect dissemination bias. The main literature search, in August 2008, covered the Cochrane Methodology Register Database, MEDLINE, EMBASE, AMED and CINAHL. In May 2009, PubMed, PsycINFO and OpenSIGLE were also searched. Reference lists of retrieved studies were also examined. In Part I, studies were classified as evidence or method studies and data were extracted according to types of dissemination bias or methods for dealing with it. Evidence from empirical studies was summarised narratively. In Part II, 300 systematic reviews were randomly selected from MEDLINE and the methods used to deal with publication and related biases were assessed. Studies with significant or positive results were more likely to be published than those with non-significant or negative results, thereby confirming findings from a previous HTA report. There was convincing evidence that outcome reporting bias exists and has an impact on the pooled summary in systematic reviews. Studies with significant results tended to be published earlier than studies with non-significant results, and empirical evidence suggests that published studies tended to report a greater treatment effect than those from the grey literature. Exclusion of non-English-language studies appeared to result in a high risk of bias in some areas of research such as complementary and alternative medicine. In a few cases, publication and related biases had a potentially detrimental impact on patients or resource use. Publication bias can be prevented before a literature review (e.g. by prospective registration of trials), or detected during a literature review (e.g. by locating unpublished studies, funnel plot and related tests, sensitivity analysis modelling), or its impact can be minimised after a literature review (e.g. by confirmatory large-scale trials, updating the systematic review). The interpretation of funnel plot and related statistical tests, often used to assess publication bias, was often too simplistic and likely misleading. More sophisticated modelling methods have not been widely used. Compared with systematic reviews published in 1996, recent reviews of health-care interventions were more likely to locate and include non-English-language studies and grey literature or unpublished studies, and to test for publication bias. Dissemination of research findings is likely to be a biased process, although the actual impact of such bias depends on specific circumstances. The prospective registration of clinical trials and the endorsement of reporting guidelines may reduce research dissemination bias in clinical research. In systematic reviews, measures can be taken to minimise the impact of dissemination bias by systematically searching for and including relevant studies that are difficult to access. Statistical methods can be useful for sensitivity analyses. Further research is needed to develop methods for qualitatively assessing the risk of publication bias in systematic reviews, and to evaluate the effect of prospective registration of studies, open access policy and improved publication guidelines.
Article
Full-text available
As of 2005, the International Committee of Medical Journal Editors required investigators to register their trials prior to participant enrollment as a precondition for publishing the trial's findings in member journals. To assess the proportion of registered trials with results recently published in journals with high impact factors; to compare the primary outcomes specified in trial registries with those reported in the published articles; and to determine whether primary outcome reporting bias favored significant outcomes. MEDLINE via PubMed was searched for reports of randomized controlled trials (RCTs) in 3 medical areas (cardiology, rheumatology, and gastroenterology) indexed in 2008 in the 10 general medical journals and specialty journals with the highest impact factors. For each included article, we obtained the trial registration information using a standardized data extraction form. Of the 323 included trials, 147 (45.5%) were adequately registered (ie, registered before the end of the trial, with the primary outcome clearly specified). Trial registration was lacking for 89 published reports (27.6%), 45 trials (13.9%) were registered after the completion of the study, 39 (12%) were registered with no or an unclear description of the primary outcome, and 3 (0.9%) were registered after the completion of the study and had an unclear description of the primary outcome. Among articles with trials adequately registered, 31% (46 of 147) showed some evidence of discrepancies between the outcomes registered and the outcomes published. The influence of these discrepancies could be assessed in only half of them and in these statistically significant results were favored in 82.6% (19 of 23). Comparison of the primary outcomes of RCTs registered with their subsequent publication indicated that selective outcome reporting is prevalent.
Article
Altruism and trust lie at the heart of research on human subjects. Altruistic individuals volunteer for research because they trust that their participation will contribute to improved health for others and that researchers will minimize risks to participants. In return for the altruism and trust that make clinical research possible, the research enterprise has an obligation to conduct research ethically and to report it honestly. Honest reporting begins with revealing the existence of all clinical studies, even those that reflect unfavorably on a research sponsor's product.
Article
Background: Clear, transparent, and sufficiently detailed abstracts of conferences and journal articles related to randomized controlled trials (RCTs) are important, because readers often base their assessment of a trial solely on information in the abstract. Here, we extend the CONSORT (Consolidated Standards of Reporting Trials) Statement to develop a minimum list of essential items, which authors should consider when reporting the results of a RCT in any journal or conference abstract.Methods and Findings: We generated a list of items from existing quality assessment tools and empirical evidence. A three-round, modified-Delphi process was used to select items. In all, 109 participants were invited to participate in an electronic survey; the response rate was 61%. Survey results were presented at a meeting of the CONSORT Group in Montebello, Canada, January 2007, involving 26 participants, including clinical trialists, statisticians, epidemiologists, and biomedical editors. Checklist items were discussed for eligibility into the final checklist. The checklist was then revised to ensure that it reflected discussions held during and subsequent to the meeting. CONSORT for Abstracts recommends that abstracts relating to RCTs have a structured format. Items should include details of trial objectives; trial design (e.g., method of allocation, blinding/masking); trial participants (i.e., description, numbers randomized, and number analyzed); interventions intended for each randomized group and their impact on primary efficacy outcomes and harms; trial conclusions; trial registration name and number; and source of funding. We recommend the checklist be used in conjunction with this explanatory document, which includes examples of good reporting, rationale, and evidence, when available, for the inclusion of each item.Conclusions: CONSORT for Abstracts aims to improve reporting of abstracts of RCTs published in journal articles and conference proceedings. It will help authors of abstracts of these trials provide the detail and clarity needed by readers wishing to assess a trial's validity and the applicability of its results.