ABSTRACT – Systematic reviews of acupuncture
have tended to support its use, but few applied
rigorous inclusion criteria. We tested the credi-
bility of conclusions of systematic reviews of
acupuncture published since 1996 by applying
rigorous inclusion criteria. Reinterpretation used
randomised and double blind trials with valid
outcomes or design, and with information avail-
able from at least four trials or from 200 patients.
Qualified support for acupuncture was originally
reported in 12 out of 35 systematic reviews, and
strong support was found in another six.
Applying stricter inclusion criteria, however,
showed that none of the 35 reviews supported
acupuncture, predominantly because there were
too few patients in the randomised, double blind
studies. Six reviews with more than 200 patients
in randomised, double blind studies had good
evidence of no benefit. Systematic reviews of
acupuncture have overstated effectiveness by
including studies likely to be biased. They pro-
vide no robust evidence that acupuncture works
for any indication.
KEY WORDS: acupuncture, bias, systematic
The place of acupuncture and other complementary
therapies in mainstream healthcare is controversial.
Systematic reviews and meta-analyses have claimed
that these therapies can be distinguished from con-
trols,1and many people are turning to alternative
therapies, including acupuncture.2One in five
European adults have chronic pain, and 13% of them
use or have used acupuncture.3A Department of
Health publication for primary care clinicians in
2000 claimed that there was reasonable evidence of
efficacy for acupuncture in chronic lower back pain.4
Unfortunately many systematic reviews and meta-
analyses of complementary therapies like acupunc-
ture have included clinical trials whose design is not
randomised or blinded. Reanalysis of a systematic
review of homeopathy,5 using more rigorous inclu-
sion criteria for trials, like randomisation and
blinding, gave a reduced estimate of effectiveness.6A
systematic review of systematic reviews of home-
opathy found no convincing evidence for efficacy in
Trials of acupuncture are frequently small, use out-
comes of little practical relevance, or are invalid
for some reason (short duration trials in chronic
conditions, for example). These factors all affect the
credibility of the findings. Our aim was to examine
systematic reviews of acupuncture published in the
last decade, and to compare the authors’ original
conclusions with our own after applying criteria of
quality, validity and size.
The aim was not to review the efficacy of acupunc-
ture, but rather the efficacy of systematic reviews
to accurately assess the evidence for acupuncture,
using criteria for inclusion of trials that are known to
minimise the possibility of bias, and which are used
commonly in systematic reviews of most other
We searched PubMed, the Allied and Complementary
Medicine Database (AMED), and the Cochrane
Library for systematic reviews of acupuncture for any
I THE SCIENTIFIC BASIS FOR ALTERNATIVE MEDICINE
Clinical Medicine Vol 6 No 4 July/August 2006381
CJ Derry, Student
S Derry MA, Senior
HJ McQuay DM
Professor of Pain
RA Moore DSc
FRSC, Director of
Pain Research and
Systematic review of systematic reviews of
acupuncture published 1996–2005
CJ Derry, S Derry, HJ McQuay and RA Moore
THE SCIENTIFIC BASIS FOR ALTERNATIVE MEDICINE
There is a burgeoning interest in many forms of alternative medicine and
growing enthusiasm for providing these alternative approaches within the NHS
framework. The increasing use of alternative medicine may reflect unhappiness
with some aspects of scientific medicine and perhaps particularly our inability
to resolve non-specific aches and pains, general unhappiness or a loss of joie
de vivre. The perceived failure may be associated with an unrealistic
expectation of perfect health. Sadly, few can reach the state of happiness
portrayed in much commercial advertising.
Time allocated to a constructive discussion of symptoms and associated
concerns may well have a beneficial effect and may contribute in part to the
important and beneficial effect of the placebo response. Is alternative medicine
merely a revival of the former approach by general practitioners where
unhurried consultation and discussion followed by a prescription of either the
red or green medicine was effective for its placebo effect? Could payment for
the services of alternative medicine in itself be an encouragement to feel
better? The benefits of constructive discussion and the time to offer support
and guidance are clear. The key question is whether in addition there is
scientific evidence for the benefit of the procedure itself.
In the first of this series, the evidence for the scientific benefits for
acupuncture is considered by the Pain Research and Nuffield Department of
Anaesthetics, The Churchill, Oxford.
condition in humans, published from January 1996 to August
2005, using the terms ‘acupuncture’ and ‘systematic OR meta-
analysis’. We also looked for relevant reviews in our own in-house
databases and reference lists of retrieved articles.
We accepted reviews published in English that examined the
efficacy of traditional Chinese or mechanical acupuncture,
electro-acupuncture, laser acupuncture or acupressure, elec-
trical nerve stimulation but not transcutaneous (TENS) or dry
needling (using empty hypodermic needles or acupuncture nee-
dles at trigger points for myofascial pain). Where one review
clearly updated a previous review, only the most recent publica-
tion was used. If more than one review covered the same trials
for the same outcome and indication, the most recent was taken.
Reviews of adverse effects of acupuncture were not included.8
Two reviewers extracted the following information from each
sensitivity analysis presented
on their assessment of whether there was no evidence of
benefit, qualified support, or strong support)
number of studies included
number of patients included
whether there was assessment of quality for included studies
whether exclusions due to poor quality were made, or a
main findings, including whether a pooled analysis was done
original authors’ conclusions on efficacy
original authors’ conclusions on strength of evidence (based
authors’ affiliation to complementary medicine depart-
In addition, we made our own assessment of the strength of
the evidence presented in each review. We specified a priori
criteria for quality, validity and size to remove studies that were
susceptible to bias, and might leave uncertainty in the results. To
be sure that any beneficial or harmful effect of acupuncture is
not an artefact of biased design, several factors are important.
Trials need to:
blind to intervention), with appropriate controls. Trials that
are not properly randomised and double blind have been
shown to overestimate treatment effects. For trials of
acupuncture, the practitioner cannot be blinded to the
treatment groups. The most appropriate design uses sham
acupuncture and outcome assessors blinded to treatment
group. During ‘sham acupuncture’, needles are inserted at
non-specific acupuncture points and usually penetrate the
skin only a few millimetres, or needles are used that only
indent the skin. Sham acupuncture involves all the various
aspects of acupuncture, including practitioner’s time and
attention, and is not equivalent to an inert placebo. Waiting
list controls, where patients remain on a waiting list, with no
treatment and no practitioner’s time or attention, cannot be
blinded to their treatment group; they have no treatment of
example experimental pain), have groups comparable at
be randomised and double blind (both patient and assessor
study patients with recognised clinical conditions (not for
baseline, and use relevant outcomes (eg patient pain scoring,
number of headache-free days, long-term outcomes for
study sufficient numbers of patients to minimise effects of
chance. We know that small studies, or large studies with
small numbers of events, can be affected by the random play
of chance and mislead results.9–14How much information is
needed to be sure of a result depends on how sure one wants
to be, and how large is the effect. For large effects (50%
absolute risk increase, for instance) we need about 400
patients or 200 events;11where the absolute risk increase is
small, at below 20%, the number of patients needed in trials
rises to the thousands. To set some sensible lower limit for
numbers of patients involved in trials of sufficient quality
and validity to make it worth trying to calculate statistical
significance, we arbitrarily specified four trials and/or 200
patients as a minimum.
Relative benefit (or risk) was calculated with 95% confidence
intervals using a random effects model,15 with no statistically
significant difference between treatments assumed when the
95% confidence intervals included unity.
We found 55 systematic reviews satisfying our inclusion criteria,
of which 20 were excluded because a more recent review covered
the same topic and included the earlier studies (n=17), or
because they were not in English (n=3). Full details of the
reviews, authors’ and reviewers’ conclusions, and references for
included and excluded studies are presented in two supplemen-
tary files available from the authors upon request. The 35
included studies1,16–49examined the use of acupuncture in var-
ious painful conditions (n=18), stroke (n=2), nausea and vom-
iting (n=2), depression (n=2), and other conditions including
insomnia, smoking cessation, weight loss, and asthma (n=11).
All but four of the 35 reviews made a formal assessment of
methodological quality of included studies. Most of the reviews
(22/35) claimed to use only randomised studies, and most had
included trials that were not both patient and assessor blind. No
review excluded trials from analysis because of low quality,
though a small number performed sensitivity analysis according
to study quality.
Most reviews (24/35) had information on fewer than 1,000
patients (Fig 1). However, the number of patients contributing
to analysis of efficacy was often substantially smaller than the
total number of patients in all the trials included in a review,
because many reviews included studies with no relevant efficacy
Most reviews commented upon the relevance of the chosen
outcomes and the validity of trials. Some were unclear about
what effect they were reporting, and some reported inappro-
priate outcomes, especially short-term outcomes for chronic
conditions. Waiting list controls were sometimes used.
Of the 35 reviews, 17 concluded that there was either no evi-
dence of benefit, or evidence of no benefit. Twelve had a qualified
CJ Derry, S Derry, HJ McQuay and RA Moore
Clinical Medicine Vol 6 No 4 July/August 2006
conclusion of some benefit for acupuncture, with authors com-
menting on issues around small trial size or poor methodology,
or the need for further research, but still using words, usually in
the abstract or conclusion, supporting the use of acupuncture.
For instance, a Cochrane review of acupuncture for idiopathic
headache maintained, ‘the existing evidence supports the value of
acupuncture for the treatment of idiopathic headaches. However,
the quality and amount of evidence are not fully convincing’.33A
second commented that acupuncture ‘may be beneficial to reduce
symptomatic knee pain’ though ‘reviewers concluded that the
poor quality of the trials, including the small sample size proclude
[sic] its recommendation’.17
Of the 35 reviews, six had an authors’ strong conclusion of
benefit (Table 1) meaning that authors made comments like
‘acupuncture effectively relieves chronic low back pain’,31 ‘sup-
ports the use of P6 acupoint stimulation in patients without
antiemetic prophylaxis’,26or ‘there is strong evidence suggesting
that acupuncture is effective in the short term for lateral
Of the 18 reviews with qualified or strong support for
acupuncture, ten came from departments connected with com-
plementary therapy. Five of the six studies with strong support
were from departments connected with complementary
therapy. Of the 17 reviews showing no benefit, eight were affili-
ated to departments of complementary medicine. Cochrane
reviews were less likely to support acupuncture, though one gave
strong support and three qualified support (Table 1).
Our assessment was that none of these 35 systematic reviews
could demonstrate robust evidence of effectiveness for acupunc-
ture when strict criteria of quality, validity, and size were used to
judge the evidence. Using criteria known to reduce the possi-
bility of bias, commonly used in systematic reviews assessing
medical interventions, most acupuncture reviews had trivial
amounts of good quality evidence. Only six had more than 200
patients in randomised, double blind trials, and in these
acupuncture was not significantly better than control.
The authors of six reviews (not the same six) made strong
claims of benefit that were not upheld in our evaluation of
quality, size and validity, and it is useful to examine these six in
of acupuncture for back pain, and provided short-term
outcomes.20 Five of these trials were not blind, and had a
statistically significant benefit (relative benefit 1.8; 95%
Ernst and White included nine randomised trials in a review
confidence interval 1.3 to 2.4). The four blind studies
(n=173) showed no significant benefit (relative benefit 1.2;
0.9 to 1.5). All of the benefit of acupuncture reported in the
review derived from the non-blind studies.
Ezzo et al included seven randomised trials of acupuncture
for knee osteoarthritis, of which three were high quality
trials using sham acupuncture (n=174).1Of these, only one
trial (103 patients) showed consistent benefit for all short-
term pain outcomes, with no benefit for function outcomes.
Two trials reported longer-term (3 months) findings but
with conflicting results.
Trinh et al included six studies to conclude that there was
strong evidence to support acupuncture for lateral
epicondyle pain.46 Of these six trials, one was not properly
randomised. Of the remaining five trials, two were not
double blind. Of the remaining three (n=175), one had
results only immediately after treatment. That left two
randomised, double blind trials, reporting valid outcomes at
two or three months after treatment. Both compared real
acupuncture with sham acupuncture, with a pooled relative
benefit of 1.2 (0.96 to 1.6), indicating no benefit.
Lee and Ernst included six studies of patients undergoing
endoscopy, only two of which were properly randomised
and blinded (n=120).28There was no meta-analysis because
of different outcome measures. Some measures of
discomfort were reduced in the acupuncture groups, and in
one trial additional sedative use was also reduced (n=10),
while in the other trial pain in some areas was reduced
Mannheimer included 33 randomised trials of acupuncture
in low back pain, only four of which (n=343) had sham
Systematic review of systematic reviews of acupuncture published 1996–2005
Clinical Medicine Vol 6 No 4 July/August 2006383
Fig 1. Total number of patients included in any
review, though not necessarily in all analyses.
<500 –1,000 –1,500–2,000
Total patients included in review
Number of reviews
Table 1. Support by original authors by type of review and
by affiliation to department of complementary therapy.
Authors’ support for acupuncture
All studies (n=35)
acupuncture controls and contributed data on pain.31
Different pain outcomes were pooled, and in four trials
short-term outcomes (about three weeks) were significantly
improved for true, compared with sham, acupuncture.
Longer-term outcomes were not significantly different in the
two trials reporting them.
The largest number of patients studied was for the use of P6
acupoint stimulation for preventing postoperative nausea
and vomiting.26The original analysis used randomisation as
the only quality criterion. We performed a sensitivity
analysis to investigate the effect of criteria of quality, validity
and size on the strength of evidence for the different
outcomes (Table 2). The use of increasingly stringent criteria
reduced or eliminated statistical significance of benefit for
nausea, vomiting, and antiemetic consumption.
The 35 systematic reviews of acupuncture published since 1995
represent what should be the highest level of evidence available.
Unfortunately, most of the reviews were based on a few small
trials of inadequate design and statistical power. Many reviews
included studies with designs known to be associated with bias
and overestimation of treatment effects, notably trials that were
not randomised, not blind, or neither randomised nor blind.
Pooled analysis of trials with flawed design does not resolve, but
rather accentuates, these problems.
In no case did reanalysis using only trials that fulfilled min-
imal criteria (randomisation, blinding, size and validity) sup-
port a strong conclusion of benefit from acupuncture. Perceived
benefits of acupuncture were derived from low quality trials
likely to be biased. The best statistical claim was for relief of back
pain after three months,31and then with limited information
and from pooling different pain outcomes. The example with
most information was acupressure for postoperative nausea and
vomiting (Table 2).26Sequential elimination of non-blind trials,
small trials, and trials insensitive because of low event rates
excluded most trials, leaving a conclusion of no effect for
acupressure in the remainder, irrespective of how statistical
significance was calculated.
Cochrane reviews were less likely to support acupuncture than
other reviews. Reviewers without affiliation to a department of
complementary medicine were less likely to support acupunc-
ture than those that had such affiliation. The sample in this
review of reviews is probably too small to draw any conclusions
Several larger well-reported randomised trials of acupuncture
have been published since these reviews. Those in fibro-
myalgia,50chemotherapy-induced nausea and vomiting,51
breech presentation,52tension headache53and migraine,54have
all been negative compared with sham acupuncture controls.
One in osteoarthritis of the knee, had statistical improvement
over sham acupuncture at three months, but not later.55The
large trials and this review of reviews come to the same general
conclusion; that over a whole range of conditions and outcomes
acupuncture cannot yet be shown to be effective.
This negative view of acupuncture after a decade of primary
and secondary research may not be wholly justified. For
instance, a commentary56on the trial of acupuncture for knee
arthritis suggested that longer term physical functioning differ-
ences might be a more important outcome than pain, because
that and other trials showed effects (but not pain relief) lasting
well beyond the use of acupuncture, compared with sham
acupuncture. It may be that we have to look at different out-
comes. Again, trials that have included both waiting list and
sham acupuncture control groups tend consistently to show
benefit for acupuncture and sham acupuncture over waiting list,
suggesting that needles convey only a part of the benefits.56
CJ Derry, S Derry, HJ McQuay and RA Moore
Clinical Medicine Vol 6 No 4 July/August 2006
Systematic reviews have tended to support the use of
They have tended to use trials with known sources of bias
Excluding trials with known sources of bias, no systematic
review had evidence of efficacy
There is no robust evidence from systematic reviews that
acupuncture works in any indication
Table 2. Post-operative nausea and vomiting, with application of increasingly more stringent criteria for quality, size and
(95% CI)Sensitivity analysisTrials/patients Trials/patientsTrials/patients
Randomised16/1,8260.73 (0.57 to 0.93)20/2,187 0.71 (0.56 to 0.91)15/1,4920.79 (0.61 to 1.02)
Randomised and blind10/1,1500.78 (0.58 to 1.05)12/1,328 0.84 (0.62 to 1.14)10/1,0480.83 (0.64 to 1.09)
Randomised, blind, over 100 pts5/8850.82 (0.58 to 1.17)6/9880.79 (0.55 to 1.14)6/8480.83 (0.62 to 1.11)
Randomised, blind, over 100 pts,
control event rate ≥20%
5/8850.82 (0.58 to 1.17)4/5670.76 (0.54 to 1.05)3/4440.89 (0.69 to 1.16)
Control event rate is the percentage of patients who had an emetic event (nausea, vomiting, or use of antiemetic medication) in the control group receiving placebo.
Low control event rates limit the sensitivity of trials in antiemesis.
Future studies might usefully investigate which aspects of the
‘acupuncture experience’ could give rise to this observed benefit.
The possibility of some small but clinically useful benefit cannot
be excluded on the basis of the evidence to hand.
It has been argued14that most published research findings are
false, and that confirmatory meta-analyses of good quality ran-
domised trials provide the least likelihood of being wrong. This
review of reviews supports this, at least for acupuncture, where
the effort seems to have been to find statistical benefits, irre-
spective of quality. The trouble is that statistical significance
does not necessarily translate into clinical benefit.
The lack of evidence makes for problems for those providing
acupuncture services, and for regulators. It is also a problem for
purchasers of healthcare. Private individuals can please them-
selves. Public or private bodies that have previously purchased
acupuncture may have to reconsider. For acupuncture and other
alternative therapies, there has been a climate of permissive
endorsement made on the basis of perceived low risk rather than
evidence of efficacy. At what point will this view change in the
face of mounting evidence of lack of evidence of efficacy?
The study was supported by the Oxford Pain Relief Trust, which
had no involvement in any aspect of the original idea or the
design, execution or writing of the review, nor in the decision to
submit the paper for publication.
1 Ezzo J, Hadhazy V, Birch S et al. Acupuncture for osteoarthritis of the
knee: a systematic review. Arthritis Rheum 2001;44:819–25.
Eisenberg DM, Davis RB, Ettner SL et al. Trends in alternative medicine
use in the United States, 1990–1997: results of a follow-up national
survey. JAMA 1998;280:1569–75.
Breivik H, Collett B, Ventafridda V, Cohen R, Gallacher D. Survey of
chronic pain in Europe: prevalence, impact on daily life, and treatment.
Eur J Pain 2006;10:287–333.
Department of Health. Complementary medicine. London: DH, 2000.
Linde K, Clausius N, Ramirez G et al. Are the clinical effects of home-
opathy placebo effects? A meta-analysis of placebo-controlled trials.
Ernst E, Pittler MH. Re-analysis of previous meta-analysis of clinical
trials of homeopathy. J Clin Epidemiol 2000;53:1188.
Ernst E. A systematic review of systematic reviews of homeopathy. Br J
Clin Pharmacol 2002;54:577–82.
Ernst E, White A. Life-threatening adverse reactions after acupuncture?
A systematic review. Pain 1997;71:123–6.
Counsell CE, Clarke MJ, Slattery J, Sandercock PA. The miracle of DICE
therapy for acute stroke: fact or fictional product of subgroup analysis?
Clarke M, Halsey J. DICE 2: a further investigation of the effects of
chance in life, death and subgroup analyses. Int J Clin Pract 2001;55:
Moore RA, Gavaghan D, Tramer MR, Collins SL, McQuay HJ. Size is
everything – large amounts of information are needed to overcome
random effects in estimating direction and magnitude of treatment
effects. Pain 1998;78:209–16.
Ioannidis J, Lau J. Evolution of treatment effects over time: empirical
insight from recursive cumulative metaanalyses. Proc Natl Acad Sci USA
13Ioannidis JP. Contradicted and initially stronger effects in highly cited
clinical research. JAMA 2005;294:218–28.
Ioannidis JP. Why most published research findings are false. PLoS Med
DerSimonian R, Laird N. Meta-analysis in clinical trials. Control Clin
Bower WF, Diao M, Tang JL, Yeung CK. Acupuncture for nocturnal
enuresis in children: a systematic review and exploration of rationale.
Neurourol Urodyn 2005;24:267–72.
Casimiro L, Brosseau L, Milne S et al. Acupuncture and electro-
acupuncture for the treatment of RA. Cochrane Database Syst Rev
Ernst E, Pittler MH. The effectiveness of acupuncture in treating acute
dental pain: a systematic review. Br Dent J 1998;184:443–7.
Ernst E, White AR. Acupuncture as a treatment for temporomandibular
joint dysfunction: a systematic review of randomized trials. Arch
Otolaryngol Head Neck Surg 1999;125:269–72.
Ernst E, White AR. Acupuncture for back pain: a meta-analysis of
randomized controlled trials. Arch Intern Med 1998;158:2235–41.
Ezzo J, Berman B, Hadhazy VA et al. Is acupuncture effective for the
treatment of chronic pain? A systematic review. Pain 2000;86:217–25.
Green S, Buchbinder R, Hetrick S. Acupuncture for shoulder pain.
Cochrane Database Syst Rev 2005(2):CD005319.
He L, Zhou D, Wu B, Li N, Zhou MK. Acupuncture for Bell’s palsy.
Cochrane Database Syst Rev 2004(1):CD002914.
Jedel E. Acupuncture in xerostomia–a systematic review. J Oral Rehabil
Jewell D, Young G. Interventions for nausea and vomiting in early
pregnancy. Cochrane Database Syst Rev 2003(4):CD000145.
Lee A, Done ML. Stimulation of the wrist acupuncture point P6 for
preventing postoperative nausea and vomiting. Cochrane Database Syst
Lee H, Ernst E. Acupuncture analgesia during surgery: a systematic
review. Pain 2005;114:511–7.
Lee H, Ernst E. Acupuncture for GI endoscopy: a systematic review.
Gastrointest Endosc 2004;60:784–9.
Lee H, Ernst E. Acupuncture for labor pain management: a systematic
review. Am J Obstet Gynecol 2004;191:1573–9.
Lee H, Schmidt K, Ernst E. Acupuncture for the relief of cancer-related
pain – a systematic review. Eur J Pain 2005;9:437–44.
Manheimer E, White A, Berman B, Forys K, Ernst E. Meta-analysis:
acupuncture for low back pain. Ann Intern Med 2005;142:651–63.
McCarney RW, Brinkhaus B, Lasserson TJ, Linde K. Acupuncture for
chronic asthma. Cochrane Database Syst Rev 2004(1):CD000008.
Melchart D, Linde K, Fischer P et al. Acupuncture for idiopathic
headache. Cochrane Database Syst Rev 2001(1):CD001218.
Mukaino Y, Park J, White A, Ernst E. The effectiveness of acupuncture
for depression – a systematic review of randomised controlled trials.
Acupunct Med 2005;23:70–6.
Pan CX, Morrison RS, Ness J, Fugh-Berman A, Leipzig RM.
Complementary and alternative medicine in the management of pain,
dyspnea, and nausea and vomiting near the end of life. A systematic
review. J Pain Symptom Manage 2000;20:374–87.
Park J, White AR, Ernst E. Efficacy of acupuncture as a treatment for
tinnitus: a systematic review. Arch Otolaryngol Head Neck Surg 2000;
Pittler MH, Ernst E. Complementary therapies for peripheral arterial
disease: systematic review. Atherosclerosis 2005;181:1–7.
Pittler MH, Ernst E. Complementary therapies for reducing body
weight: a systematic review. Int J Obes Relat Metab Disord 2005;29:
Proctor ML, Smith CA, Farquhar CM, Stones RW. Transcutaneous elec-
trical nerve stimulation and acupuncture for primary dysmenorrhoea.
Cochrane Database Syst Rev 2002(1):CD002123.
Smith CA, Crowther CA. Acupuncture for induction of labour.
Cochrane Database Syst Rev 2004(1):CD002962.
Systematic review of systematic reviews of acupuncture published 1996–2005
Clinical Medicine Vol 6 No 4 July/August 2006385