Content uploaded by Kaveh Shojania
Author content
All content in this area was uploaded by Kaveh Shojania on Jan 09, 2016
Content may be subject to copyright.
Taking Advantage of the
Explosion of Systematic
Reviews: An Efficient
MEDLINE Search Strategy
CONTEXT. Systematic reviews of the literature are an important resource for clinicians.
Unfortunately, the few published strategies for identifying these articles involve
MEDLINE interfaces not widely available outside of academic medicine. In addi-
tion, the performance of these strategies is unknown.
OBJECTIVE. To develop and evaluate a search strategy for identifying systematic
reviews by using a publicly available MEDLINE interface (PubMed).
DESIGN. Diagnostic test assessment.
DEFINITION OF SENSITIVITY. The proportion of recognized systematic reviews (indexed
in the Cochrane Library’s Database of Abstracts of Reviews of Effectiveness [DARE]
or in ACP Journal Club) that are identified by the search strategy.
DEFINITION OF POSITIVE PREDICTIVE VALUE. The proportion of articles identified in one
of three sample searches (screening for colorectal cancer, thrombolytic therapy for
venous thromboembolism, and treatment of dementia) that meet a minimum defin-
ition of systematic review.
RESULTS. Our PubMed search strategy identified 93 of 100 DARE-indexed systemat-
ic reviews, a sensitivity of 93% (95% CI, 86% to 97%). For the sample of systematic
reviews drawn from ACP Journal Club (n = 103), the PubMed strategy achieved a
sensitivity of 97% (CI, 91% to 99%). When the three sample search strings were used,
approximately 50% of retrieved articles met our minimum definition of systematic
review. In contrast, the similar precision of a PubMed search restricted to review-
type articles (as indexed by MEDLINE) was less than 10%.
CONCLUSIONS. This search strategy identified most systematic reviews without over-
whelming users with numerous false-positive results. A “single-click” filter based on
this strategy is now available as part of the Clinical Queries feature of PubMed.
G
iven the limited time available for searching and appraising the medical liter-
ature, systematic reviews are important sources of evidence for clinicians. In
contrast to traditional narrative reviews, systematic reviews of the literature address
a well-defined clinical question, use an explicit strategy to locate relevant evidence,
evaluate the retrieved studies using prospectively defined methodologic criteria, and
formally synthesize the results.
1–7
Despite the importance of systematic reviews, MEDLINE does not index these
reviews as a specific publication type.
8, 9
“Review” represents a publication type, but
most of these articles are traditional narrative reviews.
4, 6
“Meta-analysis” also consti-
tutes a publication type; however, not all systematic reviews are meta-analyses, and
not all meta-analyses are indexed by this term.
8, 9
Thus, finding systematic reviews
continues to be a major barrier to their use by clinicians.
10
157
© 2001 American College of Physicians–American Society of Internal Medicine
•
ORIGINAL ARTICLE
KAVEH G. SHOJANIA, MD
Department of Medicine
LISA A. BERO, PhD
Department of Clinical Pharmacy
Institute for Health Policy Studies
University of California, San
Francisco
San Francisco, Calif
Eff Clin Pract. 2001;4:157-162.
Edited by Brenda E. Sirovich,
MD, MS
This paper is available at ecp.acponline.org.
•
The few published search strategies for locating
systematic reviews involve versions of MEDLINE not
widely available outside of academic medicine.
8, 9, 11
More
important, the performance of these strategies has not
been evaluated. In this article, we present a search strate-
gy for identifying systematic reviews that uses PubMed,
a MEDLINE interface freely available via the Internet
from the U.S. National Library of Medicine (NLM).
12
We assess the sensitivity of this strategy using as a “gold
standard” systematic reviews recognized by two author-
itative sources, and we evaluate the positive predictive
value (or “precision”) obtained by using three sample
clinical searches.
Methods
Overview
In this article, we use the term systematic review to refer
to any literature review, meta-analysis, guideline, or
other article that explicitly indicates the use of a strategy
for locating evidence. This purposefully broad defini-
tion is not intended to replace published recommenda-
tions for the conduct of systematic reviews.
1–7
Formal
systematic reviews include other details of the search
strategy and meet additional methodologic criteria for
evidence appraisal and synthesis.
2, 6, 11
Of note, however,
is that fewer than one third of review articles in core
medical journals meet even our broad definition.
6
MEDLINE is an on-line bibliographic database
created by the NLM that contains more than 11 million
references and abstracts published since 1965.
12
Multiple
interfaces are available for searching MEDLINE; each
differs in appearance and internal logic but seeks the
same target content. Although most of these interfaces
are proprietary, the NLM has provided free public
access to MEDLINE through PubMed since 1997.
12
The PubMed Search Strategy
As with other strategies for locating systematic reviews,
8, 11
the search presented in Table 1 combines relevant publi-
cation types (“meta-analysis,” “review,” “guideline”)
with title and text words typically found in systematic
reviews. The “BUTNOT” phrase in the final search
string eliminates extraneous publication types, such as
letters, editorials, and case reports. The latter common-
ly includes a “review of the literature,” but the thor-
oughness of such literature reviews varies widely.
Moreover, case reports are generally not relevant to
answering questions relating to clinical practice, and
their sheer number can overwhelm clinicians searching
for such answers.
Sensitivity
We defined the sensitivity of the search strategy as the
proportion of systematic reviews indexed in two reposi-
tories—the Cochrane Library’s Database of Abstracts of
Reviews of Effectiveness (DARE) or by the American
•
Effective Clinical Practice ■ July/August 2001 Volume 4 Number 4
158
¶
A single-line version of the string can be found in the Appendix.
†
Clicking on “History” on the main search page for the National
Library of Medicine’s PubMed (Figure 1) reveals the number for
each search.
‡
These strings should be typed verbatim in the search window
(Figure 1) or cut and pasted from a text file in which the search
has been stored. Terms in square brackets represent accepted
abbreviations for the search fields targeted by the preceding terms.
“ti” for “title word,” “pt” for “publication type,” and “tw” for
“text word.” One can obtain these abbreviations by displaying any
record in MEDLINE format (one of the options on the Display
pull-down menu in Figure 1) or by clicking on Help/FAQ in the
left column of the main search page (Figure 1). Boolean operators,
such as AND, OR, and BUTNOT, must be written in capital
letters. The asterisk used in search numbers 3, 4, and 6 truncates
the search term so that, for instance, both “guide line” and “guide-
lines” are retrieved.
§
Searches were performed on May 2, 2001. At that time, limiting
this search string to English-language publications yielded 23,562
records.
TABLE 1
PubMed Search for Identifying Systematic
Reviews
¶
SEARCH
NUMBER
†
1
2
3
4
5
6
NUMBER OF
MEDLINE
RECORDS
§
9911
10,240
11,990
6784
38,178
30,442
§
SEARCH STRING
‡
meta-analysis [pt] OR meta-analysis
[tw] OR metanalysis [tw]
Cochrane [tw] OR Medline [tw] OR
CINAHL [tw] OR (National [tw] AND
Library [tw])
(handsearch* [tw] OR search* [tw]
OR searching [tw]) AND (hand [tw]
OR manual [tw] OR electronic [tw]
OR bibliographi* [tw] OR database*
OR #2)
(review [pt] OR guideline [pt] OR
consensus [ti] OR guideline* [ti]
OR literature [ti] OR overview [ti]
OR review [ti]) AND (#2 OR #3)
(synthesis [ti] OR overview [ti] OR
review [ti] OR survey [ti]) AND
(systematic [ti] OR critical [ti] OR
methodologic [ti] OR quantitative
[ti] OR qualitative [ti] OR literature
[ti] OR evidence [ti] OR evidence-
based [ti])
(#1 OR #4 OR #5) BUTNOT (case* [ti]
OR report [ti] OR editorial [pt] OR
comment [pt] OR letter [pt])
College of Physicians–American Society of Internal
Medicine’s ACP Journal Club—that were successfully
retrieved by our search strategy. The DARE contains
quality-assessed systematic reviews identified by regular
searches of important bibliographic databases (such as
MEDLINE) and hand-searching of major journals.
13
Included reviews must meet at least four methodologic
criteria, and all included articles must display “evidence
of a substantial effort to search for all relevant
research.”
13
The Cochrane Library contains DARE and
lists records (1899 at the time of this evaluation) in an
apparently arbitrary order. We chose as a target sample
the first 100 records that corresponded to articles
indexed in MEDLINE. The total sample required to do
this was 112 (i.e., 12 articles are not in MEDLINE).
Review articles included in ACP Journal Club must
contain “an identifiable description of the methods indi-
cating the sources and methods for searching for articles
and state the clinical topic and the inclusion and exclu-
sion criteria for selecting articles for detailed review.”
14
In addition, at least one study reviewed in the article
must meet the criteria required of original articles
addressing treatment, diagnosis, prognosis, causation,
quality improvement, or economics of health care pro-
grams.
14
These articles thus represent systematic reviews
that clinicians would wish to retrieve with high sensitiv-
ity. We hand-searched the number of consecutive issues
that was required to identify 100 “reviews” commented
on in ACP Journal Club. Achieving this number
required scanning from the most recent issue at the time
of this search (September/October 2000) back to the first
issue of 1999. This process identified 104 reviews, one of
which was excluded because it represented a Canadian
government report not indexed in MEDLINE.
Positive Predictive Value
We evaluated the positive predictive value of our search
strategy by applying the strategy to each of three clinical
topics: screening for colorectal cancer, thrombolytic thera-
py for venous thromboembolism, and treatment of demen-
tia. The search strings for each clinical topic were created
by using the Medical Subject Headings (MeSH) browser
function, located in the left column of the main search
screen in Figure 1. We then reviewed the titles and
abstracts of each retrieved article to determine whether it
met our criteria for a systematic review.
•
Effective Clinical Practice ■ July/August 2001 Volume 4 Number 4
159
FIGURE 1. PubMed home page (www.ncbi.nlm.nih.gov/entrez/query.fcgi). The National Library of Medicine allows unrestricted online
access to MEDLINE through PubMed. PubMed has several useful features. These include the “MeSH Browser,” which allows users to iden-
tify the optimal Medical Subject Headings relevant to a given clinical topic; “Clinical Queries,” where a single-click filter based on this strat-
egy is now available; and an option for storing search strings (the “Cubby” feature).
In the parlance of library science, we measured the
“precision” of the search, which, continuing with the
more familiar terminology of diagnostic tests, corre-
sponds to the positive predictive value—the proportion
of retrieved articles that represent “true positives.” We
regarded an article as a true positive only if the title or
abstract explicitly identified the article as a systematic
review or meta-analysis or if the article abstract indicat-
ed a strategy for locating the literature reviewed. Thus,
an article that contained the phrase “literature review”
in the title but merely stated that “relevant literature was
reviewed” in the abstract would not count as a true pos-
itive. MEDLINE records without an abstract could be
counted as true positives only if the title contained the
words “meta-analysis,” “metaanalysis,” or “systematic
review.”
Results
Sensitivity
As shown in Table 2, our electronic search strategy
retrieved 93 of the 100 target articles in the DARE (sensi-
tivity, 93% [95% CI, 86% to 97%]). This sensitivity reflects
our exclusion of 12 records not indexed in MEDLINE.
Including such reviews in the target population reduces
the sensitivity to 83% (95% CI, 74% to 89%). Four
15–18
of
the seven DARE articles missed by our electronic search
strategy involve aspects of behavioral medicine likely to
fall outside the interests of most clinicians, reflecting the
broad range of health care fields targeted by DARE.
The reviews drawn from ACP Journal Club, on the
other hand, are explicitly chosen for their relevance to
clinical practice. Among this sample, the electronic search
retrieved 100 of 103 references (Table 2), yielding a sensi-
tivity of 97% (CI, 91% to 99%). Careful review of the
abstracts of the three missed references indicated that fail-
ure to capture one of them
19
reflected the imperfect sensi-
tivity of the MEDLINE search engine for title words (in
this case, “review”),
21, 20
not the logic of the search strategy.
Positive Predictive Value
As shown in Table 3, approximately 50% of the retrieved
articles met the criteria for true positive in each of the
three sample searches. In other words, half of the articles
retrieved by combining a clinical search string with the
“universe of systematic reviews” created by the general
search (search 6 in Table 1) met our broad definition of
systematic review. For comparison, we measured the
positive predictive value of a search combining one of our
three sample search strings (therapy for dementia) with
the universe of articles referenced by MEDLINE as pub-
lication type “review” in 1999 (not shown in Table 1).
Among the 170 articles retrieved, only 13 met the above
definition for true positive, corresponding to a positive
predictive value of only 8% (CI, 4% to 13%).
The modest positive predictive value for our
search strategy, approximately 50%, is probably an
underestimate, since we required true-positive results to
be identifiable as systematic reviews solely on the basis of
the MEDLINE record. On retrieval of the full-text arti-
cles, some of the retrieved citations not meeting this
requirement (i.e., false positives) proved to be systemat-
ic reviews. For example, while none of the false-positive
articles on colorectal cancer screening met our definition
of a systematic review, three of the five false-positive ref-
erences for thrombolytic therapy for venous thrombo-
embolism
22–26
explicitly stated that the MEDLINE data-
base was searched,
22, 23, 26
and one of the articles even
provided the search terms used.
22
Discussion
We have presented a strategy that allows users to identi-
fy systematic reviews with a sensitivity of more than
90% for articles indexed in MEDLINE. For articles
likely to have the greatest relevance to clinicians (those
indexed in ACP Journal Club), the sensitivity exceeded
95%. This high sensitivity was achieved without many
false-positive results. Approximately 50% of articles
retrieved by sample searches met our broad definition of
systematic review, compared with fewer than 10%
obtained by simply restricting one of the sample search-
es to the publication type “review.”
Some readers might reasonably be concerned
about the broad definition used to evaluate performance
of the search strategy. We chose this approach to impose
a reasonable and easily applicable minimum require-
•
Effective Clinical Practice ■ July/August 2001 Volume 4 Number 4
160
*DARE = Cochrane Library’s Database of Abstracts of Reviews
of Effectiveness.
TABLE 2
Sensitivity of the Search Strategy*
MEASURES
Number of target
articles
Number of target
articles retrieved
by electronic
search
Sensitivity (95% CI)
“REVIEW” ARTICLES
COMMENTED ON IN
ACP JOURNAL CLUB
103
100
97%
(91%–99%)
SYSTEMATIC REVIEWS
INDEXED IN DARE
100
93
93%
(86%–97%)
ment for a systematic review, rather than formally
defining attributes required of a high-quality systematic
review. Published recommendations for the method-
ologic features of systematic reviews
1, 2, 6
do not stipulate
the minimum criteria that a review article must fulfill to
count as a systematic review. Above all, we sought to dis-
tinguish between systematic reviews, which tend to
share certain distinctive analytic features, and narrative
reviews, an entirely distinct category. Of the various
methodologic criteria typically applied to systematic
reviews, the explicit mention of a strategy for locating
evidence seemed the most basic and least controversial,
hence our focus on this feature in constructing and eval-
uating our electronic search strategy.
The usefulness of previously published electronic
search strategies for identifying systematic reviews
8, 9, 11
has been limited, since all have used MEDLINE inter-
faces that do not permit free public access (e.g., Ovid).
Moreover, few studies have evaluated the performance of
these strategies. One exception is a study describing a for-
mal search strategy (using the Ovid interface) based on
frequency analysis of terms appearing in a “quasi–gold
standard” sample obtained by hand-searching high-
impact journals.
27
The proposed strategy achieved 98%
sensitivity while retaining a precision of 20%, but this per-
formance was evaluated against the same gold standard
sample from which the search terms were identified.
We have evaluated an electronic search strategy
that identifies systematic reviews with high sensitivity
without overwhelming users with numerous false-posi-
tive results. Readers are invited to store this search in a
text file for use with subsequent searches (see the
Appendix). While most systematic reviews have
addressed aspects of therapeutics, articles reviewing
diagnosis, prognosis, cause, and even outcomes and
effectiveness research now appear with increasing fre-
quency.
7
Thus, clinicians have at their fingertips a
resource for identifying articles likely to contain evi-
dence-based answers to important clinical questions.
•
Effective Clinical Practice ■ July/August 2001 Volume 4 Number 4
161
*To generate the above searches, one would first run the search strategy shown in Table 1 and then type, for instance, “#6 AND
dementia/therapy [mh].” “mh” stands for MESH (Medical Subject Heading) term. The exact results shown above were obtained on
September 20, 2000, with “English language” and “1999/01/01-1999/12/31” used as limits. The 135 specific citations are available
from the authors on request.
TABLE 3
Positive Predictive Value of the Search Strategy
CLINICAL TOPIC
Screening for colorectal cancer
Thrombolytic therapy for venous
thromboembolism
Treatment of dementia
SEARCH TERMS*
colorectal neoplasms [mh] AND
(occult blood [mh] OR mass
screening [mh])
(plasminogen activators [mh]
OR thrombolytic therapy [mh])
AND (pulmonary embolism
[mh] OR thromboembolism
[mh] OR venous thrombosis
[mh])
dementia/therapy [mh]
NUMBER OF
ARTICLES
RETRIEVED
19
11
105
TRUE-POSITIVE
RESULTS
10
6
53
IMPLIED POSITIVE
PREDICTIVE
VALUE
(95% CI)
53%
(30%–75%)
54%
(24%–82%)
50%
(41%–60%)
• Systematic reviews are valuable to clinicians as
unbiased summaries of the state of knowledge of a
specific clinical topic. Unfortunately, they can be
difficult to find.
• We developed and tested a search strategy to identify
systematic reviews using PubMed, a freely available
MEDLINE interface.
• The strategy retrieved more than 90% of systematic
reviews listed in the Cochrane Library and ACP
Journal Club.
• When the articles identified by the strategy were
applied to three selected clinical topics, roughly half
met our criteria for a systematic review.
• Clinicians who use this strategy can be confident that
it will provide them with the majority of important
systematic reviews, with a minimum of clutter.
Take-Home Points
References
1. Oxman AD, Cook DJ, Guyatt GH. Users’ guides to the med-
ical literature. VI. How to use an overview. Evidence-Based
Medicine Working Group. JAMA. 1994;272:1367-71.
2. Oxman AD, Guyatt GH. Guidelines for reading literature
reviews. CMAJ. 1988;138:697-703.
3. Oxman AD. Checklists for review articles. BMJ. 1994;309:648-
51.
4. Cook DJ, Mulrow CD, Haynes RB. Systematic reviews: syn-
thesis of best evidence for clinical decisions. Ann Intern Med.
1997;126:376-80.
5. Bero LA, Grilli R, Grimshaw JM, Harvey E, Oxman AD,
Thomson MA. Closing the gap between research and practice:
an overview of systematic reviews of interventions to promote
the implementation of research findings. The Cochrane
Effective Practice and Organization of Care Review Group.
BMJ. 1998;317:465-8.
6. McAlister FA, Clark HD, van Walraven C, et al. The medical
review article revisited: has the science improved? Ann Intern
Med. 1999;131:947-51.
7. Petticrew M. Systematic reviews from astronomy to zoology:
myths and misconceptions. BMJ. 2001;322:98-101.
8. Hunt DL, McKibbon KA. Locating and appraising systemat-
ic reviews. Ann Intern Med. 1997;126:532-8.
9. Glanville J, Lefebvre C. Identifying systematic reviews: key
resources [Editorial]. ACP J Club. 2000;132:A11-2.
10. Bero LA, Jadad AR. How consumers and policymakers can
use systematic reviews for decision making. Ann Intern Med.
1997;127:37-42.
11. Jadad AR, Cook DJ, Jones A, et al. Methodology and reports
of systematic reviews and meta-analyses: a comparison of
Cochrane reviews with articles published in paper-based jour-
nals. JAMA. 1998;280:278-80.
12. Lindberg DA. Internet access to the National Library of
Medicine. Eff Clin Pract. 2000;3:256-60.
13. What are the criteria for the inclusion of reviews in DARE?
NHS Centre for Reviews and Dissemination. Available at
http://agatha.york.ac.uk/faq4.htm. Accessed 2 May 2001.
14. Purpose and Procedure. ACP Journal Club. Available at
www.acponline.org/journals/acpjc/purproc.htm. Accessed 20
June 2001.
15. Bauman LJ, Drotar D, Leventhal JM, Perrin EC, Pless IB. A
review of psychosocial interventions for children with chronic
health conditions. Pediatrics. 1997;100:244-51.
16. Katon W, Gonzales J. A review of randomized trials of psy-
chiatric consultation-liaison studies in primary care.
Psychosomatics. 1994;35:268-78.
17. Moner SE. Acupuncture and addiction treatment. J Addict
Dis. 1996;15:79-100.
18. Richmond RL. A comparison of measures used to assess effec-
tiveness of the transdermal nicotine patch at 1 year. Addict
Behav. 1997;22:753-7.
19. Magee LA, Ornstein MP, von Dadelszen P. Fortnightly
review: management of hypertension in pregnancy. BMJ.
1999;318:1332-6.
20. Wilczynski NL, Walker CJ, McKibbon KA, Haynes RB.
Reasons for the loss of sensitivity and specificity of method-
ologic MeSH terms and textwords in MEDLINE. Proc Annu
Symp Comput Appl Med Care. 1995:436-40.
21. Allison JJ, Kiefe CI, Weissman NW, Carter J, Centor RM.
The art and science of searching MEDLINE to answer clini-
cal questions. Finding the right number of articles. Int J
Technol Assess Health Care. 1999;15:281-96.
22. Arcasoy SM, Kreit JW. Thrombolytic therapy of pulmonary
embolism: a comprehensive review of current evidence. Chest.
1999;115:1695-707.
23. O’Meara JJ 3rd, McNutt RA, Evans AT, Moore SW, Downs
SM. A decision analysis of streptokinase plus heparin as com-
pared with heparin alone for deep-vein thrombosis. N Engl J
Med. 1994;330:1864-9.
24. Jones DR. Audit of attitudes to and use of postoperative
thromboembolic prophylaxis in a regional health authority.
Ann R Coll Surg Engl. 1991;73:219-21; discussion 222.
25. Gallus AS. Anticoagulants in the prevention of venous throm-
boembolism. Baillieres Clin Haematol. 1990;3:651-84.
26. Rogers LQ, Lutcher CL. Streptokinase therapy for deep vein
thrombosis: a comprehensive review of the English literature.
Am J Med. 1990;88:389-95.
27. Boynton J, Glanville J, McDaid D, Lefebvre C. Identifying
systematic reviews in MEDLINE: developing an objective
approach to search strategy design. Journal of Information
Science. 1998;24:137-54.
Acknowledgment
The authors thank Dr. Brian Haynes for his comments and sug-
gestions on an earlier draft of this manuscript.
Correspondence
Kaveh G. Shojania, MD, Department of Medicine, Box 0120
UCSF, San Francisco, CA 94143-0120; telephone: 415-502-1323;
fax: 415-514-2094; e-mail: Shojania@medicine.ucsf.edu.
•
Effective Clinical Practice ■ July/August 2001 Volume 4 Number 4
162
The search for systematic reviews shown in Table 1 is presented
as a six-line search so that the logic of the search would be
apparent. For everyday use, however, readers are invited to use a
single-line version of this search, which can be copied into a text
file and pasted into PubMed for subsequent searches:
((meta-analysis [pt] OR meta-analysis [tw] OR metanalysis
[tw]) OR ((review [pt] OR guideline [pt] OR consensus [ti] OR
guideline* [ti] OR literature [ti] OR overview [ti] OR review [ti])
AND ((Cochrane [tw] OR Medline [tw] OR CINAHL [tw] OR
(National [tw] AND Library [tw])) OR (handsearch* [tw] OR search*
[tw] OR searching [tw]) AND (hand [tw] OR manual [tw] OR elec-
tronic [tw] OR bibliographi* [tw] OR database* OR (Cochrane [tw]
OR Medline [tw] OR CINAHL [tw] OR (National [tw] AND Library
[tw]))))) OR ((synthesis [ti] OR overview [ti] OR review [ti] OR sur-
vey [ti]) AND (systematic [ti] OR critical [ti] OR methodologic [ti]
OR quantitative [ti] OR qualitative [ti] OR literature [ti] OR evi-
dence [ti] OR evidence-based [ti]))) BUTNOT (case* [ti] OR report
[ti] OR editorial [pt] OR comment [pt] OR letter [pt])
To avoid the inconvenience of transcribing this tortuous
string into a text file, readers can go to the
ecp Web site and
copy the above search string from the full-text version of this arti-
cle (available at ecp.acponline.org). Please note: Users of older
version of Netscape should be aware that the search string may
be truncated.
Appendix
