Content uploaded by Denny John
Author content
All content in this area was uploaded by Denny John on Sep 12, 2022
Content may be subject to copyright.
BEGINNER’S GUIDE FOR
SYSTEMATIC REVIEWS
A step by step guide to conduct systematic reviews
and Meta-Analysis
(An ICMR Publication)
Anju Sinha, Geetha R. Menon, Denny John
Foreword
I am happy to write this foreword for Beginner’s Guide for Systematic Reviews.
This book is mainly aimed at public health and social science researchers for
undertaking systematic reviews. The aim of this guide is to promote high
standards in commissioning, conducting, and providing practical guidance for
undertaking systematic reviews evaluating the eects of health interventions.
Over the last decade, the demand for use of the best available research
evidence to inform health care decision making and public policy has increased
considerably. Systematic reviews aim to identify, appraise, and summarize
the ndings from all relevant individual studies on a topic. Well conducted
Systematic Reviews provide the best evidence to guide clinical practice, they are
considered a cornerstone for the recommendations of Evidence-based practice
guidelines and should be an integral part of planning future research activities.
The Beginner’s Guide has been wrien for those with an understanding of health
research & seeking skills for conducting systematic reviews. The guide is also
aimed at those who commission systematic reviews, such as the government and
funding organisations. Though the book is mainly aimed at health practitioners,
some researchers working on systematic reviews on social sciences might also
nd it useful.
I appreciate the eorts of the contributors in preparing the Beginner’s Guide. As
we release this guide, I envision that it would serve a variety of audiences such
as policymakers, researchers, clinicians and other health professionals who
need to be aware of evidence-informed decision-making in health care.
(Dr. Balram Bhargava)
Acknowledgements
This book has been produced through collaborative eorts of the contributing
authors. The book draws on the collective learning from the systematic
reviewing conducted by the authors, and also from the learning’s that has
arisen from interacting with systematic review workshop participants from
India (ICMR, DHR, NICPR, NIREH, PGIMER) Nepal, Bangladesh, Ghana,
Germany, and Scotland, and researchers embarking on a systematic review
from developing countries.
We acknowledge Ms Supreet Kaur, who was a data programmer in the HIV
project at ICMR-NIMS when this book was being wrien, for her inputs in
chapter 5 and in preparing a list of textbooks as additional resources in this
book.
In addition, the authors are grateful to the reviewers Prof Meenu Singh from
PGIMER, Chandigarh and Prof. Sreekumaran Nair, JIPMER, Puducherry
who have a rich experience in teaching, training and conducting systematic
reviews. Their valuable inputs and review have helped in improving the
contents of this book.
We hope this book becomes a useful guide for beginners and early career
researchers who are planning to undertake systematic reviews.
Dr. Anju Sinha
Dr. Geetha R. Menon
Dr. Denny Hohn
Anju Sinha: Consultant Scientist Division of RBMCH,
ICMR Hqrs, New Delhi. Dr. Anju is a clinician by
training specializing in Epidemiology & Public
Health, with overall 27 years of research experience in
implementation of large community-based eld trials
funded by international agencies. At the Indian Council
of Medical Research she worked as a Program Ocer in
the area of Neonatal and Child Health, HIV prevention,
and Evidence Based Child Health. She has initiated a
funding scheme on secondary data, supporting systematic reviews and
building capacity of Indian scientists in systematic reviews. She is the
recipient of the Aubrey Sheiham award in Primary Health Care and the
Kenneth Warren award from the Cochrane. She is a Steering Commiee
member Public Health Evidence South Asia (PHESA), Scientic Advisory
Group member International Life Sciences Institute (ILSI), India, member
Technical Advisory Commiee of Health Technology Assessment India
(HTAIn), member Cochrane Neonatal and Cochrane Acute Respiratory
Infections Groups, Cochrane Child Health and Public Health Fields & Rapid
Review Methods Group and the Campbell Collaboration. She is Director
of the Cochrane Aliate Centre at ICMR Hqrs & Co Chairs the Cochrane
India Network. She is involved with Evidence to policy translation, policy
brieng, Health Technology Assessments and identifying new research
priorities.
Authors
Geetha R. Menon is a Senior Scientist trained as
a Biostatistician working for the ICMR-National
Institute of Medical Statistics. She has an experience
of more than 3 decades and has contributed to
biomedical research both as a primary researcher and
as a research manager. She has a doctorate degree in
Biostatistics and has signicant experience of analysing
data from large scale surveys and epidemiological
studies, doing statistical modelling, and undertaking
Systematic Reviews and Meta-Analysis. She has been involved in planning
and designing research studies, teaching biostatistics to researchers and
medical faculties and conducting capacity building workshops in research
methodology, systematic reviews and health economics. Geetha has co-
authored over 100 scientic publications in national and international
journals and has co-edited two books viz. Road Trac & Safety (Transportation
Issues, Policies and R&D) and Understanding and Treating Head Injuries. She is
the recipient of the ISCB Young Scientist Award 2006, Dr. Kelly P O’Keefe
Academician & Researcher of the Year Award 2013, Statistical Alliance for
Vital Events (SAVE) – Queen Elizabeth Advanced Scholars (QES) 2018, and
Indian Society of Medical Statistics BG Prasad Award 2020.
Authors
Denny John is Adjunct Professor, Faculty of Life and
Allied Health Sciences, Ramaiah University of Applied
Sciences, Bengaluru; and Adjunct Faculty, Amrita
Institute of Medical Sciences & Research Centre, Kochi.
Denny has experience of working across several review
types, such as eectiveness, cost-eectiveness, barriers/
facilitators, prevalence/incidence, risk/aetiology and
diagnostic test accuracy. Denny’s research focusses
mainly on economic evidence in systematic reviews
including equity components. With a background in epidemiology and
health economics, he has considerable methodological and statistical
expertise, including conducting cost-eectiveness alongside randomized
controlled trials and observational studies, as well systematic reviews and
meta-analyses. Denny is Chair, Cochrane & Campbell Economic Methods
Group (CCEMG), and Advisory Member, Disability Coordinating Group,
Campbell Collaboration, and Co-Chair, Early Career Network, Health
Technology Assessment International (HTAi). He was on the Editorial Board
for ‘Evidence, policy, impact: WHO guide for evidence-informed decision-
making’, published by World Health Organization in 2021.
Authors
BEGINNER’S GUIDE FOR SYSTEMATIC REVIEWS
List of boxes
Box 1: Quantitative review of domestic violence an example ......................19
Box 2: List of keywords and synonyms related to a question ......................26
Box 3: Keywords and synonyms for the systematic review example .........27
Box 4: Sources of published literature ..............................................................28
Box 5: Sources of grey literature .......................................................................30
Box 6: Good practice for data extraction ..........................................................32
Box 7: Description of critical appraisal process ..............................................52
Box 8: Plain language summary example ........................................................81
List of gures
Figure 1: Conceptual model of stakeholders, identied by the actors,
their roles and their actions .............................................................6
Figure 2: Common stages of a systematic review .......................................15
Figure 3: Structured questions for systematic reviews and relation
between question components in a comparative study ............19
Figure 4: LÁbbe Plot ........................................................................................64
Figure 5: Galbraith Plot ...................................................................................65
Figure 6: Baujat Plot .........................................................................................66
Figure 7: Funnel Plot .......................................................................................67
Figure 8: Forest plot comparing two interventions ...................................68
Figure 9: PRISMA 2020 Flow diagram .........................................................71
List of tables
Table 1 : Dierences between a narrative review and a systematic
review ..................................................................................................2
Table 2: Characteristics of narrative reviews, scoping
reviews and systematic reviews ......................................................4
Table 3: Software for various processes in the systematic
review ..................................................................................................7
Table 4: Types of review questions .............................................................16
Table 5: The PICO process ............................................................................17
Table 6: Types of reviews ..............................................................................23
Table 7: Saving a search strategy and the number of records .................31
Table 8: Critical appraisal of cohort studies example ...............................52
Table 9: GRADE certainty ratings ...............................................................55
Table 10: Summary of ndings ......................................................................56
Contents
Foreword ...............................................................................................................iii
Acknowledgements ..................................................................................................v
Authors ..............................................................................................................vii
List of boxes ..............................................................................................................x
List of gures ...........................................................................................................x
List of tables ............................................................................................................. x
1. Introduction ...................................................................................................1
1.1 What Is a Systematic Review?............................................................1
1.2 Why do we need a systematic review? ............................................. 1
1.3 What is the dierence between a narrative
review and a systematic review? .......................................................2
1.4 Where to nd systematic reviews? ....................................................4
2. Geing Started ..............................................................................................6
2.1 Writing a Systematic Review Protocol ............................................. 9
2.2 Components of a Systematic Review protocol ............................... 9
3. Steps in a systematic review .....................................................................15
3.1 Structure an answerable and focussed review question ..............16
3.2 Types of systematic reviews .............................................................20
3.3 Identify relevant studies through a comprehensive
search strategy ....................................................................................25
3.4 Undertake a comprehensive search ................................................28
3.5 Select databases ..................................................................................28
3.6 Tailor search strategy to database(s) ...............................................30
3.7 Save search and export results .........................................................31
3.8 Select studies for inclusion based on pre-dened criteria ...........31
3.9 Description of study characteristics ................................................ 32
3.10 Quality assessment using critical appraisal tools ......................... 51
3.11 Quality of evidence ............................................................................54
4. Meta-Analysis .............................................................................................59
4.1 Eect measures .................................................................................. 60
4.2 Heterogeneity ..................................................................................... 62
BEGINNER’S GUIDE FOR SYSTEMATIC REVIEWS
4.3 Graphical methods for identication of heterogeneity ................ 64
4.4 Meta-regression ..................................................................................67
4.5 Presentation ........................................................................................68
4.6 Reporting a systematic review and Meta-Analysis ...................... 70
5. Systematic review and Meta-Analysis software .................................... 72
5.1 Systematic review software .............................................................72
5.2 Meta-Analysis software .................................................................... 75
6. Disseminating Systematic Review Findings .......................................... 79
6.1 Evidence Summary ............................................................................ 79
6.2 Plain Language Summary (PLS) ......................................................80
6.3 Policy briefs ........................................................................................ 82
References ............................................................................................................84
Other Resources ..................................................................................................87
1. Introduction
1.1 What Is a Systematic Review?
A systematic review is a review of the evidence on a clearly formulated
question that uses systematic and explicit methods to identify, select and
critically appraise relevant primary research, and to extract and analyze
data from the studies that are included in the reviews1.
Diverse range of information sources and an explosion of knowledge have
made it impossible for clinical researchers to stay abreast with advances in a
given eld. Systematic reviews and meta-analyses were conducted in health
in 1970s and 1980s. Both the terms meta-analyses and systematic reviews
were used interchangeably and often created confusion among the readers.
Chalmers and Altman2 suggested that the term ‘Meta-Analysis’ may be
used only for statistical synthesis or quantitative methods of combining
the evidence from individual studies and it may or may not be part of a
systematic review.
Systematic reviews are most often used to study the eectiveness of a
particular drug treatment compared with a placebo or any other alternative
treatment. However, these cover a wide range of other issues like:
• Surgical and nursing techniques e.g. the best ways of carrying out knee
replacements or the best methods of dressing for chronic wounds
• Psychosocial interventions e.g. community-based interventions for
people with schizophrenia
• Public-health interventions e.g. impact of mobile health (mHealth)
interventions in health care delivery or lifestyle interventions in
reducing the prevalence of type II diabetes
• Adverse eects of drugs or other treatments
• Economic evaluations e.g. evaluation of implementation intervention
in public health or drug trials to identify which intervention is cost
eective
• Although Systematic Reviews are predominantly used in intervention
studies, researchers also use this technique in economic evaluation,
diagnostics tests accuracy, prevalence/incidence etc.
1.2 Why do we need a systematic review?
Systematic reviews are needed for the following reasons
a. To keep abreast of all previous and new research
2
BEGINNER’S GUIDE FOR SYSTEMATIC REVIEWS
b. To introduce a new treatment that is expected to be beer than an
existing one
c. To discontinue an old treatment which might be out dated, harmful or
less cost eective
d. To draft guidelines for health and social interventions or treatment
management
e. To arrive at a consensus where conicting evidence is reported
1.3 What is the dierence between a narrative review and a
systematic review?
In comparison with literature or traditional narrative reviews, systematic
reviews are much time-intensive and need a research team with multiple
skills and contributions. Table 1 provides a description of these dierences.
There are some cases where systematic reviews are unable to meet the
necessary objectives of the review question. In such a case, scoping reviews
(which are sometimes called scoping exercises/scoping studies) may be more
useful to consider. Table 2 provides the characteristics of narrative reviews,
scoping reviews and systematic reviews.
Narrative Review Systematic Review
Goals Provides summary or overview
of topic
Answers a focussed review
question
Question Can have a broad topic or a
specic question. Hypothesis
might not be stated.
Clearly dened review
question using PICO as a
guide. Hypothesis is stated.
Authors One or more Three or more
Protocol No protocol A peer review protocol or
plan is included
Objectives May or may not be identied Has clearly stated
objectives
Inclusion/
exclusion criteria
Criteria not usually specied Criteria stated before the
review is conducted
Search strategy No detailed search strategy,
mostly conducted using
keywords and snow-balling
Detailed and
comprehensive search
strategy
3
Introduction
Narrative Review Systematic Review
Sources of
literature
Non-exhaustive and not stated
always. Prone to publication
bias.
List of databases, grey
literature and other sources
are considered.
Selection criteria Usually subjective or no
selection criteria. Prone to
selection bias.
Selection process usually
clear and explicit
Appraisal of
study quality
Variation in evaluation of study
quality of studies
Use of standard checklists
for rigorous appraisal of
study quality
Extracting
relevant
information
Not explicit and clear Clear and specic
Synthesis
Summary based on studies
which have not been checked
for quality and can be
inuenced by the reviewers
needs and beliefs
Clear summaries of studies
based on high quality
evidence Narrative,
quantitative or qualitative
synthesis
Conclusions Sometimes evidence based but
could be prone to researcher
bias (inuence of author’s
personal belief)
Evidence-based
Timeline Weeks to months Months to years
Requirements Understanding of topic, and
searching of 2-3 databases
At least one of the authors
with good knowledge of
the topic, searches done for
all relevant databases
Value Provides summary of literature
on the topic
Conclusions may be subjective
hence minimal reproducibility
of ndings
Cannot be continuously
updated
Provides high-quality
evidence, and supports
evidence-based practice
Detailed and accurate
documentation of methods
using PRISMA means
results can be reproduced
Periodically updated to
include new evidence
4
BEGINNER’S GUIDE FOR SYSTEMATIC REVIEWS
Narrative
A priori review
protocol
No Yes (some) Yes
PROSPERO
registration of the
review protocol
No NoaYes
Explicit, transparent,
peer reviewed search
strategy
No Yes Yes
Standardized data
extraction forms No Yes Yes
Mandatory Critical
Appraisal (Risk of
Bias Assessment)
No NobYes
Synthesis of ndings
from individual
studies and the
generation of
‘summary ndings’
No No Yesc
a Current situation; this may change in time.
b Critical appraisal is not mandatory; however, reviewers may decide to assess and report
the risk of bias in scoping reviews.
c by using statistical Meta-Analysis (for quantitative eectiveness, or prevalence or
incidence, diagnostic accuracy, aetiology or risk, prognostic or psychometric data), or
meta-synthesis (experiential or expert opinion data) or both in mixed method reviews3.
1.4 Where to nd systematic reviews?
Before undertaking a systematic review, it is necessary to look for ongoing
reviews or completed reviews on the topic of interest. Below are some useful
websites to start searching for systematic reviews.
• Database of Abstracts of Reviews of Eects (DARE) hps://www.crd.york.ac.uk/
CRDWeb/HomePage.asp
• Cochrane Database of Systematic reviews
hps://www.cochranelibrary.com/cdsr/reviews
• National Institute for Health and Care Excellence UK Database of Uncertainties
about the Eects of Treatments (DUETs) ; hps://www.evidence.nhs.uk/
5
Introduction
• National Institute for Health Research – Health technology Assessment NIHR-
HTA
https://www.nihr.ac.uk/explore-nihr/funding-programmes/health-technology-
assessment.htm
• Campbell Library of Systematic Reviews
hps://campbellcollaboration.org/beer-evidence.html
• Evidence for policy and Practice Information (EPPI ) Centre - hps://eppi.ioe.
ac.uk/webdatabases4/Intro.aspx?ID=9
• Database of promoting health eectiveness reviews : hps://eppi.ioe.ac.uk/
webdatabases4/Intro.aspx?ID=9
• Agency for Healthcare Research and Quality
hps://www.ahrq.gov/research/ndings/evidence-based-reports/search.html
• Scoish Intercollegiate Guidelines Network: hps://www.sign.ac.uk/our-
guidelines.html
• BMJ Evidence based medicine:hps://ebm.bmj.com/pages/collections/ebm_
verdict/
• PROSPERO International prospective register of systematic reviews
• Turning Research into practice (TRIP): hps://www.tripdatabase.com/
• Search lters in major databases e.g. Medline, EMBASE, PSYCHLIT, CINAHL (for
search lters see ‘searching for specic study types’ below)
Summary
• A systematic review is a study that identies a specic review question,
identies relevant studies using a comprehensive search strategy,
appraises the quality of these studies and summarizes their results in a
scientic manner
• Systematic reviews can be conducted on many dierent types of
primary studies
• Systematic reviews can be used for informing policies that impact
quality, safety and values of health care
• Systematic reviews provide summary evidence from available literature
but narrative reviews do not follow scientic review methodology
2. Getting Started
Before initiating a systematic review, it is important to consider four main
aspects in managing the review:
- A systematic review team should include
experts with a range of skills including expertise in information
retrieval, epidemiologist or clinical expert, systematic review methods,
statistics, and other aspects e.g. health economist if required for cost-
eectiveness/cost-benet analysis reviews and qualitative expects for
research methods where appropriate.
– An advisory group including health
care professionals, patient representatives, service users and experts in
research methods who may be consulted at key stages may be necessary
for the funding agencies.
The timelines for completing various evidence synthesis
activities may vary. However, organizations such as Cochrane and
Campbell Collaboration suggest completing a review within a year.
- Various studies have emphasized the
importance of engaging stakeholders to ensure that systematic reviews
are shaped by members from the policy and practice community
who would be using them4,5. Figure 1 provides a conceptual model of
stakeholders and the actors.
- The selection of software will need to be considered for various
stages of the systematic review process. Table 3 provides a list of some of
the software applications found useful at various stages of the review.
Advocacy groups
Business
Citizens
Decision-enforcers
Decision-makers
Publishers
Research funders
Researchers
Editors/peer-reviewers
Endorsers
Evidence holders
Funders
Publishers
Communicators
Question askers
Reviewers
Scope inuencers
Service providers
Service users
Users of the review
Suggest sources of literature
Submit articles
Undertake the review
Endorse the review
Facilitate access to the review
Read the review
Share the review
Integrate ndings into decisions
Set the review’s methodological
standards
Provide funding and/or in-kind
contributions
Share knowledge and experience
of scope and context
7
Getting Started
management
Zotero
Mendeley
EndNote
Reference Manager
EPPI Reviewer
Free
Paid
Paid
Paid
Paid
RRCHNM
Elsevier
Clarivate Analytics
Thomson Reuters
EPPI Centre
Covidence
EPPI Reviewer
Rayyan
MS-Excels
SUMARI
DistillerSR
EPPI Reviewer
Free/Paid
Paid
Free
Free
Free
Free for
students, 4
mnths
Paid
Covidence
EPPI Centre
Qatar Foundation
Microsoft
JBI
Evidence Partners
EPPI Centre
EPPI Reviewer
MS-Excel
SUMARI
Paid
Free
Free
EPPI Centre
Microsoft
JBI
EPPI Reviewer
MS-Excel$
SRDR
SUMARI
DistillerSR
Paid
Free
Free
Free
Free for
students, 4
months
EPPI Centre
Microsoft
CEBM
JBI
Evidence Partners
EPPI Reviewer
MS-Excel
QARI
MAStARI
ACTUARI
Paid
Free
Free
Free
Free
EPPI Centre
Microsoft
JBI
JBI
JBI
Policy maker Policy focus
component of
deworming
Uses the review nding
to justify policy decisions
on use of deworming in
a country
Example
8
BEGINNER’S GUIDE FOR SYSTEMATIC REVIEWS
RevMan
STATA
SPSS
R
CMA
MedCalc
Mix 2.0
OpenMeta Analyst
MAStARI
Free
Paid
Paid
Free
Paid
Paid
Paid
Free
Free
Cochrane
STATA Corp
SPSS Inc
CRAN
CMA Corp
MedCalc Inc
Biostat XL
CEBM
JBI
Qualitative nVivo
QARI
Paid
Free
QSR International
JBI
MAXQDA Paid VERBI GmbH
$MS-Excel would need to be formaed to conduct the various components of the review
process. JBI softwares are available only for reviews with the Joanna Briggs Institute
(JBI), Australia.
The following sites have some learning resources for beginners:
• Campbell Collaboration: hps://www.campbellcollaboration.org/
• Cochrane: hps://www.cochrane.org/
• EPPI-Centre: hps://eppi.ioe.ac.uk/cms/
• Systematic Review Toolbox: hp://systematicreviewtools.com/
• International Initiative for Impact Evaluation (3ie): hps://www.3ieimpact.org/
• Centre for Evidence Based Medicine hps://www.cebm.net/
• Centre for Reviews and Dissemination Databases: hps://www.crd.york.ac.uk/
crdweb/
• Health Evidence Network (HEN), WHO: hps://www.euro.who.int/en/data-and-
evidence/evidence-informed-policy-making/health-evidence-network-hen
• Africa Evidence Network: www. hps://www.africaevidencenetwork.org/en/
• The EQUTOR Network: hps://www.equator-network.org/
• The GRADE Working Group: www. hps://gradeworkinggroup.org/
• NIHR HTA: hps://www.journalslibrary.nihr.ac.uk/#/
9
Getting Started
2.1 Writing a Systematic Review Protocol
A review protocol is a guide for a well wrien systematic review. It explains
the rationale for conducting the systematic review, states the hypothesis and
outlines the methodology to be used. It is important to note that the protocol
is a priori statement of aims and methods of the systematic review, and
referred back to whenever is needed during the systematic review process.
Research question(s), aims and methods are considered in advance to identify
the relevant literature to ensure the conduct of the review with minimal
bias, access to peer review, greater eciency in review process6. Protocol
development is often an iterative process that requires discussion within the
review team, advisory group and sometimes with the funding agency. Peer
review and publication makes the protocol publicly available.
A Cochrane review protocol is considered as an individual publication. Non-
Cochrane protocols should be registered on PROSPERO - an international
database of prospectively registered systematic reviews in health and social
care7. Key features from the review protocol are recorded and maintained
as a permanent record. Systematic reviews should be registered at inception
(i.e. at the protocol stage) to help avoid unplanned duplication and to enable
the comparison of reported review methods with what was planned in the
protocol8. This prevents duplication (research waste) and makes the process
easy when the full systematic review is sent for publication.
By writing a protocol and adhering to it during the review process, the
researcher makes it clear that the decisions taken while conducting the
review are not arbitrary, the decision to include or exclude studies in the
review are not guided by individual choices or prejudices (bias) of the
authors or prior knowledge about their results. There are several resources
available for the beginners such as the Cochrane Handbook9, PRISMA
Extension for protocols (PRISMA-P)10, Institute of Medicine-Standards for
systematic reviews11 etc.
Systematic Review should be undertaken by a team of individuals with
dierent areas of expertise. The team should consist of a person with clinical
expertise, a person with systematic review experience, a methods person
with statistical expertise, and someone with multidisciplinary experience.
Someone from the relevant location/population allows double-checking of
inclusion of studies and data collection.
2.2 Components of a Systematic Review protocol
The protocol for a systematic review is wrien using the following format.
10
BEGINNER’S GUIDE FOR SYSTEMATIC REVIEWS
This section for a Cochrane systematic review has a structured
format. Under the sub-heading the health
condition/disease under consideration is described with denitions and
epidemiological information as evidenced from latest research.
(for intervention reviews) has a description
of the intervention to be evaluated in the systematic review; recent
publications on this intervention from the literature should be aligned to
the review. The mechanism of action of the intervention is described in a
paragraph: , followed by justication
for conducting the systematic review under:
. In this section the authors are required to mention about other
published systematic reviews on the topic, if there are any, the gaps therein
that would be addressed in the present review.
The primary and secondary objectives of the review are stated
under objectives. Additionally, the pre-specied sub-group analyses within
the major comparisons need to be listed here.
This section of the protocol elaborates on the
(inclusion/exclusion criteria):
(randomized, quasi-randomized, cluster-randomized in intervention
reviews), the study population age groups, sex,
gestation etc. need to be described)(the formulation,
mode of administration, doses etc. need to be mentioned) and
under dierent comparison should
be described, The procedure to be followed for screening of titles abstracts
and full text articles requires that all the steps are completed by two persons.
In case of disagreements, a third person (usually more experienced) involved
as a team member is approached for resolution. Reviewers may choose to
perform a blinded review of the articles retrieved through search.
A comprehensive and up
to date and reproducible search is a hallmark of a systematic review. This
includes electronic Searches as well as searching other resources (hand
searching, non-indexed journals, conference proceedings and grey literature,
cross-references/citation searching/manufacturers/personal contacts). It is a
pre-requisite that at least 2-3 electronic databases are searched, in order to
qualify as a systematic review. The reviewers need to decide what databases
(MEDLINE via Pub med, EMBASE, Cochrane central register of controlled
trials (CENTRAL) Cumulative Index to Nursing and Allied Health Literature
(CINAHL) and sources will be searched, in the context of their topic. The
time period for search needs to be specied, search terms and key words
have to be wrien and a search strategy needs to be wrien down. Search is
11
Getting Started
a complex activity in the conduct of a systematic review, requiring skills as
well as access to databases, and resources. It is recommended to seek help
from an information specialist/librarian. Cochrane review groups assist the
review authors with searching through their designated search coordinators.
It is advisable to consider each of the components of the articipants,
Interventions, Comparator/comparison and Outcomes () to derive the
search terms. It is important to mention how the reviewers would search for
unpublished data (grey literature), conference abstracts. The name of the
person who will run the searches, if known, should also be mentioned.
Searching regional databases (e.g. IndMed), clinical trial registries (e.g.
CTRI), hand-searching of non-indexed journals, conference proceedings
and unpublished (grey) literature should be aempted by the reviewers and
described in the methodology.
This section comprises of the following sub-
headings and their description.
In a systematic review all steps are performed by two
reviewers independently. This is crucial for avoiding personal biases at any
step of the conduct of the review. Selection of studies is done as per the
laid down inclusion exclusion criteria. Two review authors independently
review the titles and abstracts of articles identied by searches for eligibility.
Studies are classied as included, excluded or unclear. Full articles are
retrieved after title and abstract screening to evaluate whether the study
should be included or not based on the PICO of the review. Disagreements
between the two reviewers are resolved by discussion or consultation with
the third reviewer.
Two review authors should independently
extract data from the included studies on a predesigned and pretested data
extraction sheet. Authors of the original studies may be contacted in case
of incomplete information in the published article included in the review.
The data to be extracted include general information (study ID, date of
extraction, title, authors, and source of study if not published); study
characteristics (study design, participants and inclusion/exclusion criteria
used in the study); details of interventions (Including doses, treatment
duration, comparison details, and duration of follow up).
‘Risk of bias assessment
tool and criteria are described in the Cochrane Handbook for Systematic Reviews
of Interventions and are used to assess risk of bias for included studies. Two
review authors should independently assess risk of bias in the included
studies by assessing randomisation sequence generation; allocation
12
BEGINNER’S GUIDE FOR SYSTEMATIC REVIEWS
concealment; blinding of participants, personnel, and outcome assessors;
incomplete outcome data; selective outcome reporting; and other sources
of bias.
Methods of statistical analyses are detailed
under this sub-heading. The Cochrane Handbook provides a detailed
description of the analytical methods to be used for dichotomous viz. risk
ratios (RRs) and risk dierences (RDs) with 95% condence intervals (CIs).
For continuous outcomes, measures of eect as weighted mean dierences
(MDs) with 95% CIs and standardised mean dierence (SMD) should be
reported.
Analyses should consider the level at which
randomisation was done -individual or cluster. In the event that cluster-
randomised studies are going to be included appropriate adjustment for
clustering would be required (multiply the standard error derived from the
condence interval of the eect estimate by the square root of the design
eect). The generic inverse variance method in Review Manager 5.3 software
can perform Meta-Analysis using inated variances.
Authors of original trials included in the review
should be contacted in case of incomplete/missing data
Statistical heterogeneity can be assessed via
visual inspection of forest plots of included trials, using the 2 test and the I2
statistic. Trial characteristics (participants, design, interventions, outcomes,
and risk of bias) are examined to identify the source of any observed
heterogeneity. There are cut-os recommended by Cochrane review groups
for results of the I2 test: < 25% none, 25% to 49% low, 50% to 74% moderate,
and 75%+ high heterogeneity12.
Reporting biases are assessed by trying
to identify whether the study was included in a trial registry, whether a
protocol was published, and whether the methods section provides a list
of outcomes. The reported outcomes can be compared with what has been
mentioned in the trial protocol by the trial authors versus the outcomes
reported in the published article.
: In this section the systematic reviewers describe the
statistical methods of combining data from included studies in a Meta-
Analysis if possible. However, the studies should be similar, to be combined
in the Meta-Analysis. Statistical guidelines are available in the Cochrane
Handbook for Systematic Reviews of Interventions9. The Rev-Man software13,
free to download can be used for conducting the analyses. A xed-eect
13
Getting Started
or a random-eects model may be chosen as appropriate. In case of high
heterogeneity Meta-Analysis is not recommended, only a narrative summary
of trial ndings may be provided.
Each individual study included in a systematic review
should be assessed for key sources of bias. Selection bias could occur due
to systematic dierences in baseline characteristics between the groups
compared in a study, or in randomized trials, from an inadequate generation
of a random allocation sequence or inadequate concealment of allocations
before group assignment. Other biases such as detection bias, performance
bias, arition bias and outcome reporting bias can also occur. It is important
that review authors report the methods used to assess the risk of bias in
individual studies, as well as the ndings of the assessment. The Cochrane
Collaboration’s tool for assessing the risk of bias in a systematic review,
each study is graded as low (-), high (+) or unclear (?) across dierent
types of bias using a domain-based qualitative description of critical areas
of potential bias in clinical trials. For meta-analyses, authors can conduct
sensitivity analyses that exclude trials at high risk of bias to determine the
eect on the results.
GRADE approach, as outlined in the GRADE Handbook,
is used to assess the quality of evidence for the clinically relevant outcomes14.
Two review authors independently assess the quality of evidence for each
outcome. Evidence from RCTs is initially considered high but may be
downgraded one level for serious (or two levels for very serious) limitations
on the basis of the following: design (risk of bias), consistency across studies,
directness of evidence, precision of estimates, and presence of publication
bias. A software ‘GRADE pro GDT’ is used to create a ‘Summary of ndings’
table to report the quality of the evidence. The GRADE approach yields an
assessment of the quality of a body of evidence according to one of four
grades. 1. High: We are very condent that the true eect lies close to that
of the estimate of the eect. 2. : We are moderately condent in
the eect estimate: The true eect is likely to be close to the estimate of
the eect, but there is a possibility that it is substantially dierent. 3. :
Our condence in the eect estimate is limited: The true eect may be
substantially dierent from the estimate of the eect. 4. : We have
very lile condence in the eect estimate: The true eect is likely to be
substantially dierent from the estimate of eect.
Sub-group analyses
are conducted to explore the reason for heterogeneity detected. Review
authors should a priori determine and describe the possible sub-groups in
their review under the comparisons envisaged.
14
BEGINNER’S GUIDE FOR SYSTEMATIC REVIEWS
Sensitivity analyses are conducted to assess the impact
of high risk of bias on the outcome of meta-analyses by adding studies with
high risk of bias to pooled studies with low risk of bias. Similarly, other
assumptions can also be changed to see their changing eect on the overall
estimate.
A list of references of studies included, excluded in the review
and additional references used in the background section should be
provided. The study ID usually comprises of the Surname of the author and
the year of publication.
Authors may list the names of people who have helped
them in the process of conducting the systematic review but who do not
qualify as authors.
The detailed search strategy is usually included in this
section
The contribution of each review author should be
mentioned in this section.
Any conicts of interests should be stated in this
section
If the authors have received funding to conduct
the systematic review, they should mention it here. If not, they should
acknowledge the internal support of their respective organizations.
: Sinha A, Pradhan A, Thumburu
KK, Gupta N. Probiotics for the prevention or treatment of hyperbilirubinemia
in late preterm and term neonates. Cochrane Database of Systematic Reviews
2017, Issue 8. Art. No.: CD012781. DOI: 10.1002/14651858.CD012781
3. Steps in a systematic review
A systematic review is based on a clearly formulated question, identies
relevant studies, appraises the quality and summarizes the evidence by an
explicit methodology. Figure 2 shows the common stages in a systematic
review.
"EBQUFEGSPN(PVHI0MJWFSBOE5IPNBT
3FWJFXJOJUJBUJPO
'PSNSFWJFXUFBN
&OHBHFTUBLFIPMEFST
3FWJFXRVFTUJPONFUIPEPMPHZ
'PSNVMBUFRVFTUJPODPODFQUVBMGSBNFXPSLBQQSPBDI
4FBSDI4USBUFHZ
4FBSDITDSFFOGPSJODMVTJPOVTJOHFMJHJCJMJUZDSJUFSJB
%FTDSJQUJPOPGTUVEZDIBSBDUFSJTUJDT
$PEFUPNBUDIPSCVJMEBDPODFQUVBMGSBNFXPSL
2VBMJUZBOESFMFWBODFBTTFTTNFOU
"QQMZRVBMJUZBQQSBJTBMDSJUFSJB
4ZOUIFTJT
6TFDPODFQUVBMGSBNFXPSLTUVEZDPEFTRVBMJUZKVEHNFOUT
3FQPSUJOH
*OUFSQSFUDPNNVOJDBUFmOEJOHTXJUITUBLFIPMEFST
16
BEGINNER’S GUIDE FOR SYSTEMATIC REVIEWS
3.1 Structure an answerable and focussed review question
Formulating a research question is the most critical and dicult part of
any research design. The review question underpins all the aspects of the
review methodology and every single step of the review is determined by
the focussed review question. A review question denes the nature and
scope of the review, identies the keywords, determines the search strategy,
provides guidance for selecting the primary research papers and guides the
data extraction and synthesis of results. When formulating a review question,
it is important to ensure that the question asked is an open question and not
a statement. For example, rather than saying that “antiseptic washes prevent
nosocomial infections in patients undergoing surgery “it would be beer
to ask “Are antiseptic washes more eective than non-antiseptic washes at
preventing nosocomial infections in patients undergoing surgery?”. Table
4 lists some review questions that guide in identifying primary research
papers for a systematic review.
Type
Treatment
or therapy
Which treatment is more
eective? Does it do beer
than harm?
Is hydrocolloid occlusive dressing
beer than conventional gauze
dressing in the healing of chronic
wounds?
Prevention How to reduce the risk of
disease?
Does increasing physical activity
reduce the risk of developing
diabetes?
Diagnosis How to select and interpret
diagnostic tests?
Is MRI scan more eective than X ray
in identifying hairline fractures?
Prognosis How to anticipate the likely
course of the disease?
Are babies who are bole fed more
likely to be obese in adulthood
compared to babies who are breast
fed?
Causation What are the risk factors
for developing a certain
condition?
Does exposure to smoking in mothers
who smoke during pregnancy increase
risk of foetal death?
Once a tentative review question is formed it is essential to structure it into
a well framed structured question that includes three or four elements.
A structured question includes the population, the intervention, the
comparative intervention and the outcomes that are measured. The acronym
for this is PICO which stands for Population, Intervention, Comparator, and
Steps in a systematic review
17
Outcome. Depending on the type of study design there are variants of the
acronym for eg. PEO for patient exposure and outcome, PICOC Population,
Intervention, Comparator/s, Outcomes, Context etc. Depending on the
study design, the components of the review question are accordingly
identied. Table 5 provides a list of review questions and the corresponding
components of the research question.
Question
type
Intervention
Treatment
(Therapy)
The patient’s
disease or
condition
A therapeutic
measure for
eg. surgical
intervention
or life style
change
Standard of
care, another
intervention
or placebo
Mortality
rate, work
days lost,
pain,
disability
Prevention The patients
risk factors and
general health
condition
A preventive
measure, drug
or life style
change
May not be
applicable Disease
incidence,
mortality
rate, work
days lost
Diagnosis Target disease or
condition A diagnostic
test or
procedure
Current
reference
standard or
gold standard
test for the
problem
Measure of
test utility,
sensitivity,
specicity,
odds ratio
Prognosis
(Natural
History)
The main
prognostic
factor or clinical
problem in
terms of its
severity and
duration
Time or
watchful
waiting
Usually not
applicable.
Sometimes
the standard
treatment
Survival
rates,
mortality
rates, rates
of disease
progression
Etiology
or harm
(Causation)
Risk factors,
current health
problems,
general health
condition
The
intervention or
the exposure
of interest
including
some
indication of
strength(dose)
of the risk
factor and the
duration of the
exposure
May not be
applicable Disease
incidence,
rates of
disease
progression,
mortality
rates
18
BEGINNER’S GUIDE FOR SYSTEMATIC REVIEWS
The templates below and the gure 3 shows how to build research questions
in dierent scenario and how the components are related
Y
In_______________, what is the eect of ________________on _______________
compared with _________________?
For ___________ does the use of _________________ reduce the future risk of
____________ compared with ______________?
DIAGNOSIS OR DIAGNOSTIC TEST
Are (Is) ________________ more accurate in diagnosing _______________
compared with ____________?
Does ____________ inuence ______________ in patients who have
_____________?
ETIOLOGY
Are ______________ who have _______________ at ______________
risk for/of ____________ compared with _____________ with/
without______________?
How do _______________ diagnosed with _______________ perceive
__________________?
Steps in a systematic review
19
Problem statement: Lile is known on the eectiveness of advocacy
programmes as compared to other treatments on women’s quality of life
among those who have experienced domestic violence
Review question: For women who have experienced domestic violence,
how eective are advocacy programmes compared to other treatments
on improving the quality of life
(Adapted from Khan, Kunz, Kleijnen, & Antes, 200316)
20
BEGINNER’S GUIDE FOR SYSTEMATIC REVIEWS
3.2 Types of systematic reviews
Although systematic reviews are predominantly conducted to assess the
eectiveness of health interventions, researchers and health professionals
are also concerned with other questions that need dierent approaches.
Forcibly pushing the question into the PICO format (population,
intervention, comparator and outcome), even though the question may be
on diagnostic test accuracy or prognosis can confound the remainder of the
review process. Based on the type of question addressed, about 13 types of
reviews have been identied by Zachary et al17. Table 6 provides the list of
dierent review types.
3.2.1 Eectiveness reviews
Eectiveness is the extent to which an intervention has an eect. The PICO
approach is recommended here to dene the population (e.g. demographic
and socioeconomic, factors and seing), intervention (e.g. variations
in dosage/intensity, delivery mode, and frequency/duration/ timing of
delivery), comparator (active or passive) and outcomes (primary and
secondary) including benets and harms.
3.2.2 Experiential (qualitative) reviews
Experiential reviews are qualitative in nature that focusses on the perspective
of the individuals’ experience, analysing human experiences and cultural,
social phenomena. They can be important in exploring and explaining why
Aim: The aim of this study is to evaluate the eectiveness of advocacy
programmes as compared to other treatments on improving the quality
of life of those women who have experienced domestic violence
Objectives:
* Search papers on eectiveness of advocacy programmes and other
treatments
* Collect data on the eectiveness of advocacy programmes and other
treatments
* Compare the primary studies on the dierent treatments
* Compare the ndings of this review with other reviews
* Provide guidelines for helping women who have experienced
domestic violence
Title: A systematic review on the eectiveness of advocacy programmes
as compared to other treatments on improving the quality of life of those
women who have experienced domestic violence.
Steps in a systematic review
21
interventions are or are not eective from a person-centred perspective. With
qualitative evidence, there is no outcome or comparator to be considered.
PICO is recommended for question development with P and I dened as
patient and experience eg. response to pain. Context may be geographic
location, specic racial or gender or seing such as acute care or primary
healthcare or community.
3.2.3 Costs/economic evaluation reviews
Costs/Economics reviews assess the costs of a certain intervention, process,
or procedure. Health economic evaluations are useful to inform health
policy. The PICO approach best ts here with the population, intervention
and comparator that include the nature of services/care delivered, time
period of delivery, dosage/intensity, co-interventions. Context can also be
considered in these types of questions e.g. health seing(s).
3.2.4 Prevalence and/or incidence reviews
Prevalence or incidence reviews measure disease burden. These types of
reviews inform health care planning and allocation of resources, delivery of
health services and evaluate changes and trends in diseases over time. The
CoCoPop framework can be used for such reviews. The health condition,
disease, symptom, and its measurement, diagnosis needs to be
identied.
Environmental factors dene the context or specic seing relevant to the
review question
3.2.5 Diagnostic test accuracy reviews
Systematic reviews assessing diagnostic test accuracy are important for
clinicians and other healthcare practitioners to determine the accuracy of the
diagnostic tests to identify the presence or absence of a condition for treatment
plan in a patient. For these review questions, PIRD is recommended where
the population consists of all participants who will undergo the diagnostic
test. It could be a comparison of this test with the gold standard, for the
diagnosis of the intended condition viz disease, disability or injury.
3.2.6 Aetiology and/or risk reviews
Systematic reviews of aetiology and risk are important for making health
policy decisions and prevention of adverse health outcomes. These reviews
determine whether and to what degree a relationship exists between an
exposure and a health outcome. PEO is recommended for these types of
review questions. The risk factor associated with the condition, the dose
and nature of the exposure, the duration of exposure, disease, symptom or
22
BEGINNER’S GUIDE FOR SYSTEMATIC REVIEWS
health condition, the population at risk, the context/location, are relevant in
such reviews. The outcomes of interest include the health policy issues.
3.2.7 Expert opinion/policy reviews
Expert opinion and policy analysis systematic reviews focus on the synthesis
of narrative text and/or policy to either complement empirical evidence or,
in the absence of research studies, stand alone as the best available evidence.
In the absence of research studies, the best available evidence can be drawn
from text and opinion to guide researchers and policy makers. PICo can be
used where reviewers need to describe the characteristics of the population,
such as age, gender, level of education or professional qualication,
interventions
may be areas of practice management. The use of a comparator
and outcome is not required.
3.2.8 Psychometric reviews
Psychometric systematic reviews are conducted to assess the quality in
terms of its validity, reliability, responsiveness etc of health measurement
instruments for a specic construct. The construct, name of the outcome
measurement, the target population, the type of measurement instrument of
interest (e.g. questionnaires, imaging tests) and the measurement properties
on which the review investigates, needs to be clearly specied.
3.2.9 Prognostic reviews
Prognostic research provides information on the course of a disease and
potential outcomes. These reviews should identify the relationship between
specic prognostic factors and an outcome and/or prognostic/prediction
models and prognostic tests.
3.2.10 Methodology systematic reviews
Methodology Systematic reviews are performed to examine any
methodological issues relating to the design, conduct and review of research
studies and also evidence syntheses. The types of studies (RCTs and quasi-
RCTs), the comparisons of interest and the primary and secondary outcome
measures should be identied.
3.2.11 Implementation reviews
Systematic reviews of implementation research studies are performed to
study the strategies used to integrate evidence based practices to real world
seings.
Steps in a systematic review
23
3.2.12 Predictors reviews
Reviews of studies to identify predictors of poor clinical outcomes,
improvement in health and social outcomes, quality of life, behavioural
changes etc. are conducted by synthesising evidence from observational
studies
3.2.13 Barriers and facilitators reviews
Reviews that synthesize research on barriers to and facilitators of the various
health outcomes. It provides an overview of factors related to uptake and
implementation of a health intervention and health promoting behaviours.
These reviews help the policy makers in making eorts to overcome the
barriers and promote facilitating factors.
Aim
To evaluate the
Population,
Intervention,
Comparators,
Outcomes
(PICO)
Eectiveness What is the
eectiveness
of exercise for
treating diabetes
compared to
no treatment
or comparison
treatment?
Population,
Phenomena of
Interest, Context
(PICo)
Experiential
(Qualitative)
What is
experience of
children with
epilepsy?
Population,
Intervention,
Comparator/s,
Outcomes,
Context (PICOC)
Costs/
Economic
Evaluation
What is the cost-
eectiveness of
HIV vaccines
in low- and
middle-income
countries?
Condition,
Context,
Population (Co
CoPop)
Prevalence
and/or
Incidence
What is the
prevalence
of diabetes in
India?
24
BEGINNER’S GUIDE FOR SYSTEMATIC REVIEWS
Aim
Population,
Index Test,
Reference Test,
Diagnosis of
Interest (PIRD)
Diagnostic
Test Accuracy
What is the
diagnostic test
accuracy of
nutritional tools
(such as the
Malnutrition
Screening Tool)
compared to
the Patient
Generated
Subjective
Global
Assessment
amongst
patients with
colorectal cancer
to identify
undernutrition?
Population,
Exposure,
Outcome (PEO)
Etiology and/
or Risk
Are adults
exposed to
arsenic at risk of
lung cancer?
Population,
intervention or
Phenomena of
Interest, Context
(PICO)
Expert
opinion/
policy
What are
the national
policies to
reduce maternal
mortality?
To evaluate the
Construct of
interest or the
name of the
measurement (s),
Population, Type
of measurement,
instrument,
Measurement
properties
(CoPoTIM)
Psychometric What is the
reliability,
validity,
responsiveness
and
interpretability
of methods
(manual
muscle testing,
isokinetic
dynamometry,
hand held
dynamometry)
to assess muscle
strength in
adults?
Steps in a systematic review
25
Aim
Population,
Prognostic
Factors (or
models of
interest),
Outcome (PFO)
Prognostic In adults with
low back pain,
what is the
association
between
individual
recovery
expectations and
disability
outcomes?
Types of
studies, Types
of Data, Types
of Methods,
Outcomes
(SDMO)
Methodology Organizational
ethics research:
A systematic
review of
methods and
analytical
techniques
SCOOPS (seing,
condition/
circumstance,
output, outcome,
process, strategy)
Implementa-
tion Systematic
review on
implementation
of mobile health
(mHealth)
projects in
Africa: What
works? What
doesn’t work
and why?
Condition,
Predictor,
Population
(CoPrePop)
Predictors What are the
childhood
predictors of
adult obesity?
Population,
Factors (Barriers/
Facilitators),
Intervention
(PFaI)
Barriers and
facilitators Barriers and
facilitators to
health screening
in men: A
systematic
review
3.3 Identify relevant studies through a comprehensive search
strategy
The search strategy used in the review ideally should follow established
guidelines such as MECIR (Methodological Expectations of Cochrane
Intervention Reviews)18 or MECCIR (Methodological Expectations of
26
BEGINNER’S GUIDE FOR SYSTEMATIC REVIEWS
Campbell Collaboration Reviews)19. As per the Cochrane Handbook;
‘Systematic reviews of interventions require a thorough, objective and
reproducible search of a range of sources to identify as many relevant studies
as possible. This is a major factor in distinguishing systematic reviews
from narrative reviews and helps to minimize bias and therefore assist in
achieving reliable estimates of eects.
When a systematic review is conducted, it is important to retrieve all the
relevant studies both published and unpublished that may answer the
proposed research question. A comprehensive search strategy underlies the
quality of search and the quality of ndings of the review. A search has
to be both sensitive and specic. Sensitive means the search that picks up
all the research studies that are potentially relevant and specic means, it
selects only those that are directly relevant.
Step 1 Identify the component parts in the review question ie the P, I, C, O.
The previous section discussed in detail on the structure of a review question
using these components.
Step 2 Identify any keywords and synonyms
The second step is to identify keywords and synonyms for all component
parts of the review question. Keywords describe subject areas, and having
a good set of keywords, would minimize the number of irrelevant returns.
Keywords can be identied using dictionaries, textbooks, lecture notes and
published articles. Synonyms are words which have the same or similar
meaning. Authors use dierent words and phrases to describe subject areas.
It is necessary to ensure that all synonyms and keywords are included in
the search strategy to avoid missing potentially relevant documents. Box 2
provides an example of key words and the synonyms.
Terms that have the same/close
meaning Hypertension vs High blood pressure
Terms that have dierent spellings or
acronyms Leukemia vs Leukaemia
Complex concepts described
inconsistently Long-term patient-reported satisfaction after
contralateral prophylactic mastectomy … vs
Surviving breast cancer: women’s experiences with
their changed bodies
Umbrella terms and specic names Sexually-transmitted infections
Herpes, genital warts, syphilis, gonorrhoea, chlamydia
Keywords and database-specic
“subject headings” Cancer, tumour, tumour, carcinoma
Neoplasms (MESH)
Steps in a systematic review
27
Step 3. Construct search strategy string
To retrieve the most relevant search results, a search string needs to be
constructed. A search string is a combination of keywords, truncation
symbols, and Boolean operators that is entered into the search box of a
library database or search engine.
For eg: A systematic review to analyse long-term continence disturbance
after lateral internal sphincterotomy for chronic anal ssure was carried out.
The PIO component of the research question was
The keywords and the synonyms for the components are as given in Box 3
Chronic anal ssure(P) ‘anal ssure’, ‘ssure-in-ano’, ‘chronic ssure’,
Lateral internal
sphincterotomy(I) ‘lateral internal sphincterotomy’, ‘sphincterotomy’,
‘ssurectomy’, ‘advancement ap’, ‘diltiazem’, ‘nitro-
glycerine’ and ‘botulinum toxin’
Long term continence
disturbance(O) ‘incontinence’, ‘continence disturbance’, ‘accidental bowel
motion’, ‘urge incontinence’, ‘liquid incontinence’, ‘faecal
incontinence’, ‘soilage’, ‘seepage’
&KURQLF
¿VVXUH
$QDO
¿VVXUH
$QDO
¿VVXUH
&KURQLF
¿VVXUH
$QDO
¿VVXUH
&KURQLF
¿VVXUH
For each PICO element we combine all alternative terms with an OR
For e.g. for P the combination would be: #1 anal ssure OR ssure-in-ano OR chronic
ssure.
Likewise, we combine the terms for the other two elements, #2 for I and #3 for O.
In the end we use an AND to incorporate results of all PICO elements i.e. #1 AND#2
AND #3.
The truncation wildcard * and? can be used with the terms for a wider search.
28
BEGINNER’S GUIDE FOR SYSTEMATIC REVIEWS
3.4 Undertake a comprehensive search
After completing the search strategy string, a comprehensive search should
be done using data bases and all other sources of information that are most
relevant to the review question. Sources of information could be online data
bases, specialist data bases, journal articles, grey literature, subject gateways,
conference papers and proceedings, dissertation abstracts, contacting the
experts and books.
3.5 Select databases
3.5.1 Published literature- Where to search?
The data bases for searching published literature are varied and depend on
the type of review and the context or subject of the systematic review. Box
4 provides a list of sources.
Name Note
Core
Cochrane Library
Cochrane Reviews
Cochrane Protocol
Other reviews
Trials
Intervention and diagnostic reviews
Critically appraised and re-structured abstracts
Register of clinical trials
Medline Three dierent versions: PubMed, OVID Medline
and EBSCO Medline (Books@ovid, Full text,
Medline from 1946, and Epub Ahead of print,
In-Process, In-Data review & other Non-Indexed
Citations)
Embase Pharmacological literature, conference abstracts
Medical devices(from 1947)
Web of Science Conference abstracts, citation searching, social
sciences
Education, Conference Proceedings Citation Index -
Science (CPCI-S; 1990 onwards).
Conference Proceedings Citation Index - Social
Science & Humanities (CPCI-SS&H; 1990 onwards)
Campbell Systematic
Reviews
Social sciences
Steps in a systematic review
29
Name Note
SCOPUS Conference abstracts, citation searching (from 1996),
patents, scientic webpages, Trade Publications,
Book series
Clinicaltrials.gov
Clinical Trials Registry of
India (CTRI)
South Asian Database of
Controlled Clinical Trials
(SADCCT)
Trials registered in US and global
Trials registered in India
Controlled clinical trials and their sources in India
and other South Asian countries
Indexing of Indian
Medical Journals
(IndMED)
Indexed selected peer reviewed medical journals
published from India. It supplements international
indexing services like PubMed. It covers about 100
journals indexed from 1985 onwards.
CINAHL Nursing and allied health
Psychinfo Psychology & psychiatry
ERIC Education
TOXLINE Eects of drugs and chemicals
PedRO Physiotherapy (randomized controlled trials and
systematic reviews only)
PEDE (Pediatric
Economic Database
Evaluation)
Paediatric economic evaluations
inventory of health state utility weights reported in
cost-utility analyses
CEA Registry Cost-utility analyses on a wide variety of diseases
and treatments
3.5.2 Grey literature- Where to search?
Grey literature are materials and research produced by organizations outside
of the traditional commercial or academic publishing and distribution
channels. These publication types include reports, working papers,
government documents, white papers and evaluations. Grey literature covers
published material not indexed in databases such as Medline, Embase etc,
which index principally journal literature. These include technical reports,
ocial publications, conference papers, dissertations, patents, research in
progress, usually produced by academic, government and professional
organisations. It is important to search grey literature sources in order to
30
BEGINNER’S GUIDE FOR SYSTEMATIC REVIEWS
minimise bias in your search results. Box 5 provides a list of sources of grey
literature.
Name Note
Google Scholar Initial background searches
Include along with PubMed
Open Grey (www.
opengrey.eu) System for information on grey literature in Europe
Social Science
Research Network
(hp://ssrn.com/)
Specialised research networks (economics, business &
management) in each of the social sciences. Includes
abstracts database of forthcoming papers and working
papers as well as Electronic Paper Collection of full text
documents.
ProQuest Masters, MPhil and PhD theses database of international
universities
Shodhganga Reservoir of theses from Indian universities
3.6 Tailor search strategy to database(s)
e.g. :
Search the Text Word eld for free-text terms and MeSH Terms for controlled
vocabulary terms.
e.g. sunshine[tw], sunlight[tw], Sunbathing[mh], Suntan[mh]
e.g. :
Search the Title and Abstract eld :ti, ab for free-text terms and
Emtree terms ‘term’/exp for controlled vocabulary terms.
e.g. sunshine:ti,ab, sunlight:ti,ab, ‘sun exposure’/exp, ‘sunlight’/exp
Syntax for PubMed (From Cochrane Handbook, 6.4.11.1)
#1 randomized controlled trial [pt]
#2 controlled clinical trial [pt]
#3 randomized [tiab]
#4 placebo [tiab]
#5 drug therapy [sh]
#6 randomly [tiab]
#7 trial [tiab]
#8 groups [tiab]
Steps in a systematic review
31
#9 #1 OR #2 OR #3 OR #4 OR #5 OR #6 OR #7 OR #8
#10 animals [mh] NOT humans [mh]
#11 #9 NOT #10
3.7 Save search and export results
It is vital that the search strategy is transparent and repeatable, and hence
for each source searched, a record should be made (Table 7). The full list of
results from each search and source must be recorded for transparency and
can be eciently managed using reference management software such as
Endnote, Zotero etc. Many systematic review software such as Covidence,
EPPI Reviewer etc, allow importing the search results from RIS le directly
from such reference management software.
3.8 Select studies for inclusion based on pre-dened criteria
The relevant articles are screened using the inclusion criteria at dierent
levels of reading to impose a number of lters of increasing rigor, i.e. rst
reading of article titles and abstracts to remove spurious hits; and for those
passing through this stage an assessment of the full text. In order to manage
errors during the screening process, it is a good practice that two reviewers
undertake both title/abstract and full-text screening. A kappa analysis can
be performed to check for consistency in the interpretation of the selection
criteria between the two reviewers. A kappa rating of ‘substantial’ (0.5 or
above) is recommended to pass the assessment. It is important that the
selection of each article captured during the search should be recorded as
per PRISMA owchart (see Figure 8).
20
January 1 1990 to
#1 (“calcium hydroxide” OR
(medicament OR “intracanal medicament”) OR
dressing
24834
#2 (irrigation OR irrigating OR irrigated OR irrigant
OR irrigate OR(rinse OR rinsing OR rinsed)
55869
#3 ((remov*) OR (eliminat*) 565298
#4 (endodontics OR “root canal” OR “dental pulp
activity” OR teeth OR tooth)
129733
#5 #1 AND #2 AND #3 AND #4 156
32
BEGINNER’S GUIDE FOR SYSTEMATIC REVIEWS
3.9 Description of study characteristics
Once the full text of each article is identied for inclusion in the systematic
review, the next step is to extract appropriate and relevant data using a
standardized data extraction/coding form. Coding instruments, such as
coding sheets and codebook, are designed for specic research21 synthesis
and are based on the types of interventions, outcome variables, and other
data. Brown et al22 categorizes that coded data fall into the following four
basic categories: (1) methodological and substantive features, (2) study
quality, (3) intervention descriptors, and (4) outcome measures. In order to
design an accurate and comprehensive coding scheme, there is a need for a
thorough knowledge of the included studies that are to be included in the
synthesis. The important variables identied to be coded in every research
synthesis, include: (i) Source of the study; (ii) Year of publication; (iii) Type
of research design. Once the coding sheet is completed, a codebook has to
be developed to guide the coding process, containing each variable that is
important, and should be pilot tested.
Many reviewers extract information on study characteristics, methodology,
population, interventions and outcomes, with the outcomes varying based
on the types of study designs included. For example, if RCTs are included,
the outcomes are usually expressed as risk ratios (RR), odds ratio (OR)
or dierence between means for continuous outcomes; for diagnostic
studies, the outcomes extracted are the measures of test performance (e.g.
sensitivity and specicity). Box 6 provides good practice guidelines for
data extraction from primary studies
Good practice could involve the following steps, which improve
transparency, repeatability and objectivity:
• Data extractions should always present the primary data as reported
in the primary study; if any corrections or transformations are needed
these should be presented additionally so that all data are traceable to
the primary study
• Notation of the location of data within each article and means of
extraction if data are located within gures.
• Description of any pre-analysis calculations or data transformations
(e.g. standard deviation calculation from standard error and sample
size and calculation of eect sizes.
Steps in a systematic review
33
• Details of a pre-tested data extraction form.
• Data extraction in a subset of articles by multiple reviewers
and checking, for example with a kappa test (for human error/
consistency)
• Inclusion of appendixes of extracted information
• Contact made with authors requesting data where it is missing from
relevant articles
Adapted from Collaboration for Environment Evidence, 201323
It is dicult to design a single form that meets a reviewer’s requirement.
A data extraction form should be comprehensive including all the relevant
information needed for reporting and analysing the data. Below is a data
extraction form provided by the Cochrane collaboration22. This can be used
for both RCT and non RCT reviews.
34
BEGINNER’S GUIDE FOR SYSTEMATIC REVIEWS
Data collection form
This form can be used as a guide for developing your own data extraction
form. Sections can be expanded and added, and irrelevant sections can be
removed. It is dicult to design a single form that meets the needs of all
reviews, so it is important to consider carefully the information you need
to collect, and design your form accordingly. The information included on
this form should be comprehensive, and may be used in the text of your
review, ‘Characteristics of included studies’ table, risk of bias assessment,
and statistical analysis.
Notes on using a data extraction form:
• Be consistent in the order and style you use to describe the information
•
clear that the information was not found in the study report(s), not that
• Include any instructions and decision rules on the data collection form, or
•
/ Protect document)
(surname of rst author and year rst full report of study was published e.g.
Smith 2001)
(e.g. duplicate publications, follow-up
studies)
1. Date form completed (dd/mm/yyyy)
2. Name/ID of person extracting data
3. Report title
(title of paper/ abstract/ report that data are extracted from)
Steps in a systematic review
35
4.
(if there are multiple reports of this study)
5.
6.
7.
(e.g. full report, abstract, leer)
8.
(including role of funders)
(for study authors)
9.
2. Eligibility
(Insert inclusion criteria for each
characteristic as dened in the Protocol)
in text
(pg & ¶/
g/table)
10.
Randomised trial ...
Non-randomised trial ...
Controlled before-after study
• Contemporaneous data
collection
• At least 2 intervention and 2
control clusters
...
Interrupted time series OR
Repeated measures study
• At least 3 timepoints before
and
3 after the intervention
• Clearly dened intervention
point
...
...
Other design (specify):
...
11. ...
12.
intervention
...
36
BEGINNER’S GUIDE FOR SYSTEMATIC REVIEWS
(Insert inclusion criteria for each
characteristic as dened in the Protocol)
in text
(pg & ¶/
g/table)
13.
...
14.
15.
16.
DO NOT PROCEED IF STUDY EXCLUDED FROM REVIEW
Include comparative information for each
group (i.e. intervention and controls) if
available
text
(pg & ¶/g/
table)
17.
(from which study
participants are
drawn)
18.
(including location
and social context)
19.
20.
21.
22.
Steps in a systematic review
37
text
(pg & ¶/g/
table)
23.
24.
(e.g. parallel, crossover,
non-RCT)
25.
(by individuals, cluster/
groups or body parts)
26.
27.
28.
(from recruitment to
last follow-up)
29.
See Chapter 8 of the Cochrane Handbook. Additional domains may be required for
non-randomised studies.
Domain
Low/ High/
Unclear
(pg & ¶/g/table)
30.
generation
(selection bias)
...
31.
(selection bias)
...
32.
(performance bias)
...
38
BEGINNER’S GUIDE FOR SYSTEMATIC REVIEWS
Domain
Low/ High/
Unclear
(pg & ¶/g/table)
(if required) ...
33.
(detection bias)
...
(if required) ...
34.
(arition bias)
...
35.
(reporting bias)
...
36. ...
37.
Provide overall data and, if available, comparative data for each intervention
or comparison group.
text
(pg & ¶/g/
table)
38.
(or total pop. at start of study for
NRCTs)
39.
(if applicable, no., type, no. people per
cluster)
40.
41.
(if not provided below by outcome)
42. Age
Steps in a systematic review
39
text
(pg & ¶/g/
table)
43. Sex
44.
45.
46.
47.
(additional to study intervention)
48. Other relevant
49.
50.
51.
Copy and paste table for each intervention and comparison group
7.1 Intervention Group 1
text
(pg & ¶/g/
table)
52.
53.
(specify whether no. people or clusters)
54.
(include sucient detail for replication,
e.g. content, dose, components; if it is
a natural experiment, describe the pre-
intervention)
55.
56. Timing
(e.g. frequency, duration of each
episode)
40
BEGINNER’S GUIDE FOR SYSTEMATIC REVIEWS
text
(pg & ¶/g/
table)
57. Delivery
(e.g. mechanism, medium, intensity,
delity)
58.
(e.g. no., profession, training, ethnicity
etc. if relevant)
59.
60.
(i.e. intervention cost, changes in other
costs as result of intervention)
61.
(e.g. sta numbers, cold chain,
equipment)
62.
Copy and paste table for each outcome.
Outcome 1
text
(pg & ¶/g/
table)
63.
64.
(specify whether from start or
end of intervention)
65.
66.
(with diagnostic criteria if
relevant and note whether
the outcome is desirable or
undesirable if this is not
obvious)
Steps in a systematic review
41
text
(pg & ¶/g/
table)
67.
68.
(if relevant)
69.
(indicate whether high or low
score is good)
70.
...
Yes/No/Unclear
71.
(e.g. assumptions made for
ITT analysis)
72.
(e.g. baseline or population
risk noted in Background)
73.
Copy and paste the appropriate table for each outcome, including additional
tables for each time point and subgroup as required.
For randomised or non-randomised trial - Dichotomous outcome
in text
(pg & ¶/
g/table)
74.
75.
76.
42
BEGINNER’S GUIDE FOR SYSTEMATIC REVIEWS
in text
(pg & ¶/
g/table)
77.
(specify whether
from start or end of
intervention)
78.
Note whether:
... post-intervention
OR
... change from baseline
And whether
... Adjusted OR
... Unadjusted
Intervention
No.
events No.
participants No.
events No.
participants
79. Intervention
No.
events No.
participants No.
events No.
participants
80.
81.
82. Any other
83.
(e.g. by individuals,
health professional,
practice, hospital,
community)
Steps in a systematic review
43
in text
(pg & ¶/
g/table)
84.
(e.g. adjustment for
correlation)
85.
(if yes, specify why,
e.g. correlation
adjustment)
...
Yes/No/Unclear
86.
...
Yes/No/Unclear
87.
88.
in text
(pg & ¶/
g/table)
89.
90.
91.
92.
(specify whether from start
or end of intervention)
93.
44
BEGINNER’S GUIDE FOR SYSTEMATIC REVIEWS
in text
(pg & ¶/
g/table)
94.
Note whether:
... post-
intervention OR
... change from
baseline
And whether
... Adjusted OR
... Unadjusted
Intervention
Mean SD (or
other
vari-
ance)
No.
partici-
pants
Mean SD (or
other
vari-
ance)
No.
par-
tici-
pants
95.
Intervention
Mean SD (or
anoth-
er vari-
ance)
No.
partici-
pants
Mean SD (or
other
vari-
ance)
No.
par-
tici-
pants
96.
97.
98.
99.
(e.g. by individuals, health
professional, practice,
hospital, community)
100.
(e.g. adjustment for
correlation)
101.
(if yes, specify why)
...
Yes/No/
Unclear
Steps in a systematic review
45
in text
(pg & ¶/
g/table)
102. ...
Yes/No/
Unclear
103.
104.
in text
(pg & ¶/g/
table)
105.
106.
107.
108.
(specify whether
from start or end of
intervention)
109.
110. Inter-
vention
result
SD (or
other
variance)
Control
result SD (or
other
vari-
ance)
Overall results SE (or other
variance)
111. Intervention Control
46
BEGINNER’S GUIDE FOR SYSTEMATIC REVIEWS
in text
(pg & ¶/g/
table)
112.
113.
114. Any other
115.
(e.g. by individuals,
health professional,
practice, hospital,
community)
116.
117.
(if yes, specify why)
...
118.
...
119.
(e.g. by individuals,
health professional,
practice, hospital,
community)
120.
Steps in a systematic review
47
text
(pg & ¶/g/
table)
121.
122.
123.
124.
(specify whether
from start or end of
intervention)
5.
intervention or
126. Interven-
tion re-
sult
SD (or
other
vari-
ance)
Control
result SD (or
other
vari-
ance)
Overall results SE (or other
variance)
127. No.
Intervention Control
128.
129. No.
130. Any other
48
BEGINNER’S GUIDE FOR SYSTEMATIC REVIEWS
text
(pg & ¶/g/
table)
131.
(individuals, cluster/
groups or body parts)
132.
133.
(specify)
...
Yes/No/
Unclear
134.
...
Yes/No/
Unclear
135.
136.
text
(pg & ¶/g/
table)
137.
138.
139.
140.
(e.g. days, months)
Steps in a systematic review
49
text
(pg & ¶/g/
table)
141.
142.
143.
8.
intervention
9.
intervention
144. Mean value
(with variance measure)
145.
146.
147.
(with variance measure)
148.
(individuals or cluster/
groups)
149.
150.
(specify)
...
Yes/No/
Unclear
151.
...
Yes/No/
Unclear
152.
50
BEGINNER’S GUIDE FOR SYSTEMATIC REVIEWS
text
(pg & ¶/g/
table)
153. ...
Yes/No/
Unclear
154.
Change in
level SE
Change in
SE
155.
156.
(consider disadvantaged
populations, and
possible dierences in
the intervention eect)
...
Yes/No/Unclear
157.
intervention
(e.g. lower socioeconomic
groups)
...
Yes/No/Unclear
158.
(any issues of partial or
indirect applicability)
...
Yes/No/Unclear
159.
Steps in a systematic review
51
(pg & ¶/g/table)
160.
161.
other relevant
162.
(what and from whom)
163.
(from whom, what and
when)
164.
(from whom, what and
when)
165.
3.10 Quality assessment using critical appraisal tools
The indicators for quality assessment of the included studies should be
considered and can be integrated into the study coding form. Critical
appraisal assesses quality and relevance of the research and is dened as
‘the process of carefully and systematically examining research to judge its
trustworthiness and relevance in a particular context25. Critical appraisal is
a complex process and is guided by the research question, and associated
review method. It is vital that the assessment process be standardized and
transparent as possible (see Box 7).
52
BEGINNER’S GUIDE FOR SYSTEMATIC REVIEWS
A critical appraisal was conducted for the observational studies included in the
Meta-Analysis, using the Critical Appraisal Skills Programme (UK) checklist,
assessing the validity of the results from each study on a scale of high, medium,
and satisfactory: high quality, the study was prospective and scored well on main
quality parameters such as study method, result validity, precision of outcomes,
and generalizability; medium quality, study method was sound and results were
presented with precision; satisfactory quality, the study did not score well or did
not contain any information on the main quality parameters such as study method,
result validity, precision of outcomes, or generalizability.
Adapted from Zhou, Shukla, John, & Chen (2015)26
A generic assessment looks at the quality of the execution of the study,
however may not necessarily consider whether the study is a good t for
answering the review question15. Review-specic judgements assess the
appropriateness of the study design and analysis for answering the research
question or how well matched the study is to the focus of the review in
terms of its topic. Depending on the type of review undertaken, reviews
may consider both of these assessments or only one.
The precise order in which critical appraisal and data extraction are
undertaken varies from one systematic review to another depending on the
type of systematic review. Mostly, there is an iterative relationship between
the two, and there is no set guideline as to which should come rst, however
in interventional systematic reviews, risk of bias is assessed before data
extraction for quantitative synthesis in Meta-Analysis
There are various checklists of critical appraisal tools that one can use.
However, the selection of the checklist should be explained (such as use
for RCT or non-RCT study designs) or adapt them to their own review
with the decisions stated and justied. Similar to the screening process, it
is advisable that two reviewers independently conduct the critical appraisal
of the selected studies and group the studies into high, medium and low
quality.
Quantitative Qualitative
Liu (2013)
(2001)
(2010)
Alatinga
(2011)
(2009)
Sinha
(2006)
1. Is the research
aim clearly
stated? (Yes/
No)
1 1 1 1 1 1
Steps in a systematic review
53
Quantitative Qualitative
Liu (2013)
(2001)
(2010)
Alatinga
(2011)
(2009)
Sinha
(2006)
2. Description of
the context?
(Yes/No)
1 1 1 1 1 1
3. Description of
the sampling
procedures?
(Yes/No)
1 1 1 1 1 1
4. Are sample
characteristics
suciently
reported?
(sample size,
location, and
at least one
additional
characteristic?)
(Yes/No)
1 1 1 1 1 1
5. Is it clear how
the data were
collected (e.g.
for interviews),
is there an
indication of
how interviews
were
conducted?
(Yes/No)
1 1 1 1 1 1
6. Methods of
recording of
data reported?
(Yes/No)
1 1 1 1 1 0
7. Methods
of analysis
explicitly
stated? (Yes/
No)
1 1 1 1 1 1
8. Did the study
address
a clearly
focussed issue?
(Yes/No)
0 1 1 1 1 1
9. Was the cohort
recruited in
an acceptable
way? (Yes/No)
0 1 1 1 1 1
54
BEGINNER’S GUIDE FOR SYSTEMATIC REVIEWS
Quantitative Qualitative
Liu (2013)
(2001)
(2010)
Alatinga
(2011)
(2009)
Sinha
(2006)
10. Was the
exposure
accurately
measured to
minimise bias?
(Yes/No)
1 1 1 1 1 0
11. Was the
outcome
accurately
measured to
minimise bias?
(Yes/No)
1 1 1 1 1 0
12. Have the
authors
identied all
important
confounding
factors? (Yes/
No)
1 1 1 1 1 1
13. Have they
taken account
of the
confounding
factors in the
design and/or
analysis? (Yes/
No)
1 1 1 1 1 1
14. Was the follow
up of subjects
complete
enough? (Yes/
No)
1 1 1 1 1 1
15. Was the follow
up of subjects
long enough?
(Yes/No)
1 1 1 1 1 1
(Adapted from Panda, Dror, Koehlmoos, et al., 2013)27
3.11 Quality of evidence
GRADE (Grading of Recommendations, Assessment, Development
and Evaluations) is a transparent framework for developing and
presenting summaries of evidence. It is the most widely adopted tool
for grading the quality of evidence and for making clinical/public health
recommendations.
Steps in a systematic review
55
With regards to the systematic review question, the quality of evidence for
each study outcome is rated using GRADE quality rating. GRADE has four
levels of evidence- also known as certainty in evidence or quality of evidence:
very low, low, moderate, and high (Table 9). Evidence from randomised
controlled trials starts at a high quality and evidence from observational
data starts at low quality due to the residual confounding.
The certainty in evidence is increased or decreased for reasons as described
below.
Certainty What it means
Very low
Low
Moderate
High
For each risk of bias, imprecision, inconsistency, indirectness, and publication
bias, authors have the option of decreasing their level of certainty by one
or two levels (e.g. from high to moderate), using a ‘Summary of ndings’
table.
Standard Cochrane ‘Summary of ndings’ tables includes the following
elements using one of the accepted formats. Further guidance on each of
these are available on Cochrane Training website.
1. A brief description of the comparison addressed in the ‘Summary
of ndings’ table, including both the experimental and comparison
interventions.
2. A list of the most critical and/or important health outcomes, both
desirable and undesirable, limited to seven or fewer outcomes.
3. A measure of the typical burden of each outcomes (e.g. illustrative risk,
or illustrative mean, on comparator intervention).
4. The absolute and relative magnitude of eect are measured for each (if
both are appropriate).
56
BEGINNER’S GUIDE FOR SYSTEMATIC REVIEWS
5. The numbers of participants and studies contributing to the analysis of
each outcomes.
6. A GRADE assessment of the overall certainty of the body of evidence
for each outcome (which may vary by outcome).
7. Space for comments.
8. Explanations (formerly known as footnotes).
‘Summary of ndings’ tables are supported by detailed tables, known as
‘evidence proles’, that provide greater detail than the tables of both of
the individual considerations feeding into the grading of certainty and of
the results of the studies. An example of a ‘Summary of ndings’ table is
provided in table 10.
28
Relative
CI)
Quality
(GRADE)
Risk
with
Vehicle
Risk with
Ciclopirox
8% lac-
quer
Study population
4 per 1000
41 per 1000
(8 to 219)
Study population
RR 9.29
(1.72 to
50.14)
460 (2
RCTs)
Low NNTB=3
over the
they are not
RR 1.61
(0.89 to
2.92)
460
(2
RCTs)
Low The most
commonly
reported
adverse
events were
application
site reactions
(transient
tingling,
burning, or
pain with
treatment
use), rashes
(mild
70 per 1000
112 per 1000
(62 to 204)
Steps in a systematic review
57
erythema
in the skin
surrounding
the nail), and
alterations in
nail colour
or shape.
These adverse
reactions did
not require
additional
treatment.
Study population
96 per 1000
303 per 1000
(185 to 492)
RR 3.15
(1.93 to
5.12)
460
(2
RCTs)
ModeratebNNTB = 2
Clinical
cure- not
measured
- - - - - -
*The risk in the intervention group (and its 95% CI) is based on the assumed risk in the
comparison group and the relative eect of the intervention (and its 95% CI).
CI: Condence interval; RR: Risk ratio; NNTB: Number needed to treat for an additional
benecial outcome
GRADE Working Group grades of evidence
High quality: We are very condent that the true eect lies close to that of the estimate
of the eect
Moderate quality: We are moderately condent in the eect estimate: The true eect
is likely to be close to the estimate of the eect, but there is a possibility that it is
substantiallydierent
Low quality: Our condence in the eect estimate is limited: The true eect may be
substantially dierent from the estimate of the eect
Very low quality: We have very lile condence in the eect estimate: The true eect is
likely to be substantially dierent from the estimate of eect
Relative
CI)
Quality
(GRADE)
Risk
with
Vehicle
Risk with
Ciclopirox
8% lac-
quer
58
BEGINNER’S GUIDE FOR SYSTEMATIC REVIEWS
a Downgraded by two levels to low-quality evidence: one level due to imprecision as
there are very large and variable condence intervals across both studies, with low
event rates (fewer than 100). Also downgraded by one level due to risk of bias, as most
information is from studies at unclear risk of bias.
b Downgraded by two levels to low-quality evidence: one level due to imprecision since
the 95% CI includes both a meaningful increase in risk, and no increase in risk, and one
level due to risk of bias, as most information is from studies at unclear risk of bias.
Adapted from: Topical and device-based treatments for fungal infections of
the toenails28.
4. Meta-Analysis
Meta-Analysis is a process of using quantitative methods to summarize
the results from multiple studies, obtained and critically reviewed using
a rigorous process (to minimize bias) for identifying, appraising, and
synthesizing studies to answer a specic question and draw conclusions
about the data gathered. The purpose of this process is to gain a summary
statistic (i.e., a measure of a single eect) that represents the eect of the
intervention across multiple studies. Gene V Glass coined the term “Meta-
Analysis “in 1976. Since then the popularity of Meta-Analysis has increased
signicantly. The purpose is to increase the power of the evidence generated
by combining small studies and improve the precision of the estimates by
reducing uncertainty. It is similar to a simple cross-sectional study, in which
the subjects are individual studies rather than individual persons.
A systematic review is a meta-analytic review only if it includes a
quantitative estimation of the magnitude of the eect and its uncertainty
(condence limits). Collection of studies that examine the same phenomenon
or relationships can be combined by using meta-analytical procedures
and empirical observations. Thus, Meta-Analysis can be used to combine
outcomes of treatment and comparison groups in randomized controlled
trials; prevalence rates in cross sectional studies; correlation coecients in
studies of association or pre-post /before-after event rates in before after
studies. Meta-Analysis combines eect sizes from dierent studies to
provide an overall estimate while examining and quantifying the variability
among study ndings and the eect of sampling on primary studies.
The process of conducting a Meta-Analysis is similar to any other empirical
study. It includes ve steps:
1. Formulating a research question (part of a systematic review)
Dening a research question
Dene Inclusion/Exclusion criteria
Locating and selecting studies
2. Data collection and evaluation
Data extraction
Dene eect estimates
Tabulation of relevant parameters
3. Data analysis and interpretation
Testing homogeneity of studies
Investigate sources of heterogeneity
Choose appropriate statistical techniques
60
BEGINNER’S GUIDE FOR SYSTEMATIC REVIEWS
4. Presentation
Based on the outcome of interest the data from primary studies are extracted
from the results section.
4.1 Eect measures
Continuous outcome
For each individual study, we assume two underlying populations
representing the experimental versus control groups on a continuous
outcome. Let µE and µC be the experimental and control population means,
and and be the population variances, respectively. Such a design is
applicable in studies that evaluate treatment outcome in behavioural sciences,
education, medicine, etc. Under the assumptions of normal distribution
and homoscedasticity, the usual parametric eect-size is the standardized
mean dierence (SMD), which is the dierence between the experimental
and control population means, µE and µC, divided by the pooled population
standard deviation.
There are two approaches for estimating the SMD, one is the Cohens d
method and the other is the Hedges’g method (with a small correction for
small sample bias).
1
Cohen’s d:
=
()= 1
+1
+
2(
+
2)
Hedges’ g
c(m)=1-3/(4m-1) with
m=
+
2
SE (g)= c(m)x SE(d)
d=()
)
2
Here
n1=sample size of group 1; n2 =sample size of group 2