Introduction to Systematic Review

Abstract

In the era of evidence-basedEvidence-based decision-making systematic reviews and meta-analyses are being widely used in many medical practices, public health departments, government programs, business offices, and academic disciplines including education and psychology.

Full text

Chapter 1
Introduction to Systematic Review
In the era of evidence-based decision-making systematic reviews and meta-analyses
are being widely used in many medical practices, public health departments, govern-
ment programs, business offices, and academic disciplines including education
and psychology. Obviously, not everyone involved in the evidence-based decision-
making is evidence-informed, and aware of the various levels and/or quality of
the evidence as well as the issues that directly impact on the validity and trust-
worthiness of the results. Research synthesis is an essential part of evidence-based
decision-making.
1.1 Introduction to Evidence-Based Decision-Making
The world is increasingly moving towards evidence-based decision-making due to its
proven ability to guide practitioners and policy makers to find out which interventions
or methods or programs work effectively and which don’t. It enables decision-making
on programs, methods, interventions, treatments, practices, or policies based on the
best available evidence in the form of historical record, experiential outcome, relevant
data, contextual understanding and discipline knowledge. The main emphasis is to
find out the method or procedure that has produced expected and reliable results and
delivered the targeted outcome. The decision-makers use evidence to decide which
method/intervention works the best to achieve the objectives, and avoid the ones
that are ineffective. Thus the effectiveness of the evidence-based decisions directly
depend on the quality of the evidence and it is crucial to be evidence-informed, that
is, the decision-makers must be aware of the factors that impact on the quality of
evidence along with any shortcomings in the process of gathering, processing and
presenting evidence.
© Springer Nature Singapore Pte Ltd. 2020
S. Khan, Meta-Analysis, Statistics for Biology and Health,
https://doi.org/10.1007/978-981- 15-5032- 4_1
3

4 1 Introduction to Systematic Review
Although evidence-based decision-making originated and has been frequently
used in medical procedures and health sciences, it is also used in a wide range of
areas including agriculture, education, management, business, psychology, and social
sciences. Obviously the validity of overall evidence, and hence the decisions based
on them, are directly dependent on the quality of the underlying sources of evidence
and the methodology employed to extract and process them. Hence the quality of
evidence is crucial to any good evidence-based decision-making.
In an article (Breckon 2016) states that the policy-makers often pay lip-service
to the idea of basing decisions on available evidence, but anyone who has ever tried
to get evidence listened to and acted upon knows how hard it can be to achieve cut-
through. What, then, are the best ways for getting research used by public decision-
makers? It is hard to know when there are so many approaches to choose from.
A recent review, conducted by the Evidence for Policy and Practice Information
and Co-ordinating Centre (EPPI-Centre) at University College London, counted 150
different techniques in all. This article cited the recent work of (Langer et al. 2016)
to summarise the grouping of the above techniques used to form evidences into
six categories: Awareness, Agree, Access and communication, Interact, Skills, and
Structure and process.
1.2 Gathering Research Data
Gathering of research data is a crucial first part of any systematic review, and
research synthesis is the foundation of evidence-based decision-making. Evidence
from different sources on a specific research question or topic of interest is gath-
ered and analysed using systematic review methodologies. This is followed by the
combination of quantitative study outcomes from independent studies to find an esti-
mate of the synthesized outcome representing the unknown population parameter of
interest—a process known as meta-analysis. The strict implementation, assessment
and monitoring of the underlying selection and exclusion criteria, study protocol and
quality assessment ensure that the results of the synthesis are reproducible.
Gathering and reviewing data systematically is called the systematic review whose
first step is to identify studies that satisfy the predetermined inclusion criteria, and
extract the relevant summary statistics from the selected studies adhering to agreed
review procedures and protocols. Meta-analysis, on the other hand, is the quantitative
part of the synthesis and enables us to arrive at a numerical summary of results
(discussed in detail in other chapters).
The quality of the results produced through research synthesis depends on the
quality of the studies and their design. If the selected studies are of high quality,
then the results constitute the highest level of evidence. However, there are genuine
issues related to such syntheses that directly impact on the quality of the final result.
It is absolutely essential that policy-makers, and the producers and end users of such
syntheses are aware of the weaknesses and strengths of the underlying processes and

1.2 Gathering Research Data 5
techniques so that they could assess the robustness of their results. This chapter intro-
duces some basic concepts and methods in research synthesis and critically examines
the strengths and weaknesses of the technique to provide insightful guidance to help
professionals who are engaged in evidence-based decision-making.
Introduction of research synthesis in the process of the evidence-based decision-
making is a milestone to avoid selection bias, achieve consistency and maintain
high quality in assessing the studies with uniform standard. A number of rigorous
systems with specific selection criteria have been introduced to improve the system-
atic reviews to achieve repeatability or reproducibility of studies. The statistical
meta-analysis is the key to synthesise quantitative summary data from independent
studies to estimate the common effect size (Khan et al. 2016).
1.3 Essentials of the Systematic Review
In a general term, the synthesis starts off with a systematic review which is a process
of searching, gathering and investigating the literature on a specific topic to identify,
select and analyse any evidence of interest. It is rigorous and comprehensive to make
it transparent, minimise bias and enable future replicability. In the past, narrative
reviews were used to assess evidence and was conducted by key leaders in the field on
broad topics in some informal, unsystematic and subjective ways. Since the narrative
reviews are conducted by individuals there is a very high chance of personal bias
in the conclusion or evidence even if the authors are expert in the area. Narrative
reviews are normally not objective in assessing the literature and evidence, and hence
not replicable. The systematic review, on the other hand, is an attempt to objectively
identify all the relevant literature with a view to selecting studies based on specific
criteria, collect the documents, review their contents and critically analyse them to
assess the underlying evidence. This kind of reviewis objective and free from personal
bias or preferences. Khan et al. (2003) describes five different steps in performing a
systematic review to ensure its objectivity.
1.3.1 Steps in Systematic Reviews
Systematic reviews must be comprehensive, exhaustive and meet the expectation
of reproducibility. To ensure these key characteristics, the following five steps have
been suggested: (1) Framing the research or study questions for the intended review,
(2) identifying all relevant work in the published and unpublished literature, (3)
assessing the quality of studies, (4) extracting and summarizing the evidence, and
(5) interpreting the findings.

6 1 Introduction to Systematic Review
There are studies, including (Yannascoli et al. 2013), that provide comprehensive
summary of the steps in conducting good quality systematic reviews. In spite of some
differences in the details, the key steps in any systematic review literature are about
the same. Readers may find a recent study by Memon et al. (2020)useful.
Any systematic review should start with good planning and agreed strategies to
implement the plan focused to addressing the research question. The research team
requires agreement on the list of tasks and the stratergies to handle any foreseeable
problems to be addressed. The team should decide on an agreed list of what to be
done, who will do what, how the tasks will be distributed/managed, and under what
timeframe they will work. The distribution of the tasks to the members of the team
and any back-up arrangements should also be a part of the planning. The successful
implementation of the planning would require regular monitoring, assessment and
review of the progress and adjustment in any areas of the review.
To implement any systematic review, formation of a research team comprising of
experts from all relevant areas covering the study topic is the first step.
(a) Reasons for the study
The research team must clearly specify and agree on the reasons for the study as
these are the driving force for any systematic reviews. The team must be fully aware
and convinced of the reasons behind the proposed review and why the study is
important. If the reasons behind any systematic review are not convincing and strong
and the potential outcomes are not important it may not be worth spending time and
resources. The stronger the reasons for the review, the firmer will be the commitment
of the members of the research team and better will be the quality of the outcome.
It is essential to keep in mind that the expected contribution of the review to the
existing literature and how to minimize, if not eliminate any review bias to make it
reproducible.
(b) Research question
One of the very first and key issues the research team must address is the formula-
tion of the research question. This requires initial literature review to check if the
research question to be investigated has already been addressed by others, and if there
are enough accessible materials to answer the research question. Once determined,
research question is the key driver of the review. All the planning considerations
and activities will be centred around the research question. The review team must
critically discuss the appropriateness of the research question, its importance and
validity, and how to address it with the available literature relevant to the problem.
(c) Inclusion and exclusion criteria
Strict inclusion and exclusion criteria to be laid down at the outset of gathering
research data to determine which identified studies to be included in the review
are essential to avoid personal or selection bias in selecting studies identified by
literature search. The specific conditions and protocols to select studies or articles in

1.3 Essentials of the Systematic Review 7
the proposed review should be explicitly stated before searching databases. There are
many considerations that could potentially impact on the inclusion/exclusion criteria
but the most relevant ones (e.g., study period, study type/design, RCTs, language,
outcome measures) must be clearly stated and implemented throughout the searching
process of the review.
1.4 Literature Search—Strategies, Terms, and Databases
Extensive and comprehensive search of all literature relevant to the research question
are undertaken to identify and collect all materials pertaining to the review. Search
should be inclusive of all published and unpublished studies in any language and from
any country. Before embarking on the search, the team must prepare a search strategy,
list the relevant databases and appropriate search engines, create access account to
databases (e.g. PubMed, Cochrane, EMBASE, Medline, ISI etc. for health/medical
studies), if needed. Study time period should be specified for the search to reflect that
only the studies conducted within the relevant period are considered for the review.
During the search all different combinations of the key/technical words, phrases and
terms as well as their all possible combinations related to the topic of interest must
be included using all available search engines. The search should be extended to
all major languages to make sure that the publications in non-English languages are
fully covered. It is important to record the search date to note the cut-off date up to
which the review entries are included from a particular database.
(a) Reviewing the search outcomes—independent search
At least two members of the review team should conduct independent searches in
all relevant databases and resources taking into account both electronic and paper
version of the materials, and then reconcile the information gathered independently
by members of the team from the identified studies. If needed, a third reviewer may be
engaged to reach agreement on the selection of any disputed studies. Any limitations
or weaknesses of the search should be documented and included in the review report.
At the first stage, the selection of studies is based on the checking the title/heading
of the articles by the independent reviewers. The studies selected at the first stage
are then critically analysed and checked based on the abstract to decide next stage of
selection. The full-text review is then conducted on the studies selected in the second
stage. Thus the identified studies could be excluded in several stages (title, abstract,
full-text) based on the selection and exclusion criteria.
The list of references or bibliographies of the items selected by full-text checking
should be reviewed to identify any additional studies on the topic of interest. The same
search, review, and checking processes and stages should be applied to any studies
identified during this reference search stage to decide on their inclusion/exclusion.
Use of platforms such as Rayyan (rayyan.qcri.org) and Endnote should be considered
when selecting studies for inclusion/exclusion.

8 1 Introduction to Systematic Review
(b) Collection of studies and extraction of data
Once the members of the team who are responsible to conduct the search indepen-
dently identify the articles/studies to be included in the systematic review all related
documents, and records including full-text article, must be collected and listed for
review and record. A well-documented summary of key information on each study
may help conduct the review in a systematic and orderly way. The analytical and crit-
ical review of these documents would lead to the review report and ultimate evidence
to address the research question.
Data extraction from the included study documents is the next step. Data on the
items of interest should be independently collected on a spreadsheet in a predeter-
mined format. The format should allow sufficient flexibility to accommodate vari-
ation as different author may report the data in different format or scale or unit. It
may be a good idea to pilot the data extraction sheet with a subset of the studies
to make sure that the format is robust enough to deal with the diversities, if any.
Data extraction should be conducted by at least two independent reviewers. The data
from independent studies should be compared item by item, and agreement should
be reached on the final figures before embarking on the analyses of the data. In case
of any dispute/disagreement a third reviewer or an expert in the field should be called
to make the final choice.
For the better management of resources all relevant documents from the selected
studies may be saved in a separate folder with a back-up copy in a separate device.
Referencing softwares such as EndNote are invaluable to keep track of all documents
and make referencing easy and handy. In case of any missing or confusing data, the
authors of the relevant articles should be contacted for clarification or requesting the
missing information.
1.4.1 Reporting a Flow-Chart for Study Selection
Here is an example of reporting flowchart of selecting studies for the systematic
review of D1 versus D2 gastrectomy for gastric adenocarcinoma (cf. Memon et al.
2011). In this study the initial search resulted in 29 records, but finally only 6 of them
satisfied the inclusion critera and were included in the meta-analysis (Fig. 1.1).
If any other team of researchers were doing the same systematic review and meta-
analysis independently they should be reporting exactly the same flowchart up till
date of the search. This is because every systematic review must be reproducible.
1.5 Levels of Evidence
Not every type of study provides the same level of evidence. The level of evidence
in any systematic review depends on the design of the primary studies. Often the

1.5 Levels of Evidence 9
Fig. 1.1 Flow chart providing information through the different phases of study selection
research question of the investigation determines the choice of study design. The
primary studies provide the original data and analysis for the research synthesis.
An example of a primary literature source is a peer-reviewed research article. Other
primary sources include preprints, theses, reports and conference proceedings.
The level of evidence from primary sources are broadly categorised based on the
study design as follows (highest to lowest):
(a) Experimental: Randomised or non-randomised controlled trials
(b) Quasi-experimental studies (such as before-and-after study, interrupted time
series)
(c) Observational analytic studies (e.g. cohort study, case-control study).
More detailed rating (highest to lowest) of level of evidence for quantitative questions
in the healthcare studies is found in (Melnyk and Fineout-Overholt 2011).

10 1 Introduction to Systematic Review
1.6 Assessing Quality of Studies
The quality of the selected studies that meet inclusion criteria in the systematic review
directly impact on the quality of evidence produced by the review.
By quality we mean the internal validity of the studies. Internal validity is the
extent to which the analytic study is free from systematic errors and any difference
between interventions is therefore only due to the intervention of interest. The internal
validity is threatened by the methodological errors and varieties of biases such as
selection, information and confounding biases. Depending on the type of study,
scholars/experts have suggested different tools to improve study quality. There are
more than a hundred such tools, and sometimes classified by study design. In addition,
checklists exist to ensure the reporting protocol for the systematic reviews. The
most popular and frequently used tools for assessing quality of primary studies are
summarised in the next section.
The Meta-analysis Of Observational Studies in Epidemiology (MOOSE) group
(Moher et al. 2000) proposed a checklist containing specifications for reporting of
meta-analyses of observational studies. On the other hand the Preferred Reporting
Items for Systematic review and Meta-Analysis Protocols (Moher et al. 2009)was
published in 2015 aiming to facilitate the development and reporting of systematic
review protocols that can be adopted for randomised controlled trials.
There are several measures of study quality in the literature. One old measure
to assess the quality of studies to be included in meta-analysis that are based on
the randomised controlled trials is the Jadad score (Jadad et al. 1996). This score is
also known as the Oxford Quality Scoring System which ranges from zero to five,
zero being the lowest quality and five being the highest achievable quality based
on reporting of randomization, blinding, and withdrawals reported during the study
period.
In qualitative syntheses, researchers stop at the systematic review stage and
the information from independent studies selected by systematic search addresses
the findings without conducting meta-analysis. However, in quantitative syntheses,
numerical data from the selected studies are pooled through meta-analysis. In both
cases the synthesis must be based on all the trials/studies, both published and unpub-
lished, selected via a comprehensive literature search. A general perception is that
the quality of meta-analysis is of the highest level if the study is based on independent
randomised controlled trials (RCTs).
1.6.1 Tools for Assessing Study Quality
To make sure that the evidence is of high quality various tools have been suggested
to assess the quality of the study. Researchers have been continuously trying to come
up with safeguards against biases and design flaws of the individual studies. These

1.6 Assessing Quality of Studies 11
safeguards comprise the tools against which studies are assessed. A brief summary
of some key tools are discussed below.
QUOROM The Quality of Reporting of Meta-analyses (QUOROM) addresses stan-
dards for improving the quality of reporting of meta-analyses of clinical randomised
controlled trials (RCTs) was proposed by Moher et al. (1999). The QUOROM docu-
ment consists of statements, a checklist, and a flow diagram. The checklist describes
the preferred way to present the abstract, introduction, methods, results, and discus-
sion sections of a report of a meta-analysis. The flow diagram provides information
about both the numbers of RCTs identified, included and excluded, and the reasons
for exclusion of trials.
CONSORT Consolidated Standards of Reporting Trials encompasses various initia-
tives developed by the CONSORT Group (Moher et al. 2012) to deal with the prob-
lems arising from inadequate reporting of randomized controlled trials (RCT). The
CONSORT Statement consists of a minimum set of recommendations for reporting
randomized trials. It offers a standard way for authors to prepare reports of trial find-
ings, facilitating their complete and transparent reporting, and aiding their critical
appraisal and interpretation.
PRISMA Preferred Reporting Items for Systematic Reviews and Meta-Analyses
(PRISMA) is an evidence-based minimum set of items for reporting in system-
atic reviews and meta-analyses (Moher et al. 2009). PRISMA team focuses on the
reporting of reviews evaluating randomized trials, but can also be used as a basis for
reporting systematic reviews of other types of research, particularly evaluations of
interventions.
PRISMA-P Preferred Reporting Items for Systematic Review and Meta-Analysis
Protocols (Moher et al. 2015) aiming to facilitate the development and reporting of
systematic review protocols.
MOOSE The Meta-analysis Of Observational Studies in Epidemiology (MOOSE)
group (Stroup et al. 2000) proposed a checklist containing specifications for reporting
of meta-analyses of observational studies in epidemiology, including background,
search strategy, methods, results, discussion, and conclusion. Use of the check-
list should improve the usefulness of meta-analyses for authors, reviewers, editors,
readers, and decision makers.
ROBINS-I Risk Of Bias In Non-randomised Studies of Interventions (ROBINS-I)
proposed by Sterne et al. (2016) is a new tool for evaluating risk of bias in estimates
of the comparative effectiveness (harm or benefit) of interventions from studies that
did not use randomisation to allocate units (individuals or clusters of individuals) to
comparison groups. The tool is particularly useful to those undertaking systematic
reviews that include non-randomised studies.
The main aim of all these processes and protocols is to make the systematic review
as objective as possible by removing potential bias from all possible sources to ensure
high level of evidence. Needless to say that the success of these protocols depends
on the strict adherence to the criteria throughout the systematic review.

12 1 Introduction to Systematic Review
1.7 Concluding Remarks
It is inevitable that more and more decision-makers and organisations will be using
systematic reviews and meta-analyses as the practice of evidence-based decision-
making continues to grow wider. It is essential that everyone involved in the evidence-
based decision-making be evidence-informed so that they could evaluate the studies
for inclusion and their quality so that any recommendations by any systematic review
can be viewed in the context of their overall study quality. The research team must be
well-skilled to decide on what should and should not be included strictly following
the agreed procedure and criteria as well as meeting the underlying assumptions
and satisfying the technical requirements. In case of disagreement/dispute, expert
opinion, past experience and discipline knowledge will be a useful guide for the
research team.
References
Breckon J (2016) Evidence base for evidence-informed policy. Significance Mag 13:12–13
Jadad AR, Moore RA, Carroll D, Jenkinson C, Reynolds DJM, Gavaghan DJ, McQuay HJ (1996)
Assessing the quality of reports of randomized clinical trials: is blinding necessary? Control Clin
Trials 17(1):1–12
Khan KS, Kunz R, Kleijnen J, Antes G (2003) Five steps to conducting a systematic review. J R
Soc Med 96(3):118–121
Khan S, Doi SAR, Memon MA (2016) Evidence based decision and meta-analysis with applications
in cancer research studies. Appl Math Inf Sci 10(3):1–8
Langer L, Tripney J, Gough DA (2016) The science of using science: researching the use of research
evidence in decision-making. UCL Institute of Education, EPPI-Centre
Melnyk BM, Fineout-Overholt E (2011) Evidence-based practice in nursing & healthcare: a guide
to best practice. Lippincott Williams & Wilkins
Memon MA, Subramanya MS, Khan S, Hossain MB, Osland E, Memon B (2011) Meta-analysis
of D1 versus D2 gastrectomy for gastric adenocarcinoma. Ann Surg 253(5):900–911. https://doi.
org/10.1097/SLA.0b013e318212bff6
Memon MA., Khan S, Alam K, Rahman MM, Yunus RM (2020) Systematic REviews: Under-
standing the Best Evidences for clinical Decision-making in Health care: Pros adnd COns,
Surgical Laparoscopy, Endoscopy & Percutaneous Techniques.
Moher D, Cook DJ, Eastwood S, Olkin I, Rennie D, Stroup DF (1999) Improving the quality of
reports of meta-analyses of randomised controlled trials: the QUOROM statement. Quality of
reporting of meta-analyses. Lancet 354(9193):1896–1900
Moher D, Cook DJ, Eastwood S, Olkin I, Rennie D, Stroup DF (2000) Improving the quality of
reports of meta-analyses of randomised controlled trials: the QUOROM statement. Oncol Res
Treat 23(6):597–602
Moher D, Liberati A, Tetzlaff J, Altman DG (2009) Preferred reporting items for systematic reviews
and meta-analyses: the PRISMA statement. Ann Intern Med 151(4):264–269
Moher D, Hopewell S, Schulz KF, Montori V, Gøtzsche PC, Devereaux P, Altman DG (2012)
CONSORT 2010 explanation and elaboration: updated guidelines for reporting parallel group
randomised trials. Int J Surg 10(1):28–55
Moher D, Shamseer L, Clarke M, Ghersi D, Liberati A, Petticrew M, Stewart LA (2015) Preferred
reporting items for systematic review and meta-analysis protocols (PRISMA-P) 2015 statement.
Syst Rev 4:1

References 13
Sterne JA, Hernán MA, Reeves BC, Savovi´c J, Berkman ND, Viswanathan M, Boutron I (2016)
ROBINS-I: a tool for assessing risk of bias in non-randomised studies of interventions. BMJ
355:i4919
Stroup DF, Berlin JA, Morton SC, Olkin I, Williamson GD, Rennie D, Moher D, Becker BJ,
Sipe TA, Thacker SB (2000) Meta-analysis of observational studies in epidemiology: a proposal
for reporting. Meta-analysis of observational studies in epidemiology (MOOSE) group. JAMA
283(15):2008–2012
Yannascoli SM, Schenker ML, Carey JL, Ahn J, Baldwin KD (2013) How to write a systematic
review: a step-by-step guide. Univ Pennsylvania Orthop J 23:64–69