What is a systematic review?
Terry Hanley & Laura Cutts
Counselling Psychology Review, Vol. 28, No. 4, December 2013 3
© The British Psychological Society – ISSN 0269-6975
Content & Focus: This Special Edition of Counselling Psychology Review is focused on systematic reviews.
Whilst considering the topic for the editorial to begin this Special Edition, we considered one overarching
question to be of fundamental importance to attempt to tackle: What is a systematic review? We decided to
have this as the focus of the editorial in part as a result of discussions with colleagues (both trainee and
qualified psychologists) whose awareness of the answers to the following questions proved limited: What is a
systematic review?; Why are they conducted?; and What does one look like? Following a brief introduction
focused on the history and context surrounding the systematic review, we have, therefore, aimed to address
each of these questions in turn. To end this initial section of the editorial, we provide readers with a check
list of possible sections contained within a systematic review. The aim of this is to hopefully elaborate on the
definitions and the discussions already considered, in order to help the reader more clearly understand what
a systematic review really is. Following this we provide an overview of the seven papers incorporated into this
Special Edition. Five of these provide very practical examples of the factors noted below in action while two
provide further methodological reflections around the use of such research designs.
Keywords: Systematic review; applied psychology; check list.
ROFESSOR Archie Cochrane is cited as
the ‘architect’ of systematic reviews; in
1979 he put forward the idea that,
within the medical profession, critical
summaries of research trials should be
produced (Bower, 2010, p.2). Since the
1970s, systematic reviews have become very
influential in the health care professions.
For example, they play an important role in
the development of the clinical guidelines
set out by the National Institute of Health
and Clinical Excellence (NICE). In the
process of developing their guidelines for
specific problems, NICE adopt a grading
scheme which details how the quality of
evidence is rated. This grading scheme
places systematic reviews of randomised
controlled trials at the top of the pile
(www.nice.org.uk). NICE guidelines in turn
have a large influence on what services are
commissioned. Therefore, as a research
methodology, systematic reviews hold a large
amount of political power and influence
(Hanley et al., 2013).
What is a systematic review, and why
would I want to do one?
Imagine a scenario where, for example, you
wanted to know what research has to say
about the effectiveness of psychological
therapy. In this case you might want to
conduct a review of the literature, because a
review of the literature would bring together
research conducted in this specific area, and
help you answer your question. However, this
approach (or methodology) is potentially
limited. For example, you might only review
studies that you already know have been
conducted (such as pieces of research which
colleagues have conducted, or told you
about), or ones which confirm your hypoth-
esis or argument, whilst neglecting to review
those which disprove your position. There-
fore, a literature review can be criticised for
not being rigorous enough (Bower, 2010).
Alternatively, you might think about
conducting a systematic review, because a
systematic review is designed to overcome this
bias and is a more rigorous, and systematic,
way of reviewing research in a specific area.
4Counselling Psychology Review, Vol. 28, No. 4, December 2013
At its core a systematic review is a
‘method of critically appraising,
summarising, and attempting to reconcile
the evidence’ (Petticrew & Roberts, 2006,
p.15). Dempster (2011, p.15) defines a
systematic review as:
‘a comprehensive review of literature
which differs from a traditional literature
review in that it is conducted in a
methodical (or systematic) manner,
according to a pre-specified protocol to
minimise bias, with the aim of synthesising
the retrieved information.’
So a systematic review is what it says on the
tin – a review of the literature which is system-
atic. Historically, qualitative research was
excluded from systematic reviews (Dixon-
Woods, Fitzpatrick & Roberts, 2001).
However, in recent years there has been a
move towards including diverse types of
evidence within systematic reviews (Dixon-
Woods & Fitzpatrick, 2001), and the guid-
ance on undertaking systematic reviews
published by the NHS Centre for Reviews
and Dissemination specifically considers the
inclusion of qualitative research evidence
(Centre for Reviews and Dissemination,
2009). Methodological papers have, there-
fore, considered both the procedures for the
synthesis of qualitative research evidence
(Timulak, 2009) and how to combine both
quantitative and qualitative research within a
single systematic review (Dixon-Woods et al.,
2005; Harden & Thomas, 2005).
What does a systematic review look like?
Whether conducting a systematic review of
solely quantitative research, qualitative
research, or a combination of both, there is
generally a protocol of steps to follow. Within
the Cochrane Handbook for Systematic Reviews of
Interventions it is stated that a systematic
review has the following characteristics:
l‘a clearly stated set of objectives with pre-
defined eligibility criteria for studies;
lan explicit, reproducible methodology;
la systematic search that attempts to
identify all studies that would meet the
lan assessment of the validity of the
findings of the included studies, for
example, through the assessment of risk
of bias; and
la systematic presentation, and synthesis,
of the characteristics and findings of the
(Higgins & Green, 2008, p.6).
This gives a flavour of what elements might be
contained within a systematic review. Within
this section we take this further and reflect on
what sections you might expect to see within a
systematic review paper. Following this we
have provided readers with a check list which
brings together some of these thoughts and
can hopefully act as a useful tool for those
individuals who are considering producing a
systematic review paper.
Within the introduction to a systematic
review paper two things are required: a brief
discussion of the literature in the area, and a
clear statement of the study aim and
research question considered. Following
this, the methodology section should detail
the process undertaken in the systematic
review. Given the requirement for the
systematic review to have an ‘explicit, repro-
ducible methodology’ (Higgins & Green,
2008, p.6), the methodology section will
often be very detailed. Within this, you
would expect to see a number of important
sections. Firstly, the author(s) should outline
the search procedures used, specifying
where and when they have conducted their
searches, and what search terms they have
used. Eligibility criteria also need to be
discussed: the criteria against which the
author(s) decided whether or not a citation
was relevant to the research. The author(s)
will also commonly discuss data extraction:
what data they extracted from the citation
and how. Quality criteria will outline how the
author(s) have assessed the quality of the
citations, and whether or not any papers
were excluded on the basis of quality (this
section is sometimes combined with eligi-
bility criteria). The procedures of data
synthesis need to been described, and finally,
ethical considerations may be discussed.
Terry Hanley & Laura Cutts
Counselling Psychology Review, Vol. 28, No. 4, December 2013 5
Within the findings section of a systematic
review, typically the study flow is represented
(often diagrammatically). This will outline
how many citations were found at different
levels of the search, and how many were
included/excluded. Following this, the char-
acteristics of the included studies will be
described, and the author(s) will typically
report on the outcome of the quality assess-
ment described above. Finally, the findings
resulting from the synthesis of the data will
be reported. In the final section of the
systematic review paper, readers should
expect to see authors discuss the findings of
the research in relation to their initial
research question and the previous litera-
ture. Limitations of the review and sugges-
tions for further research will typically be
considered, in addition to the implications
or recommendations resulting from the
study. The paper should end on the conclu-
sions drawn from the research.
What is a systematic review?
Brief contextual literature review
Research question or study aim
Eligibility criteria: inclusion and exclusion criteria
Quality criteria and assessment
Characteristics of included studies
Quality of included studies
Synthesis of data
Revisiting the research question
Discussion in relation to previous research
Limitations of the review
Table 1: Check list for systematic review papers.
Overview of the present edition
This Special Edition provides a wide scope to
reflect upon. However, each of these papers
fits into two distinct categories, notably
either as a research paper or a methodo-
logical paper. In relation to the former we
list the titles and authors below:
lWhere do counselling psychologists based
in the UK disseminate their research?
A systematic review.
(Ruth Gordon & Terry Hanley)
lPost-traumatic growth following bereave-
ment: A systematic review of the literature.
(Christina Michael & Mick Cooper)
6Counselling Psychology Review, Vol. 28, No. 4, December 2013
lPsychological treatments for eating
disorders: What is the importance of the
quality of the therapeutic alliance for
(Pavlina Antoniou & Mick Cooper)
lA systematic review of qualitative studies
on shame, guilt and eating disorders.
lThe relationship between children’s
outcomes in counselling and psycho-
therapy and attachment styles.
The titles of these papers speak for them-
selves and thus need little more reflection.
Additionally, and in line with the purpose of
this Special Edition, each paper demon-
strates how the boxes noted above (outlining
what a systematic review is) might be ticked
off. The second category gets a bit more
methodological with the inclusion of the
lHealth Technology Assessment method-
ology: An overview and example of its
potential use in the field of Primary Care
Psychological Therapies in the NH.S
lExperiences of conducting qualitative
(Ladislav Timulak & Mary Creaner)
In these we explicitly enter into the method-
ological complexities of such work. Hope-
fully these papers will support the
development of understanding and lead to
further reflections upon the process
conducting a systematic review.
To end, the ‘Dialogues and Debates’
section once again provides much more
food for thought. Thank you for reading and
we hope you enjoy this Special Edition.
About the Authors
Terry Hanley is Programme Director of the
Doctorate in Counselling Psychology at the
University of Manchester and Editor of
Counselling Psychology Review.
Laura Cutts is a Lecturer in Counselling
Psychology at the University of Manchester.
Bower, P. (2010). Undertaking systematic reviews in
counselling and psychotherapy. Lutterworth: British
Association for Counselling & Psychotherapy
Centre for Reviews and Dissemination (2008).
Systematic reviews. CRD’s guidance for undertaking
reviews in health care. York: CRD. Retrieved
8 October 2013, from:
Dempster, M. (2011). A research guide for health and
clinical psychology. Hampshire: Palgrave Macmillan.
Dixon-Woods, M., Agarwal, S., Jones, D., Young, B. &
Sutton, A. (2005). Synthesising qualitative and
quantitative evidence: A review of possible
methods. Journal of Health Services Research & Policy,
Dixon-Woods, M. & Fitzpatrick, R. (2001). Qualitative
research in systematic reviews. British Medical
Journal, 323, 765–766.
Dixon-Woods, M., Fitzpatrick, R. & Roberts, K. (2001).
Including qualitative research in systematic
reviews: opportunities and problems. Journal of
Evaluation in Clinical Practice, 7(2), 125–133.
Hanley, T., Cutts, L., Gordon, R. & Scott, A. (2013).
A research informed approach to counselling
psychology. In G. Davey (Ed.), Applied psychology.
London: BPS Wiley-Blackwell.
Harden, A. & Thomas, J. (2005). Methodological
issues in combining diverse study types in
systematic reviews. International Journal of Social
Research Methodology, 8(3), 257–271.
Higgins, Julian, P.T. & Green, S. (2008). Cochrane
Handbook for Systematic Reviews of Interventions.
Chichester, West Sussex: The Cochrane
Collaboration and John Wiley & Sons Ltd.
Petticrew, M. & Roberts, H. (2006). Systematic reviews
in the social sciences. A practical guide. Oxford:
Timulak, L. (2009). Meta-analysis of qualitative
studies: A tool for reviewing qualitative research
findings in psychotherapy. Psychotherapy Research,
Terry Hanley & Laura Cutts