Cochrane or Cock-eyed? How Should We Conduct Systematic Reviews of Qualitative Research?

Full text

Cochrane or cock-eyed? how should
we conduct systematic reviews of
qualitative research?
Andrew Booth
Senior Lecturer in Evidence Based Healthcare Information, School of Health
and Related Research, University of Sheffield, S1 4DA Email:
A.Booth@sheffield.ac.uk
Paper presented at the Qualitative Evidence-based Practice Conference,
Taking a Critical Stance. Coventry University, May 14-16 2001
Abstract
The quantitative versus qualitative debate has taken significant steps towards
reconciliation within the wider field of evidence based practice. Nevertheless, far
more insidious discrimination remains. Systematic review methodology exhibits all
the characteristics of "institutionalised quantitativism" in that criteria for a "good"
review are almost entirely determined by the quantitative methods promoted and
perpetuated by the Cochrane Collaboration. Nobody who understands qualitative
research would insist that its primary studies demonstrate alien concepts such as
"sample size" or "statistical power". Yet comparably fundamental absurdities
persist with regard to qualitative syntheses. Why should systematic reviewers of
qualitative research pursue a "gold standard" comprehensive literature search when
concepts such as "data saturation" have an established pedigree? Why should they
apologise for an absence of meta-analysis when little-known techniques such as
meta-ethnography could be included in a reviewer's toolbox? Why shouldn't they
apply systematic, explicit and reproducible principles of thematic or concept
analysis to create syntheses that advance our understanding of qualitative issues
and highlight research gaps? The author draws on experience of a dozen systematic
reviews, a third qualitative, to suggest how systematic reviews of qualitative
research might acquire a methodology that is more sympathetic to the paradigm
within which they are conducted.
Introduction
Meta-synthesis, the "science of summing up"(Light & Pillemer 1984), is a building
block of evidence based practice. It may be either quantitative (meta-analysis) or
qualitative (meta-ethnography). Its value lies in the recognition that busy
practitioners find it almost impossible to make decisions based upon the massive

and increasing volume of research evidence (Sleep & Clark, 1999). They therefore
require an overview of work in a specific area that has been compiled by "finding,
evaluating and synthesising evidence from research studies" (Hek, Langton &
Blunden, 2000). Historical precedent for the value of such an enterprise dates back
at least to Aristotle:
"it is necessary, while formulating the problems of which in our
further advance we are to find the solutions, to call into council the
views of those of our predecessors who have declared an opinion on
the subject, in order that we may profit by whatever is sound in their
suggestions and avoid their errors". (Aristotle, De Anima quoted in
Bland et al, 1995)
Early experiences of evidence based practice, particularly in the context of critical
appraisal, were invariably accompanied by the postscript "But what about
qualitative research?". To a certain extent this polarisation into quantitative and
qualitative camps has been headed off at every level of the evidence based practice
paradigm. Sackett, one of the founding fathers of evidence based healthcare, issued
a message of reconciliation through a BMJ Editorial (Sackett & Wennberg, 1997).
Similarly Greenhalgh, another leading proponent of the fledgling movement, made
a persuasive appeal that "How evidence based is my practice" should be answered
in qualitative as well as quantitative terms (Greenhalgh, 1996) and also advocated
the importance of Narrative Based Medicine (Greenhalgh, 1999). Within the
specific technique of critical appraisal the Critical Appraisal Skills Programme
(CASP) in Oxford, among many others, has developed a checklist for appraising
qualitative research studies. At the producer, as opposed to consumer, end of the
spectrum, recognition by the Cochrane Collaboration of qualitative research was
signalled by the formation of a Qualitative Research Methods Network.
Notwithstanding such advances systematic review methodology exhibits all the
characteristics of "institutionalised quantitativism" in that criteria for a "good"
review are almost entirely determined by the quantitative methods promoted and
perpetuated by the Cochrane Collaboration. This appears to be not so much a
conscious effort by the Collaboration to discriminate against this particular
paradigm of research methodology as a profound lack of awareness of the potential
contribution of qualitative methods of meta-synthesis to our knowledge of
healthcare interventions. A search of the Cochrane Systematic Review
Methodology database in early 1997 yielded only one reference on qualitative
systematic reviews - the seminal work Meta-ethnography by Noblit and Hare
(1988).
Methods
A citation search for the Noblit and Hare reference on the Science Citation and
Social Science Citation Indexes via the Web of Science yielded 15 relevant

articles. These comprised both methodological discussions and case studies of the
use of meta-ethnographic approaches. This led, in turn, to a subject search using
the keywords "meta-ethnograph$", "metaethnograph$", "meta-synthesis", and
"metasynthesis" on the MEDLINE, CINAHL and Science and Social Science
Citation Indexes to retrieve additional articles. This cursory literature search was
designed to provide an external frame of reference against which the experience of
the author, in conducting four collaborative systematic literature reviews, could be
compared. These four systematic reviews, underpinned to varying degrees by the
methods described by Noblit and Hare, were as follows:
 A systematic review for the NHS Health Technology Assessment
Programme on the Use of Modelling in the Planning and
Prioritisation of Clinical Trials (Chilcott et al, In press)
 A systematic review for the NHS Health Technology Assessment
Programme on the Use of Health-related Quality of Life Measures in
Economic Evaluation (Brazier et al, 1999)
 A systematic review for the English National Board for Nursing,
Midwifery and Health Visiting on the nursing contribution to
rehabilitation (Nolan et al, 1997)
 A systematic review for the American Association for Retired
Persons on coping with multiple chronic health conditions in older
persons (Dowzer et al, 2000)
Literature Review
Techniques in meta-ethnography originated within the field of education as a
means of synthesising the findings from school inspection reports (Noblit and
Hare, 1988). Following identification of a definable set of studies an iterative
process of analysis takes place, typically using the using the constant comparative
method (Glaser & Strauss, 1967; Kearney 1998a). This leads to the identification
of themes or metaphors that permeate the literature (Beck, 2001). The potential of
meta-ethnography as a tool for knowledge synthesis was identified at an early
stage of its development (Klein, 1989). From education it was a comparatively
short migration to the contiguous field of sociology where Zhao (1991) placed
such techniques within the broad categorisation of meta-study and its associated
branches of metatheory, metamethod and metadata-analysis.
Within the healthcare literature meta-synthesis appears most frequently in nursing
research studies (Sherwood, 1999). Occasionally meta-ethnography's origins in
education resurface intriguingly in a nursing context (Beck, 2001).
It is interesting to observe clusters of studies around a small number of broad
topics:

 Chronic disease (Barroso and Powell-Cope, 2000; Thorne &
Paterson, 1998)
 Coping (Burke et al, 1998; Paterson et al, 1998)
 Caring (Beck, 2001; Frederiksson, 1999; Sherwood, 1997)
Ironically the burgeoning interest in the application of the Cochrane
Collaboration's systematic reviews methods to the fields of social work, law and
education has resulted in the formation of a parallel organisation, the Campbell
Collaboration. This in turn has led to a resurgence for meta-ethnography in the
education literature, alongside other methods of synthesis such as narrative
reviews, vote-counting reviews, meta-analyses and best evidence synthesis
(Davies, 2000).
Applications of meta-synthesis within evidence based practice
Qualitative meta-synthesis can make a number of possible contributions to
evidence based practice. It can be used:
1. For the generation of models and theories (Estabrooks et al, 1994;
Finfgeld, 1999; Thorne and Paterson, 1998).
2. For the subsequent validation and gauging of the empirical support
for theories (Forte, 1998).
3. In methodological reviews as a means of determining the existence of
certain bodies or "schools" of thought.
4. To chart the development of concepts longitudinally in the literature
over a period of time (Thorne and Paterson, 1998).
5. To complement the findings and interpretation of quantitative
systematic reviews, for example in looking at issues around patient
acceptance or compliance.
6. As a means of identifying significant domains or attributes to enable
the development of prototype instruments or scales.
Unlike a quantitative systematic review or meta-analysis where the review process
is conducted linearly in a series of discrete, sequential steps and where deviation
from an a priori protocol poses threats to validity, the corresponding method in a
qualitative review involves continual iteration and retracing of steps. In fact it is
arguable whether an a priori review protocol is appropriate to an interpretative
process such as meta-ethnography (itself more akin to the process of discovery
initiated by a grounded theory approach in primary qualitative research). However
for ease of presentation it will suffice to examine each of the stages of the review
process for a qualitative review in turn.

Study Identification
Why should systematic reviewers of qualitative research pursue a "gold standard"
comprehensive literature search when concepts such as "data saturation" have an
established pedigree? According to NHSCRD guidelines for conducting systematic
reviews of effectiveness (2000):
"The aim of the search is to generate as comprehensive a list as
possible of primary studies, both published and unpublished, which
may be suitable for answering the questions posed in the review.
Identification of relevant literature by a thorough unbiased search
strategy is crucial. This is because the validity of the findings of a
review is directly related to the comprehensiveness of the literature
search that is used to capture the relevant studies."
Here is an example of an instance where the "big is beautiful " approach to
systematic reviews required by quantitative reviews is implicitly extended to cover
qualitative reviews. Many of the techniques for checking the validity of systematic
reviews are underwritten by the fundamental assumption that one is attempting to
identify, if not the whole population of studies, as many relevant articles as
possible. The funnel plot, for example, is an attempt to quantify the effect that
missing studies from this optimal population might have on the results of the
review. The intent of the literature search for a quantitative review is to identify a
statistically representative sample of the published literature.
Methodological reviews, such as those produced for the NHS Health Technology
Assessment Programme, make much of the virtue of the comprehensiveness of the
data sources that have been searched. However, such reviews belong more
appropriately within a qualitative paradigm. Is it necessary for a qualitative review
to be equally comprehensive in its coverage? At this stage it will be appropriate to
draw a parallel, not with quantitative systematic reviews, but with primary
qualitative research. Statistical representativeness is not a primary requirement for
qualitative data collection where systematic non-probabilistic sampling is more
commonly used (Mays and Pope, 1996). If we think of the papers in a qualitative
review as being 'informants' then we seek to identify specific groups of papers that
possess characteristics that are relevant to the phenomenon being studied. We aim
to include a wide range of types of papers (i.e. reflecting as many of the themes or
schools of thought as possible) and, preferably to select 'key informants' (i.e.
papers that lead us to important sources of knowledge in the form of additional
citations). The concept of 'data saturation' is also of relevance here. Once a
particular theme has been identified further occurrences of this theme are only of
interest in strictly quantitative terms unless they expand on or modify an already-
identified theme. Interestingly quantitative reviewers are currently seeking
methods to define a 'law of diminishing returns' beyond which further literature
searching has little justification in order to manage the inordinate expense of the

searching process. For qualitative reviews the answer to this problem already exists
in the principles of data saturation used in primary studies.
Case study : Systematic review on the Use of Modelling in the Planning and
Prioritisation of Clinical Trials
Although the proposal originally submitted to the NHS HTA Programme focused
on traditional methods of comprehensive literature searching we also piloted
methods of 'citation snowballing' to investigate their potential for methodological
reviews. A team of subject experts identified seven key papers representing a
number of perspectives on the use of modelling. References at the end of these
papers were examined in context and a further 34 papers were identified in this
manner. References at the end of these 34 papers led in turn to a further 200
papers. Unfortunately the resources for this project did not allow more than two
'rounds' of this approach - we were unable to follow up the next 'round' of
references (n=425). In an ideal world this process would be continued until a point
of data saturation is reached i.e. no new references are identified.
Within this context the technique of citation searching is of particular interest.
Citation searching, although necessarily complementary to comprehensive
literature searching, is viewed with some caution by quantitative reviewers. It can
lead to reference or citation bias whereby supporting articles are more likely to be
cited and conflicting articles are more likely to be omitted. However, in contrast to
the effectiveness literature, where such dangers are likely to be critical, there is
little reason to believe that this is equally problematic in the literature of
methodology where techniques of argument and counter-argument are essential for
the development of ideas and schools of thought. Using the analogy of the 'key
informant', however, one might propose that a 'key' article in this context is one
that references as broad a selection of seminal articles as possible rather than one
which is itself cited by numerous articles! In practice triangulation of both
approaches with citation searching (i.e. forward and backwards) will be required.
So does the following statement from the NHS CRD guidelines apply equally to
qualitative reviews? :
"The thoroughness of the literature search is one factor that
distinguishes systematic reviews from traditional reviews. It is also
important to ensure that the process is as thorough and unbiased as
possible".
It might be argued that, while it is certainly important for the search process to be
free from bias, it is more important that the search process be systematic, explicit
and reproducible rather than comprehensive. Thoroughness in this context should
apply to the rigour of the search process not to its comprehensiveness. To argue
otherwise would most definitely be "cock-eyed".

Literature searching for qualitative systematic reviews should exhibit the following
characteristics:
a) Identifying major "schools of thought" in a particular area whilst being alert to
the identification of variants, minority views and dissenters. It is particularly
important to identify negative or disconfirming cases (Paterson et al, 1998)
b) Searching within a broad range of disciplines so as to bring different views (e.g.
clinician, consumer, manager, health economist, statistician, research
commissioner etceteras) to bear on the topic in hand.
c) Using complementary electronic and manual search techniques to ensure that
materials are not missed either through the inadequacies of indexing or through
selective coverage of databases
Study Appraisal
Techniques in study appraisal are also dominated by the quantitative paradigm.
Quantitative systematic reviews run the danger of underestimating the evidence
from relevant literature if they only include that of a certain methodological quality
(Edwards et al, 1998). This might result in missing or distorting the true message
that the review is trying to identify. Perversely, using 'weaker' study designs tends
to have the effect of amplifying the strength of a demonstrable effect and so
quantitative reviewers are usually willing to trade the loss of additional
perspectives against improved precision of findings. Clearly, in a qualitative
review, if research is rejected on the basis of design alone there is a very real risk
of denying valuable insights that contribute to our interpretation of a phenomenon.
Qualitative research, with its relativist
perspective, acknowledges the existence of
multiple views
of equal validity (Popay et al, 1998) and is not readily amenable to
the use of the hierarchies of evidence promulgated by evidence based medicine
(Barbour, 1998; Barbour, 2001).
Edwards and colleagues (1998) recognise the importance of assessing the message
or 'signal' within each individual piece of research. They advocate balancing an
assessment of methodological quality against the weight of its message, rather than
rejecting studies that fall below a certain quality threshold. They argue that
fundamentally flawed research will still be rejected, but other papers will have the
importance attached to their signal tempered by the amount of 'noise' around that
signal. The balance of these two elements, termed the 'signal to noise ratio', is of
great relevance to techniques of meta-ethnography.
Barbour (2001) has recently introduced an important dissenting voice to the
stampede to adopt critical appraisal techniques in order for qualitative research to
"pass muster" in the new regime of evidence based practice. She describes a
number of "technical fixes" (such as purposive sampling, grounded

theory,
multiple coding, triangulation, and respondent validation) that simply may
serve to reassure those that question the contribution that qualitative research might
make and yet not advance a broader
understanding of the rationale and assumptions
behind qualitative research.
Study Analysis
With the cautions of Barbour (2001) about the cosmetic overuse of multiple coding
still fresh in mind it is perhaps imprudent to assert the need for two observers in
qualitative reviews when data extraction and, more importantly, coding of themes
takes place (Compare Paterson et al, 1998). Barbour (2001) correctly identifies that
"multiple coding concerns the same issue as the quantitative equivalent "inter-rater
reliability" and is a response to the
charge of subjectivity sometimes levelled at the
process of qualitative
data analysis". On the surface it appears another example of
qualitative reviewers having to play by the rules pioneered by quantitative
reviewers. However to what extent is the use of multiple coders a property of the
material being reviewed and how much is it a function of essential systematic
review methodology?
Here the answer to our question does not seem to lie in the fact that both
quantitative and qualitative reviewers have arrived at a common device of multiple
observers. Rather its importance lies in why it is being done. In quantitative
reviews the aim is to achieve an acceptable degree of concordance between
researchers, thereby increasing confidence in the validity of the review. This is
usually assessed using the device of a kappa statistic. However as Barbour (2001)
again observes, admittedly in the context of primary qualitative research:
"the degree of concordance between researchers is not really
important; what is ultimately of value is the content
of disagreements
and the insights that discussion can provide for refining coding
frames. The greatest potential of multiple
coding lies in its capacity to
furnish alternative interpretations
and thereby......in alerting
researchers to all potentially
competing explanations. Such exercises
encourage thoroughness,
both in interrogating the data at hand and
in providing an account
of how an analysis was developed. Whether
this is carried out
by a conscientious lone researcher, by a team, or
by involving
independent experts is immaterial: what matters is that a
systematic process is followed and that this is rendered transparent in
the
written research
project".
The use of multiple observers in study analysis for qualitative reviews, such as was
the case in our comorbidity review, should be clearly defined as being for the
unique requirements of qualitative research. It should not be misinterpreted as an
attempt to adopt uncritically the model promulgated by quantitative reviews of
effectiveness.

The 'signal to noise' metaphor from the field of communication (mentioned above)
is also pertinent to debates concerning whether one should conduct meta-
ethnography of authors' own interpretations of data or only of verbatim comments
reported by authors in their texts. It is possible to see the two sorts of data, that is:
 Primary data from transcripts, and
 Secondary data from published articles
as being complementary sources of a 'signal'. Primary data has less potential for
noise but is of such large quantities that it is very difficult to pinpoint the exact
nature of a 'signal' while secondary data is much more selective, is amplified by the
authors' interpretations but therefore runs the risk of losing integrity. Interestingly
such issues are analogous to debates about individual patient data versus literature
(Stewart & Parmar, 1993) in the meta-analytic domain.
Thematic analysis can be characterised within two approaches:
A) An approach analogous to grounded theory (Wuest 2000) is used whereby the
constant comparative method produces insights from the literature that are
'discovered' rather than being a preconceived framework used by the researcher.
B) An existing organising construct is used to provide meaning to the disparate
themes identified from the analysis (Frederiksson, 1999).
The reviews outlined in this paper include both approaches to thematic analysis;
the comorbidity review used a grounded theory approach to discover approaches to
coping as documented in the literature while the rehabilitation review used a
conceptual framework by Corbin and Strauss (1988) as a means of structuring
themes according to the characteristics of the target conditions.
Study Synthesis
According to Sandowski and colleagues (1997) the accumulation of qualitative
studies in recent years has led to little cumulation of the understandings gained
from them. Techniques used have included reciprocal translations of key
metaphors and concepts and qualitative and quantitative comparative analyses to
produce narrative and theoretical integrations. The major problem, yet to be
resolved, is developing usable and communicable systematic approaches that
synthesise composite findings and yet maintain the integrity of individual studies.
Once the synthesis has been completed follow-up validation strategies may be
required to safeguard the validity of the findings (Mays and Pope, 1996). Several
techniques are available and some of these have been employed in the reviews
outlined in this paper:

a) "Triangulation", comparing oral testimony from subjects with the written record
of the systematic review, was used in the review for the American Association for
Retired Persons. Phase 2 of this project involved eliciting data from those
experiencing multiple chronic health conditions to see if their perceptions matched
the framework derived from the literature.
b) The framework developed from the methodological literature from the review
on the use of modelling in trials was validated by the random selection of case
studies to be assessed against this framework
c) A checklist derived from the literature of the use of health-related quality of life
measures in economic evaluation was validated by sampling from a single year of
the health economics literature.
d) Findings from the general literature on the role of the nurse in rehabilitation
were then examined using theoretical sampling in conditions that were
characterised according to a two-by-two matrix of acute versus chronic and young-
onset versus old-onset.
Other NHS Health Technology Assessment reviews have attempted to validate
constructs from the literature in interviews or with focus groups. From the broader
literature Khunti (1999) describes how multiple methods (systematic literature
searching, brainstorming with health professionals and patients, a focus group and
interviews with key informants) can be used for hypothesis generation. It is
interesting in this context to note that the literature review identified 37 out of 54
putative factors (68.5%) affecting the quality of care of patients with diabetes. A
combination of brainstorming and literature review identified 51 out of 54 factors
(an impressive 94.4%).
Meta-study permits critical analysis of the location of current inquiry within the
larger context of systematic patterns in knowledge development (Thorne and
Paterson, 1998). It is important to note, however, that the sheer quantities of data
involved may result in certain arbitrary decisions being made. For example Thorne
and Paterson (1998) only examine in depth one of a large number of themes
identified while it is not unusual (in common with primary qualitative research) for
researchers to report only the "top themes" identified from the literature which may
result in the loss of important concepts.
In addition to detailed exposition of multiple themes and in-depth analysis of those
regarded as being specifically important it is essential that the qualitative reviewer
indicates the existence of negative or disconfirming cases (Paterson et al, 1998).
This is particularly the case in methodological reviews where a minority approach
may have tremendous potential for development but be otherwise overshadowed
by the existence of a substantive corpus of work. This is analogous to the

importance of investigating sources of homogeneity in quantitative meta-analyses
(Thompson, 1994).
Summary
From the brief exploration above it is possible to identify some of the defining
characteristics of qualitative systematic reviews:
1. Their intention is not to identify all literature on a particular topic.
Whilst the aim is to identify specific groups of papers that possess
characteristics that are relevant to the phenomenon being studied this
need not imply statistical representativeness.
2. Particular attention is focused on negative or disconfirming cases.
This adds to the richness of the insight that the review provides on the
phenomenon of interest.
3. Such reviews acknowledge the existence of multiple views. Caution
must therefore be exercised so that valuable insights are not lost by
application of some a priori definition of quality through critical
appraisal.
4. Although use of multiple reviewers may be in common with
quantitative reviews the essential contribution of this to a qualitative
review lies in its capacity to furnish alternative interpretations,
thereby alerting researchers to potentially
competing explanations.
5. Whereas quantitative reviews seek to handle "noise" by exclusion
(through the application of a predetermined quality threshold)
qualitative reviews seek to optimise the "signal to noise" ratio.
6. Following on from (5) it will often be necessary to validate the
findings from a qualitative review with reference to a subset of case
studies, use of key informants or focus groups or other methods of
triangulation.
The Way Forward?
As qualitative methods become popular and qualitative reports abundant, it is a
sign of methodological maturity when researchers discuss techniques for
synthesising findings about related phenomena from diverse samples (Kearney,
1998b). Advocates of evidence based practice are starting to look beyond the more
established models of research synthesis for innovative approaches (Greenhalgh,
1998; Sleep & Clark, 1999). Qualitative metasynthesis is one response to concerns
about the relevance and utility (Sandelowski, 1997; Sandelowski et al, 1997) of
qualitative research, and much work remains to be done in developing and
harnessing appropriate methodologies. Given the pioneering role of the Cochrane
Collaboration in developing review methodology it is doubly unfortunate that
opportunities to develop qualitative systematic reviews are constrained by:

1. the implicit assumption that systematic review methods that are
conceived and developed in the context of quantitative reviews
should be translated uncritically for use in qualitative reviews.
2. the apparent willingness of the Cochrane Collaboration Qualitative
Research Network to adopt a "handmaiden" rather than "partnership"
role in contributing to methodological developments.
This latter constraint is best exemplified by an extract from the objectives from the
draft module of the proposed Qualitative Research Methods Group:
 Develop and disseminate appropriate methodological standards
for combining data from qualitative studies with other data within the
context of a systematic review.
There is currently no forum for the development of qualitative systematic
reviews per se (as opposed to the use of qualitative data within systematic
reviews). This deficiency is shared by methodological reviews that seek to embody
existing schools of thought or isolated viewpoints in synthetical documents that
advance the state of our knowledge. Paradoxically this includes documents, such as
those produced by the NHS Health Technology Assessment Programme, that
inform the conduct of quantitative systematic reviews! Now that is what I call
cock-eyed!
References
Barbour RS (1998) Mixing qualitative methods: quality assurance or qualitative
quagmire? Qualitative Health Research 8: 352-361
Barbour RS (2001) Checklists for improving rigour in qualitative research: a case
of the tail wagging the dog? BMJ 322: 1115-1117.
Barroso J & Powell-Cope GM (2000) Metasynthesis of qualitative research on
living with HIV infection. Qualitative Health Research 10 (3): 340-353.
Beck CT (2001) Caring within nursing education: A metasynthesis. Journal of
Nursing Education 40 (3): 101-109.
Bland CJ, Meurer LN & Maldonado G (1995) A systematic approach to
conducting a non-statistical meta-analysis of research literature. Academic
Medicine 70 (7): 642-653.
Brazier, J., Deverill, M., Green, C., Harper, R. & Booth, A. (1999). A review of the
use of health status measures in economic evaluation. Health Technology
Assessment 3 (9).

Burke SO, Kauffman E, Costello E, Wiskin N & Harrison MB (1998) Stressors in
families with a child with a chronic condition: an analysis of qualitative studies and
a framework. Canadian Journal of Nursing Research 30(1): 71-95.
Chilcott J, Brennan A & Booth A (In press) The Use of Modelling in the Planning
and Prioritisation of Clinical Trials. Health Technology Assessment
Corbin JM and Strauss A (1988) Unending work and care: managing chronic
illness at home. Jossey-Bass, San Francisco.
Davies P, Wolf FM & Holmes L (2000) An Organisational Framework for
Preparing and Maintaining Systematic Reviews in Education: A Discussion
Document.http://campbell.gse.upenn.edu/papers/9a_framework.html [accessed
01/05/2001]
Davies P (2000) The relevance of systematic reviews to educational policy and
practice. Oxford Review of Education 26 (3-4): 365-378.
Dowzer C, Basford L, Booth A & Poon, LW (2000) Living with Multiple Health
Conditions: a systematic search and literature review. [A report for the American
Association of Retired Persons]. Sheffield: School of Health and Related Research
(ScHARR), University of Sheffield.
Edwards AG, Russell LT & Stott NCH (1998) Signal versus noise in the evidence
base for medicine: an alternative to hierarchies of evidence? Family Practice 15
(4): 319-322
Estabrooks CA, Field PA & Morse JM (1994). Aggregating qualitative findings:
An approach to theory development. Qualitative Health Research 4: 503-511.
Finfgeld DL (1999) Courage as a process of pushing beyond the
struggle. Qualitative Health Research 9 (6): 803-814.
Forte JA (1998) Power and role-taking: a review of theory, research, and
practice. Journal of Human Behavior in the Social Environment 1(4): 27-56.
Fredriksson L (1999) Modes of relating in a caring conversation: a research
synthesis on presence, touch and listening. Journal of Advanced Nursing 30(5):
1167-76.
Glaser B & Strauss A (1967) The discovery of grounded theory. Chicago: Aldine.
Greenhalgh, T (1996) "Is my practice evidence-based?" BMJ 313: 957-958.
Greenhalgh T (1998) Meta-analysis and beyond: applying secondary research
methods to primary care. British Journal of General Practice 48 (433): 1540-1541.

Greenhalgh, T (1999) Narrative based medicine in an evidence based
world. BMJ 318: 323-325.
Hek G, Langton H & Blunden G (2000) Systematically searching and reviewing
literature. Nurse Researcher 7 (3): 40-57.
Jensen LA & Allen MN (1994) A synthesis of qualitative research on wellness-
illness. Qualitative Health Research 4: 349-69.
Jensen L & Allen M (1996) Meta-synthesis of qualitative findings. Qualitative
Health Research 6: 553-560.
Kearney MH (1998a) Truthful self-nurturing: a grounded formal theory of
women's addiction recovery. Qualitative Health Research 8 (4): 495-512.
Kearney MH (1998b) Ready-to-wear: Discovering grounded formal
theory. Research in Nursing & Health 21 (2): 179-186.
Klein SS (1989) Research and Practice - Implications for Knowledge Synthesis in
Education. Knowledge-Creation Diffusion Utilization 11(1): 58-78.
Khunti K (1999) Use of multiple methods to determine factors affecting quality of
care of patients with diabetes. Family Practice 16(5): 489-94
Lemmer B, Grellier R & Steven J (1999) Systematic review of nonrandom and
qualitative research literature: exploring and uncovering an evidence base for
health visiting and decision making.Qualitative Health Research 9(3): 315-28.
Light RJ & Pillemer DB (1984). Summing up: the science of reviewing
research. Cambridge, MA: Harvard University Press.
Mays N & Pope C (1996) Qualitative research in health care. London: BMJ
Publishing Group.
NHS Centre for Reviews and Dissemination (2000). Undertaking Systematic
Reviews of Research Effectiveness: Guidelines for those carrying out or
commissioning reviews. Report 4. 2
nd
edition. York: NHS Centre for Reviews and
Dissemination, University of York.
Noblit GW & Hare RD (1988) Meta-ethnography: synthesising qualitative
studies. (Qualitative Research Methods, Volume 11) London: Sage.
Nolan M., Booth A. & Nolan J (1997) New directions in rehabilitation: exploring
the nursing contribution. Research Reports series No. 6. London: English National
Board for Nursing, Midwifery and Health Visiting.

Paterson BL, Thorne S & Dewis M (1998) State of the science. Adapting to and
managing diabetes. Image - the Journal of Nursing Scholarship 30(1): 57-62.
Popay J, Rogers A & Williams G (1998). Rationale and Standards for the
Systematic Review of Qualitative Literature in Health Services
Research. Qualitative Health Research 8 (3): 341-351.
Sackett DL and Wennberg JE (1997) Choosing the best research design for each
question. BMJ 315: 1636.
Sandelowski M (1997) To be of use: Enhancing the utility of qualitative
research. Nursing Outlook 45: 125-132.
Sandelowski M, Docherty S & Emden C (1997) Qualitative meta-synthesis: Issues
and techniques. Research in Nursing & Health 20: 365-371.
Sherwood GD (1997) Meta-synthesis of qualitative analyses of caring: defining a
therapeutic model of nursing. Advanced Practice Nursing Quarterly 3(1):32-42.
Sherwood G (1999) Meta-synthesis: merging qualitative studies to develop nursing
knowledge. International Journal for Human Caring 3 (1): 37-42.
Sleep J & Clark E (1999) Weighing up the evidence: the contribution of critical
literature reviews to the development of practice. NT Research 4(4): 306-14.
Smith M & Stullenbarger E (1991) A prototype for integrative review and meta-
analysis of nursing research. Journal of Advanced Nursing 16(11): 1272-83.
Stewart L & Parmar MKB (1993) Meta-analysis of the literature or of individual
patient data: Is there a difference? Lancet 341: 418-422.
Thompson SG (1994) Why sources of heterogeneity in meta-analysis should be
investigated. British Medical Journal 309: 1351-1355.
Thorne S & Paterson B (1998) Shifting images of chronic illness. Image - the
Journal of Nursing Scholarship 30(2):173-8
Wuest J (2000) Negotiating with helping systems: An example of grounded theory
evolving through emergent fit. Qualitative Health Research 10 (1): 51-70
Zhao SY (1991) Metatheory, Metamethod, Meta-Data-Analysis - What, Why, and
How. Sociological Perspectives 34 (3): 377-390.
This document was added to the Education-line database on 11 May 2001