ArticlePDF Available

Abstract

This session comprises four papers that consider how systematic review methods may be developed in order to make the best use of complex evidence in education and health. The methods and approaches reflected upon in these papers are not drawn from a single research tradition, but share a common goal of broadening the methodological scope of systematic reviews and better understanding the utilisation of knowledge produced in this way. The first paper (Henry Potts) reports an ongoing review using a meta-narrative approach to make sense of the diverse sources of knowledge regarding electronic patient records. The review method has stressed the importance of understanding knowledge from within the research tradition in which it was produced; it is argued that this has important implications for the way that evidence is utilised in the policy making process. The second paper (Geoff Wong) reflects upon the experience of using an explicit realist approach in the synthesis of the evidence in Internet based learning. This realist synthesis offers a method of making sense of the highly heterogeneous and context dependent evidence which exists in this field thus enabling greater insights into what makes such educational interventions ‘work’. The third paper (Rod Sheaff) reports a review of the predominantly qualitative research literature on organisational structures and their impacts upon policy outcomes in health systems. A scoping study found 14389 relevant papers of which 1568 were selected for review. These studies were very variable in the amount and quality of the qualitative data, hence 'evidence', which they reported. The paper describes an attempt to adapt realist methods so as to synthesise such bodies of research in ways which take account of this variation in the strength of qualitative evidence. The fourth paper (Mark Pearson) draws upon the work of Donald Campbell and colleagues in order to gain a fuller understanding of how systematic reviews are utilised in the policy making process. It is argued that interpretive approaches to understanding policy making (such as rhetorical analysis) need to be tempered with a more nuanced understanding of research validity. The case is made that interpretive approaches not only can, but should, be melded with research validity to increase understanding of the policy making process.
1
A meta-narrative review of
electronic patient records
Henry W W Potts, Trish Greenhalgh, Deborah Swinglehurst,
Pippa Bark & Geoff Wong
UCL Medical School
9
th
Annual Colloquium of the Campbell Collaboration,
19
th
May 2009, Oslo
e-mail: h.potts@chime.ucl.ac.uk
23 systematic reviews on the electronic
patient record, so why do another one?
No clear messages for practice and policy
Research evidence growing, yet conclusions still:
evidence is conflicting
no definitive solutions
more research needed
Gap between the rhetoric (be it Tony Blair ten years ago, or
Barack Obama today) and the reality of ‘failed’ programmes
Awareness of a broader literature, not indexed on Medline,
not generally covered by existing reviews
2
Not just heterogeneity,
not just mixed methods,
but incommensurability
The meta-narrative approach
Heterogeneity and pluralism
Problems of heterogeneity multiply with more complex questions,
with multiple outcomes, varying systems and different
methodologies – different paradigms
Various approaches developed to review broad methods
Meta-narrative review (developed from the realist review)
Greenhalgh, Robert, Bate, Macfarlane & Kyriakidou (2005).
Diffusion of Innovations in Health Service Organisations: A
Systematic Literature Review. Blackwell BMJ Books.
Use a historical and philosophical perspective as a
pragmatic way of making sense of a diverse literature
3
Key questions (from Kuhn, “The structure
of scientific revolutions”, 1962)
What research teams have researched this area?
How did they CONCEPTUALISE the problem?
What THEORIES did they use to link problem with
potential causes and impacts
What METHODS did they define as ‘rigorous’ and
‘valid’?
Explore the literature
Open-ended question
Meta-narrative review (how to get started)
Research tradition C
Evaluate, summarise
Quality
criteria
Theoretical
basis
Research tradition B
Evaluate, summarise
Quality
criteria
Theoretical
basis
Research tradition A
Evaluate, summarise
Quality
criteria
Theoretical
basis
Meta-narrative map of underpinning traditions
4
Meta-narratives on the EPR in an
organisational context
Health information systems (based in health informatics and EBM,
literature generally covered in Cochrane-style reviews)
Health services research (in the biomedical literature, but focus on
change management)
Patient safety (focus on error)
Computer-supported cooperative work (developed from human-
computer interaction)
Information systems – positivist approaches
Information systems – interpretivist approaches
Information systems – technology-in-practice approaches (chiefly
Orlikowski’s technology structuration)
Critical sociology (feminist and Foucauldian)
Actor-network theory (recursive, post-structuralist approach, including
work of Marc Berg and recent papers from Norway)
Information systems
‘Conventional’ IS research is
positivist: focus on models and
‘resistance’
Practice-based IS research is
interpretivist: Orlikowksi’s
technology structuration, based
on Giddens’ structuration theory
ANT
Critical perspective
Beyond dualism of reality vs.
record-as-model
SCOT: how codes & categories
shape interpretation and use of
technologies
Health information systems
Hopeful literature
Technological determinism &
utopianism
System as ‘black box’
Little more than lip service to a
socio-technical perspective
CSCW
EPR not container of facts but
tool supporting work
Different healthcare
practitioners do different work
so need different records
Challenges idea of an
‘agreeable’ record
5
Silos or interrelated?
Silos
Most health informatics
literature ignores socio-technical
perspectives
Technology structuration largely
US organisational sociologists
and doesn’t cite/is mostly not
cited by European critical
sociologists
Not silos
Biomedicine meets socio-technical
approaches
Cross-disciplinary appeals (Pratt et
al.)
‘Multilingual’ researchers (e.g. Berg)
Socio-technical approaches aligning
CSCW and STS have common roots
in ANT, Zuboff etc.
Links between CSCW and STS over
the years (e.g. Suchman)
Coming together of CSCW, STS and
IS with newer researchers (e.g.
Ellingsen)
Østerlund draws on Orlikowski and
Berg + brings in social psychology
Berg & Bowker (1997), Sociol Quart, 38: 513-37
Berg (1999), Comp Supp Coop Work, 8: 373-401
Berg (2003), Methods Inf Med, 42: 337–44
Ellingsen & Munkvold (2007), Int J Integrated Care, 7
Østerlund (2004), J Center Inf Studies, 5: 35-43
Pratt, Reddy, McDonald et al. (2004), J Biomed Inform, 37: 128-37
Suchman (1994), Comp Supp Coop Work, 2: 21-39
De Vaujany’s categorisation
François-Xavier de Vaujany
(2005), “IT Conceptualization:
Respective Contributions of
Sociology and Information
Systems”, Journal of
Information Technology
Impact, 5(1): 39-58
Causalist (e.g. technological
determinism)
Integrative and recursive –
technology affects context
and context affects
technology (e.g. technology
structuration)
6
Synthesis: A set of tensions
EPR as tool or container vs EPR as actor
Cognitive view of the human subject (user as an information-processor
or decision-maker) vs. relational view (user defined primarily by their
position within a social/socio-technical system)
Context as setting within which EPR is implemented vs context as the
EPR-in-use
Clinical work as decision-making vs clinical work as situated practice;
and knowledge as transferable facts vs knowledge as information-in-
context
Process of change: logic of determinism vs logic of opposition
Success as objectively and prospectively defined vs success as socially
negotiated and context-specific
Scale: bigger the better vs small is beautiful
Conclusions… and beyond the EPR?
Techno-utopian dream: a Big Computer solves everything
Gap between policymakers’ perspective and ‘coal face’
workers
Failure of evidence-based approach
Search for evidence can be very blinkered
Gap between policymakers’ perspective and many researchers’
Gap between research results and conclusions
Focus on outcomes is good, but can obscure details about
how something works Importance of the realist
approach, of theory
The RCT has limits, in treating the intervention as a ‘black
box’ and in controlling for the context, when it’s the details
of the technology and the context that matter
7
Or, as someone else recently put it…
Anderson, Brown, Dowty, Inglesant, Heath & Sasse (2009), Database State,
Joseph Rowntree Reform Trust, p. 47:
“We also need to wean Government off the idea that IT
projects can substitute for effective policy action. For too
long, ministers have used IT as a displacement activity. IT
must rather be seen as just one of the tools of modern
management; and often not be the most important tool (so
neither ministers nor voters should expect too much).
“To paraphrase the late Roger Needham, “if you think IT
is the solution to your problem, then you don't
understand IT, and you don't understand your problem
either.” ”
(emphasis added)
Reflection on the meta-narrative approach
Very different picture to traditional Cochrane
approach
Rich array of theories and methods
Systematic, but interpretive
8
Cite as: Potts HWW, Greenhalgh T, Bark P, Swinglehurst D,
Wong G (2009). A meta-narrative review of electronic patient
records. Presentation at the 2009 International Campbell
Collaboration (C2) Colloquium, Oslo, Norway, May 2009.
Available at
<http://www.campbellcollaboration.org/artman2/uploads/1/Pe
arson_extending_boundaries.pdf>
ResearchGate has not been able to resolve any citations for this publication.
Article
Full-text available
This article analyzes the different phases the Information Systems--Sociology relationship has gone through and points out some specific features of sociologists and Information Systems scientists in their conceptualization of Information Technology (IT). It shows that both academic fields develop more and more convergent theorizations. The first part is centered on an historical analysis of sociology itself. It shows the great comeback of the Object within the sociological field at the beginning of the 80s. Different models have been developed from the generalized kinds of sociology to those that have been focused on the social construction of the Object. These make up sociological groups, which we call "autonomous". The second part presents the sociological approaches used and worked out in the domain of Information Systems (IS). These are presented by means of three historical moments (causalist, actor-based and processual). For each of these stages the influence of sociologies, notably those that deal with the Object, is obvious and models are more or less "illuminated" by means of broader perspectives. In the third part, there is a discussion of ontological differences between the work of sociologists studying IS objects and the work of IS researchers drawing on sociologists' conceptual contributions. Lastly, it seems that if sociology and Information Systems sometimes diverge in the way they study sociotechnical systems, they converge gradually in their conceptualization of the IT artifact.
  • Pratt
  • Mcdonald Reddy
Pratt, Reddy, McDonald et al. (2004), J Biomed Inform, 37: 128-37
  • Østerlund
Østerlund (2004), J Center Inf Studies, 5: 35-43