ArticlePDF Available

Realistic EvaluationRay Pawson and Nick Tilley, Sage, London, 1997, 256 pages

Authors:
Book reviews/ The Social Science Journal 41 (2004) 147–161 153
Craig A. Gallet
Department of Economics
California State University at Sacramento
Sacramento, CA 95819-6082, USA
Tel.: +916-278-6223; fax: +916-278-5768
E-mail address: cgallet@csus.edu
doi: 10.1016/j.soscij.2003.10.016
Realistic Evaluation
Ray Pawson and Nick Tilley, Sage, London, 1997, 256 pages
Realistic Evaluation is an original, innovative, and thought-provoking book that I recom-
mend to any scholar interested in program evaluation. Pawson and Tilley provide their readers
with a blueprint for realistic evaluation and display a wealth of detailed examples to illustrate
their approach. The book is an application of critical realism to specific micro contexts. The
authors tried to take critical realism from the metatheory domain to the policy domain by
showing how one can use critical realism in specific policy contexts.
Theauthorsstartwithanintroductorychapterinwhichtheysumupthehistoryofevaluation.
Theythoroughlydelineatefourgenerationsofparadigmsthatprogramevaluationwent through
(i.e., experimentation, pragmatism, constructivism, and comprehensive paradigms). In chapter
two Pawson and Tilley take on the task of comparing and contrasting the “successionist” (or
positivist) and “generative” (or realist) theories of causation. They argue that experimental
evaluation cannot take into account the importance of the generative mechanisms and the
richness of heterogeneous contexts. Their constructive criticism of the successionist approach
inthischapterlaysthegroundworkfor the general framework that the authors invitethe readers
to consider (i.e., realistic evaluation).
Chapter three is where Pawson and Tilley’s critical realist contribution is formulated. The
authors provide a thorough explanation showing how programs work: “causal outcomes follow
from mechanisms acting in contexts.” They even go on to make it so simple as to put it in an
equation format for the readers.
Outcomes =mechanisms +context
Note that the “plus sign” in the above equation is not an additive sign, rather it depicts the
necessary interrelation between contexts and mechanisms that must exist in order for programs
to generate outcomes. This formulation provides the reader with a powerful tool to hypothe-
sise the existence of contexts and mechanisms in the process of explanation and evaluation.
Throughout the book, Pawson and Tilley illustrate their context–mechanism–outcome config-
uration (CMO) with a number of diagrams to help the readers visualize the complexity of the
issues at hand in a simple way. They also provide a wide range of practical interdisciplinary ex-
amples from the fields of sociology, social policy, criminology, health, and education to invite
the readers to consider the method of realistic evaluation in their own field of research. Pawson
and Tilley argue that the goal of the realistic evaluator would be to formulate hypotheses about
154 Book reviews/ The Social Science Journal 41 (2004) 147–161
potential CMO configurations. In order to achieve this goal, the authors declare themselves
to be “whole-heartedly pluralists when it comes to the choice of method.” This leads to the
discovery of what the authors call “the three Ws” or “the realist mantra”; namely: “what works
for whom in what circumstances?”
The authors turn the philosophical notion of critical realism into a methodological notion.
Theydemonstrate howactioniscausedonly if its outcome istriggeredbyamechanism acting in
a context. Pawson and Tilley introduce critical realism as an actual method of experimentation
armedwithcausality, thusbringingcriticalrealismfromanabstractlevelto the levelofresearch
in action (i.e., policy-making).
Pointing to the complexity of the working of programs, Pawson and Tilley recommend
conducting evaluations for subgroups within programs. They do, however, caution the readers
that there might be more than one mechanism at work within each subgroup, thus generating
mixed results.
After outlining the process of designing realistic evaluation in chapter four, Pawson and
Tilley go on to delineate the process of “realistic cumulation” in chapter five. This process,
also described as “theory building,” consists of building a typology of CMO configurations
through continuously and progressively refining the researchers’ understanding of the CMO
patterns by abstracting patterns of contexts, mechanisms, and outcomes during the research
process.
Chaptersixintroduces the readers to therealistdatacollection as well as therealistinterview-
ing methodology; both of which reflect the authors’ philosophy “that knowledge acquisition
is dominated by and organized around the development of realist propositions linking mech-
anisms, contexts and outcomes.” Pawson and Tilley dedicate chapter seven to a rich dialogue
betweenaprogram evaluatorandpolicy-makers regardingasmoke-cessationprogram. In chap-
ter eight, the authors point out the importance of engaging in a teacher-learner cycle between
policy-makers and program evaluators. Pawson and Tilley conclude the book with a summary
chapter for realistic evaluation.
This book is an application of critical realism to specific micro contexts. It is important to
know that, according to critical realism, all micro social contexts are informed and affected
by macro social and philosophical ontology. In my humble opinion, Pawson and Tilley fail to
blend the philosophical ontology of critical realism to its social ontology before moving into
program evaluation. Constructing a consistent and a solid paradigmatic base (philosophical
ontology, epistemology, and social ontology) is a prerequisite for a sound program evaluation.
Needlesstomentionthatin their criticism of positivismforinstance,the authors did not address
any criticism to the positivist social ontology. Having said this, I must confess that Realistic
Evaluation has taken critical realism several steps forward by setting up a blueprint and a
challenging interdisciplinary research agenda for critical realists to pursue. This achievement
I applaud! Fadhel Kaboub
Department of Economics, University of Missouri, Kansas City
5100 Rockhill Road, Kansas City, MO 64110, USA
Tel.: +1-816-235-5648; fax: +1-816-235-2834
E-mail address: kaboubf@umkc.edu
doi: 10.1016/j.soscij.2003.10.017
... The realist evaluation, a theory-driven approach, guided the inquiry. 15 The goal of realist evaluation is about learning 'for whom, in what circumstances, and in what respects a programme works', 15,16 through identifying, testing and refining programme theories. Therefore, realist evaluation starts with an initial programme theory and the goal is to obtain a more refined programme theory. ...
... The realist evaluation, a theory-driven approach, guided the inquiry. 15 The goal of realist evaluation is about learning 'for whom, in what circumstances, and in what respects a programme works', 15,16 through identifying, testing and refining programme theories. Therefore, realist evaluation starts with an initial programme theory and the goal is to obtain a more refined programme theory. ...
Article
Full-text available
Background: An estimated 7.9 million people were living with HIV in South Africa in 2017, with 63.3% of them remaining in antiretroviral therapy (ART) care and 62.9% accessing ART. Poor retention in care and suboptimal adherence to ART undermine the successful efforts of initiating people living with HIV on ART. To address these challenges, the antiretroviral adherence club intervention was designed to streamline ART services to ‘stable’ patients. Nevertheless, it is poorly understood exactly how and why and under what health system conditions the adherence club intervention works. Objectives: The aim of this study was to test a theory on how and why the adherence club intervention works and in what health system context(s) in a primary healthcare facility in the Western Cape Province. Method: Within the realist evaluation framework, we applied a confirmatory theory-testing case study approach. Kaplan–Meier descriptions were used to estimate the rates of dropout from the adherence club intervention and virological failure as the principal outcomes of the adherence club intervention. Qualitative interviews and non-participant observations were used to explore the context and identify the mechanisms that perpetuate the observed outcomes or behaviours of the actors. Following the retroduction logic of making inferences, we configured information obtained from quantitative and qualitative approaches using the intervention–context–actor–mechanism–outcome heuristic tool to formulate generative theories. Results: We confirmed that patients on ART in adherence clubs will continue to adhere to their medication and remain in care because their self-efficacy is improved; they are motivated or are being nudged. Conclusion: A theory-based understanding provides valuable lessons towards the adaptive implementation of the adherence club intervention.
... Realist evaluation is about theory-testing and refinement [21], whereby the evaluator assesses whether a programme is designed in such a way that it can achieve its intended outcomes and how it does so [22]. The initial programme theory guides the assessment of the effectiveness of the intervention and the consistency of the implementation [23]. ...
Article
Full-text available
Background Although empirical evidence suggests that the adherence club model is more effective in retaining people living with HIV in antiretroviral treatment care and sustaining medication adherence compared to standard clinic care, it is poorly understood exactly how and why this works. In this paper, we examined and made explicit how, why and for whom the adherence club model works at a public health facility in South Africa. Methods We applied an explanatory theory-building case study approach to examine the validity of an initial programme theory developed a priori. We collected data using a retrospective cohort quantitative design to describe the suppressive adherence and retention in care behaviours of patients on ART using Kaplan-Meier methods. In conjunction, we employed an explanatory qualitative study design using non-participant observations and realist interviews to gain insights into the important mechanisms activated by the adherence club intervention and the relevant contextual conditions that trigger the different mechanisms to cause the observed behaviours. We applied the retroduction logic to configure the intervention-context-actor-mechanism-outcome map to formulate generative theories. Results A modified programme theory involving targeted care for clinically stable adult patients (18 years+) receiving antiretroviral therapy was obtained. Targeted care involved receiving quick, uninterrupted supply of antiretroviral medication (with reduced clinic visit frequencies), health talks and counselling, immediate access to a clinician when required and guided by club rules and regulations within the context of adequate resources, and convenient (size and position) space and proper preparation by the club team. When grouped for targeted care, patients feel nudged, their self-efficacy is improved and they become motivated to adhere to their medication and remain in continuous care. Conclusion This finding has implications for understanding how, why and under what health system conditions the adherence club intervention works to improve its rollout in other contexts.
Preprint
Full-text available
Background: There is increasing recognition that prehabilitation is important as a means of preparing patients physically and psychologically for cancer treatment, however, little is understood about the role of prehabilitation for gynaecological cancer patients. Review question: This scoping review was conducted to collate the research evidence on multimodal prehabilitation in gynaecological cancers and the related barriers and facilitators to delivery and engagement that should be considered when designing a prehabilitation model for this group of women. Methods: Seven medical databases and four grey literature repositories were searched from database inception to September 2021. All articles, except for opinion papers and social media posts, surrounding multimodal prehabilitation in gynaecological cancers were included in the final review. All qualitative papers including gynaecological cancer patients were included, irrespective of whether the intervention of interest was unimodal or multimodal. A realist framework of context, mechanism and outcome was used to assist interpretation of findings. Results: In total, 24 studies were included in the final review. The studies included the following tumour groups: ovarian only (n=12), endometrial only (n=1), mixed ovarian, endometrial, vulvar (n=5) and non-specific gynaecological tumours (n=6). There was considerable variation across studies in terms of screening, delivery of prehabilitation and measured outcomes. Key mechanisms and contexts underpinning engagement with prehabilitation can be summarised by 5 overarching themes: 1) The role of healthcare professionals and organisations 2) Acceptability of prehabilitation 3) Patient motivation 4) Prioritisation of prehabilitation 5) Accessibility of prehabilitation. Implications for practice: A standardised and well evidenced prehabilitation programme for women with gynaecological cancer does not yet exist. Healthcare organisations and researchers should take into account the enablers and barriers to effective engagement by healthcare professionals and by patients, when designing and evaluating prehabilitation for gynaecological cancer patients.
Article
Full-text available
Background Audit and feedback (A&F) is a common quality improvement strategy with highly variable effects on patient care. It is unclear how A&F effectiveness can be maximised. Since the core mechanism of action of A&F depends on drawing attention to a discrepancy between actual and desired performance, we aimed to understand current and best practices in the choice of performance comparator. Methods We described current choices for performance comparators by conducting a secondary review of randomised trials of A&F interventions and identifying the associated mechanisms that might have implications for effective A&F by reviewing theories and empirical studies from a recent qualitative evidence synthesis. Results We found across 146 trials that feedback recipients’ performance was most frequently compared against the performance of others (benchmarks; 60.3%). Other comparators included recipients’ own performance over time (trends; 9.6%) and target standards (explicit targets; 11.0%), and 13% of trials used a combination of these options. In studies featuring benchmarks, 42% compared against mean performance. Eight (5.5%) trials provided a rationale for using a specific comparator. We distilled mechanisms of each comparator from 12 behavioural theories, 5 randomised trials, and 42 qualitative A&F studies. Conclusion Clinical performance comparators in published literature were poorly informed by theory and did not explicitly account for mechanisms reported in qualitative studies. Based on our review, we argue that there is considerable opportunity to improve the design of performance comparators by (1) providing tailored comparisons rather than benchmarking everyone against the mean, (2) limiting the amount of comparators being displayed while providing more comparative information upon request to balance the feedback’s credibility and actionability, (3) providing performance trends but not trends alone, and (4) encouraging feedback recipients to set personal, explicit targets guided by relevant information. Electronic supplementary material The online version of this article (10.1186/s13012-019-0887-1) contains supplementary material, which is available to authorized users.
ResearchGate has not been able to resolve any references for this publication.