No full-text available
To read the full-text of this research,
you can request a copy directly from the authors.
Unpacking Black Boxes: Mechanisms and Theory Building in Evaluation
Abstract
There is growing interest in the concept of ‘‘mechanism’’ across many areas of the social sciences. In the field of program and policy evaluation, a number of scholars have also emphasized the importance of causal mechanisms for explaining how and why programs work. However, there appears to be some ambiguity about the meaning and uses of mechanism-based thinking in both the social science and evaluation literature. In this article we attempt to clarify what is meant by mechanisms in the context of program evaluation by identifying three main characteristics of mechanisms and outlining a possible typology of mechanisms. A number of theoretical and practical implications for evaluators are also discussed, along with some precautions to consider when investigating mechanisms that might plausibly account for program outcomes.
... 201) Theory or theories in theory-based evaluations refer to the assumptions and ideas of programme designers and implementers about why programme activities they plan will lead to expected programme outcomes. Astbury and Leeuw (2010) stated that theory-based evaluations unpack programmatic 'black boxes' and explain how and why programmes work (or fail to work) in different contexts and for different programme stakeholders. A 'black box' evaluation refers to the practice or tendency of focusing only on the effects of a programme without attending to how the effects were produced. ...
... Programme evaluations in the past consisted of 'black box' evaluations, which put an emphasis on programme inputs and outputs without considering what goes on inside the programme (Pawson & Tilley, 1997). A theory-based approach to evaluation intentionally sets to make explicit the hidden and taken-for granted beliefs and practices embodied in a programme (Astbury & Leeuw, 2010). The identification and articulation of assumptions underlying a programme provides programme stakeholders with clarity on how they can align their short-and long-term goals with programme activities (Tao, 2012). ...
... According to Astbury and Leeuw (2010), a programme theory can be developed before a programme is implemented or during the course of programme implementation. Programme theory development during programme design increases stakeholders' chances of specifying intended programme outcomes, activities to be implemented and contextual factors likely to affect them. ...
... The realist approach also emphasises the importance of understanding mechanisms as emergent phenomena affected by context, as well as by interaction with other mechanisms which may enhance or cancel out each other's effects, thus causing programmes to work in different ways in different circumstances (Bhaskar, 2008;Clark et al., 2008;Pawson & Tilley, 1997;Wong et al., 2016). A critical realist theory-based evaluation therefore seeks to expose and understand a programme's inner workings, to optimise the nuanced understanding that will enable successful replication (Astbury & Leeuw, 2010;Moore et al., 2015). ...
... This attention to underlying causal processes addresses the weakness of 'black box' evaluation, which aims to establish whether a programme 'works' by assessing inputs and measuring outcomes, without generating insight into how it works, for whom or why. By contrast, theory-based evaluation is sometimes described as 'clear box' evaluation, because it focuses on precisely these questions of inner logic (Astbury & Leeuw, 2010). ...
... Although the term 'logic model' has traditionally been applied to a descriptive sequence of inputs, activities, outputs and outcomes (i.e. no causal mechanisms), in contrast to an explanatory 'theory of change' (Astbury & Leeuw, 2010), Bonell et al.'s evocative term 'dark logic' has been adopted in this thesis for those parts of the theory of change related to potential negative consequences. ...
Background
Mental health difficulties are common in the perinatal period. Peer support has been proposed as a way to improve emotional wellbeing, but the evidence base for third sector programmes is undeveloped. The overall aim of this research was to deepen understanding of what it is about perinatal mental health peer support in the third sector that works, for whom, in what circumstances, in what respects, and why.
Setting
Parents in Mind, a pilot third sector programme offering one-to-one and group perinatal peer support from trained volunteers at three sites in England.
Methods
A critical realist, mixed methods, theory-based process evaluation, supported by a realist review. An initial theory of change was used to guide the evaluation. Quantitative programme data and data from mothers’ self-report questionnaires were analysed with descriptive and inferential statistics. Qualitative interviews with 20 supported mothers, 27 volunteers, six staff and three trainers were analysed using realist principles and some techniques from Grounded Theory. Retroduction was used to explore configurations of contexts, mechanisms and outcomes and construct a final theory of change.
Results
182 mothers received peer support from 77 volunteers. Three-quarters of mothers were White British, nearly half were socio-economically disadvantaged, and three-quarters had a previous history of mental health difficulties. There were many contextual differences between the three sites, and local and national adaptations were made. The final theory of change included 16 programme theories linking contextual factors to mothers’ individual choices to use peer support, based on a mother’s beliefs about the utility of talking to others and to peers specifically; social expectations; her relationship with health and social care professionals and mental health services; perceived practical benefits; and ability to overcome barriers to access. A further 16 programme theories explained positive impact on mothers, including through feeling understood and accepted, normalisation, social comparison and information sharing; and 12 theories explained negative impact on mothers, where key peer support mechanisms were absent or through negative social comparison. Eight programme theories explained the positive impact on volunteers, based on their own experience of peer support and insight into mental health during training, gaining skills and confidence, and satisfaction at helping others; and seven theories explained the negative impact on volunteers, because of feeling emotionally ‘triggered’, stressful social dynamics between volunteers, and distress if they did not feel their support was helping mothers. Individual mothers and volunteers were affected in different ways, depending on their individual backgrounds, personalities, social situations, resources, experiences, beliefs, and needs. Parents in Mind had robust processes to keep volunteers and mothers emotionally safe, and all participants considered the benefits of peer support to greatly outweigh the risks.
Conclusions: The Parents in Mind peer support model is safe and capable of enabling positive change for both mothers and volunteers through multiple contextualised pathways. Outcome measures should recognise that mothers (and volunteers) benefit from peer support in a range of ways, and programmes should be aware of the potential for negative effects in order to mitigate these. Programmes should work with local communities to understand what they want so that the peer support offer is adapted flexibly to their needs.
... We do not aim at providing an overview of the use of the mechanism concept in the TBE literature. Astbury and Leeuw (2010), Dalkin et al. (2015), and, more recently, Schmitt (2020), and Lemire et al. (2020) provide a detailed overview of this kind, showing the widespread use and the varieties of conceptualizations of the mechanisms in TBE. These overviews identify a number of recurrent themes concerning how mechanisms are conceptualized in the context of TBE. ...
... These overviews identify a number of recurrent themes concerning how mechanisms are conceptualized in the context of TBE. For instance, Astbury and Leeuw (2010) identify the terms "hidden," "sensitive to variations in context," and "generate outcomes" as recurring ways of characterizing mechanisms. Schmitt (2020) provides a taxonomy of two types of mechanism concepts used in the evaluation literature: behavioral mechanisms (describing changes in individual reasoning and decisions that mediate behavioral effects) and process mechanisms (describing how different activities involved in an intervention or program are linked together). ...
... Therefore, we might say that the counterfactual relation is in itself rather thin. Take, for instance, Astbury and Leeuw (2010): ...
In this article, we discuss the methodological implications of data and theory integration for Theory-Based Evaluation (TBE). TBE is a family of approaches to program evaluation that use program theories as instruments to answer questions about whether, how, and why a program works. Some of the groundwork about TBE has expressed the idea that a proper program theory should specify the intervening mechanisms underlying the program outcome. In the present article, we discuss in what way data and theory integration can help evaluators in constructing and refining mechanistic program theories. The paper argues that a mechanism is both a network of entities and activities and a network of counterfactual relations. Furthermore, we argue that although data integration typically provides information about different parts of a program, it is the integration of theory that provides the most important mechanistic insights.
... Effective use of community-based and hybrid models for PrEP service delivery to AGYW will also require a better understanding of how these approaches may work to achieve higher PrEP persistence and why they may work for some AGYW and not others. Inquiry into the mechanisms, or the ways in which any single or combination of service delivery components brings about change, may help answer these questions (20,21). Better understanding of these mechanisms or "essential ingredients" can offer important insights to implementers and inform existing and future service delivery models and interventions to support improved PrEP persistence among AGYW. ...
... Transcripts were imported into ATLAS.ti (version 8) and analyzed using content analysis with higher-level abstraction and interpretation informed by theory-based evaluation concepts including mechanisms of change (20,21,(32)(33)(34). The concept of mechanisms grew out of the realist evaluation approach which emphasizes understanding how interventions work in real world settings. ...
... The concept of mechanisms grew out of the realist evaluation approach which emphasizes understanding how interventions work in real world settings. Applied to models of PrEP service delivery, mechanisms are not the components or activities of service delivery itself, but rather the ways these influence the reasoning or response of AGYW, altering their behavior and contributing to a model's success or failure to improve PrEP persistence (20,32,34,35). After reading through a subset of transcripts, two experienced qualitative researchers developed an initial code book used for both AGYW and provider IDIs to better identify and relate themes across datasets. ...
Introduction
Despite the potential for community-based approaches to increase access to pre-exposure prophylaxis (PrEP) for adolescent girls and young women (AGYW), there is limited evidence of whether and how they improve PrEP persistence. We compared PrEP persistence among AGYW receiving services through community and hybrid models in Namibia to facility-based services. We subsequently identify potential mechanisms to explain how and why community and hybrid models achieved (or not) improved persistence to inform further service delivery innovation.
Methods
Data were collected from PrEP service delivery to AGYW over two-years in Namibia's Khomas Region. We used Kaplan-Meier analysis to estimate survival curves for PrEP persistence beyond three-months after initiation and report the cumulative probability of persistence at one- and three-months. Persistence was defined as any PrEP use within three months after initiation followed by a PrEP refill or previously prescribed supply of at least 30 days at the three-month visit. Interviews were conducted with 28 AGYW and 19 providers and analyzed using a deductive-inductive thematic approach.
Results
From October 2017 through September 2019, 372 (18.7%) AGYW received services through a facility model, 302 (15.1%) through a community model, and 1,320 (66.2%) through a hybrid model. PrEP persistence at one- and three-months was 41.2% and 34.9% in the community model and 6.2% and 4.8% in the hybrid model compared to 36.8% and 26.7% in the facility model. Within the community and hybrid models, we identified three potential mechanisms related to PrEP persistence. Individualized service delivery offered convenience and simplicity which enabled AGYW to overcome barriers to obtaining refills but did not work as well for highly mobile AGYW. Consistent interactions and shared experiences fostered social connectedness with providers and with peers , building social networks and support systems for PrEP use. PrEP and HIV-related stigma, however, was widely experienced outside of these networks. Community-to-facility referral for PrEP refill triggered apprehension towards unfamiliar PrEP services and providers in AGYW, which discouraged persistence.
Conclusion
Service delivery approaches that offer convenience and simplicity and foster social connectedness may reduce access barriers and increase social support enabling AGYW to self-manage their PrEP use and achieve improved PrEP persistence.
... Finally, given its ability to depict the causal process through which an intervention is meant to work and what change it is expected to produce, ToC can also inform monitoring and evaluation processes. In depicting the key steps and expected deliverables, it facilitates the identification and development of key indicators [9,16,17]. ...
... None of the publications provided a list of excluded studies with justifications [14, 19, 21-23, 39, 41]. In addition, the five critically low did not account for biases [10,12,13,17,19,[21][22][23], or did not refer to methods being established prior to the review [39]. Six studies were assessed with the JBI Critical Appraisal Checklist for Qualitative Research [31]. ...
Background
Theory of Change (ToC) has become an established approach to design and evaluate interventions. While ToC should—in line with the growing international focus on evidence-informed health decision-making–consider explicit approaches to incorporate evidence, there is limited guidance on how this should be done. This rapid review aims to identify and synthesize the available literature on how to systematically use research evidence when developing or adapting ToCs in the health sector.
Methods
A rapid review methodology using a systematic approach, was designed. Eight electronic databases were consulted to search for peer-reviewed and gray publications detailing tools, methods, and recommendations promoting the systematic integration of research evidence in ToCs. The included studies were compared, and the findings summarized qualitatively into themes to identify key principles, stages, and procedures, guiding the systematic integration of research evidence when developing or revising a ToC.
Results
This review included 18 studies. The main sources from which evidence was retrieved in the ToC development process were institutional data, literature searches, and stakeholder consultation. There was a variety of ways of finding and using evidence in ToC. Firstly, the review provided an overview of existing definitions of ToC, methods applied in ToC development and the related ToC stages. Secondly, a typology of 7 stages relevant for evidence integration into ToCs was developed, outlining the types of evidence and research methods the included studies applied for each of the proposed stages.
Conclusion
This rapid review adds to the existing literature in two ways. First, it provides an up-to-date and comprehensive review of the existing methods for incorporating evidence into ToC development in the health sector. Second, it offers a new typology guiding any future endeavors of incorporating evidence into ToCs.
... RS er efter vores vurdering velegnet til at systematisere og teste den eksisterende viden om, hvad der virker, for hvem og under hvilke omstaendigheder. RS bygger på antagelsen om, at der kan findes en raekke kausale sammenhaenge iboende i indsatser, når man åbner indsatsers såkaldte "sorte bokse" (Astbury & Leeuw, 2010). Tilgangen tilbyder en analytisk ramme, der gør det muligt at sammenfatte viden og evidens om de virksomme elementer, der bidrager til indsatsers resultater. ...
... Mekanismer kan både omhandle individuelle aendringer hos deltagerne og/eller sociale aendringer mellem deltagerne (Bonell et al., 2022). Mekanismer antages at vaere virkelige, men de vil ofte vaere "skjulte" og befinde sig på et dybere niveau, end hvad der umiddelbart kan observeres (Westhorp, 2018;Astbury & Leeuw, 2010). Et eksempel er personers laering. ...
Vi mangler gode analysetilgange til at undersøge hvorfor og hvordan komplekse indsatser virker. Traditionelle ”systematiske reviews” som anvendes på sundhedsområdet er gode til at undersøge hvorvidt, men ikke hvorfor og hvordan indsatser virker. På de komplekse velfærds- og socialområder er der ikke bare brug for viden om hvorvidt indsatser virker, men også hvorfor og under hvilke omstændigheder virkningerne indtræffer. Den ”realistiske syntese” har potentiale til at udgøre et brugbart supplement. Der eksisterer imidlertid ikke en klar fremgangsmåde til at gennemføre realistiske synteser. Tilgangen kan derfor være udfordrende og tidskrævende for evaluator, ligesom dens resultater kan være ugennemsigtig for læseren og i værste fald en delvis eller misvisende repræsentation af den eksisterende viden på området. I denne artikel reflekterer vi over de metodiske udfordringer i realistisk syntese og diskuterer potentielle løsninger herpå. Vi diskuterer fremgangsmåder til udvikling af foreløbige programteorier, dataindsamling, kodning og syntese. I den forbindelse opstiller vi en genbeskrivende analysestrategi, der kan guide evaluator og sikre gennemsigtighed i analyse- og syntesefasen.
... Recently there has been a call for realist evaluation to be used to advance understanding of treatment for addictions (Urbanoski 2020). RE aims to make explicit the theory underpinning an intervention (Astbury and Leeuw 2010) by identifying causal mechanisms that explain how outcomes are achieved from interventions (Marchal et al. 2012;Pawson and Tilley 1997). In doing so, RE aims seeks to answer: what works, for whom, in what circumstances and over what timescale (Astbury and Leeuw 2010;Pawson 2013;Pawson and Tilley 1997;Weiss 1997). ...
... RE aims to make explicit the theory underpinning an intervention (Astbury and Leeuw 2010) by identifying causal mechanisms that explain how outcomes are achieved from interventions (Marchal et al. 2012;Pawson and Tilley 1997). In doing so, RE aims seeks to answer: what works, for whom, in what circumstances and over what timescale (Astbury and Leeuw 2010;Pawson 2013;Pawson and Tilley 1997;Weiss 1997). RE differs from other theory-based evaluation approaches (for example Theory of Change) in its focus on generative causation (Bhaskar 2008) and search for the configurations of context-mechanismoutcome to explain how programmes outcomes are caused (Blamey and Mackenzie 2007). ...
Residential treatment of addiction remains a key element in addictions management, though high rates of attrition and relapse are reported, and underlying mechanisms of change remain unclear. This study aimed to complete the first stage of a realist evaluation—to develop an initial programme theory that would explain possible contexts and mechanisms of change leading to outcomes from a residential treatment programme for substance use disorders. Realist evaluation using key informant interviews and document review. Residential treatment centre in New Zealand. Four key informants (managers and caseworkers). Retroductive analysis (inductive and deductive) of semi-structured interviews and content analysis of documents. We identified one intermediate outcome of identity change that contributed to successful completion of residential treatment. The mechanisms leading to these outcomes were the development of a sense of belonging to the recovery community and a sense of hope and purpose. These mechanisms were activated in contexts including supportive and structured culture (staff and peers) and the absence of chaos. We developed an initial programme theory to explain successful completion of a residential treatment for SUD. Theories like this are important because practitioners can use them to increase the likelihood of replication of successful treatment. We found that realist evaluation provides researchers with appropriate philosophy and methods with which to explore and start to unravel some of the complexity within residential treatment for addictions.
... This is an important concern. Using models to assist with extrapolation is becoming more widespread, such as in the realist evaluation literature (Pawson, 2006(Pawson, , 2013Astbury & Leeuw, 2010), which puts emphasis on modelling the processes and mechanisms by which an intervention is supposed to work. But even when models are used, important concerns remain about their quality. ...
Extrapolating causal effects from experiments to novel populations is a common practice in evidence-based-policy, development economics and other social science areas. Drawing on experimental evidence of policy effectiveness, analysts aim to predict the effects of policies in new populations, which might differ importantly from experimental populations. Existing approaches made progress in articulating the sorts of similarities one needs to assume to enable such inferences. It is also recognized, however, that many of these assumptions will remain surrounded by significant uncertainty in practice. Unfortunately, the existing literature says little on how analysts may articulate and manage these uncertainties. This paper aims to make progress on these issues. First, it considers several existing ideas that bear on issues of uncertainty, elaborates the challenges they face, and extracts some useful rationales. Second, it outlines a novel approach, called the support graph approach , that builds on these rationales and allows analysts to articulate and manage uncertainty in extrapolation in a systematic and unified way.
... Moreover, theory-based evaluation that enables an eclectic and modular approach is particularly useful to large evaluation teams which use a broad array of different evaluation and data collection methods (Hernandez & Hodges, 2003;Nesman et al., 2007;Stern et al., 2012). Drawing on Astbury and Leeuw (2010), we chose a combination of a conceptual component in which the programme (impact) theory was developed, and an empirical component in which it was tested and refined. Section two will describe the different steps in building the ToC as well as the different evaluation methods and data collection instruments that we used to test the different components. ...
To understand the impact of the international Master’s programmes offered at the Institute of Development Policy (University of Antwerp), a theory-based evaluation was undertaken. In the first phase, a Theory of Change (ToC) was elaborated, distinguishing between three levels of impact (individual, organisational and societal), four learning dimensions (knowledge, skills, attitudes and networks) and five implicit pathways (change agent, social network, widening access, academic diversity, international understanding). Given the multifaceted and vague nature of the ‘impact’ concept under study, we selected an international, gender-balanced, multi-sectoral team of alumni researchers who fostered inclusiveness of different perspectives, at the same time capitalising on their depth of understanding, having gone through the study experience themselves. A mixed-methods approach was adopted to validate the ToC, combining a Most Significant Change approach, categorisation and text analysis of 101 alumni impact stories. Our findings demonstrate the importance of the three levels of impact and four learning dimensions in capturing graduate impact. While the impact stories confirmed the dominant ‘change agent’ pathway, they also hinted at the importance of hybrid complementary configurations of pathways to fully grasp how impact materialises.
... Mechanisms are often 'hidden' features 13 but can produce outcomes when influenced by variations in context. 14 A realist approach is grounded in realism and supports an approach to 'untangling the complexity of real-life implementation' 15 of programmes. The methodological guidelines for theory-driven realist synthesis studies, RAMESES protocol, 16 and the Key Steps in Realist Review, 11 were used to guide the approach in this study. ...
Mental health services continues to be a high priority for healthcare and social service systems. Funding structures within community mental health settings have shown to impact service providers’ behaviour and practices. Additionally, stakeholder engagement is suggested as an important mechanism to achieving the intended goals. However, the literature on community mental health funding reform and associated outcomes is inconsistent and there are no consistent best practices for stakeholder engagement in such efforts.
Objectives
This study sought to understand how stakeholder engagement impacts outcomes when there is a change in public funding within community mental health settings.
Design
A realist synthesis approach was used to address the research question to fully understand the role of stakeholder engagement as a mechanism in achieving outcomes (system and service user) in the context of community mental health service reform. An iterative process was used to identify programme theories and context–mechanism–outcome configurations within the literature.
Results
Findings highlight that in the absence of stakeholder engagement, funding changes may lead to negative outcomes. When stakeholders were engaged in some form, funding changes were more often associated with positive outcomes. Stakeholder engagement is multifaceted and requires considerable time and investment to support achieving intended outcomes when funding changes are implemented.
Conclusions
To support successful transformation of community mental health programmes, it is important that stakeholders are meaningfully engaged during funding allocation changes. Stakeholder engagement may entail connecting around a shared purpose, individual participation and meaningful interactions and dialogue.
... We began by hypothesizing about potential mechanisms. Astbury and Leeuw reported that a clear distinction should be made between mechanisms and program interventions [29]. For example, an outcome might be an increase in the learners' knowledge or readiness resulting from some mechanism arising from some educational intervention. ...
Background
The need to engage medical students in understanding the social and environmental determinants of health in disparate communities is increasing. However, previous reviews have noted the limited community diagnosis programs and program evaluation. Given the feasibility of the programs, it is expected to be widely available online. Therefore, this study used a realist approach to identify learning patterns through an online community diagnosis program, namely context (C), mechanism (M), and outcomes (O) patterns.
Methods
A 2-week general medicine clinical practice program was conducted for 4th- and 5th-year medical students at a medical university in Japan. The program included a one-hour zoom-based lecture, feedback for students on their presentations on community diagnosis, and a structural report on community diagnosis. We developed the program based on variation theory, which views discernment and variation in situations having time, space, and social dimensions as core learning. The students' reflections on their learning through the program were thematically analyzed through CMO perspectives. The realist approach used in the online diagnosis program evaluation allows us to explore, test, and refine what mechanisms work under what conditions (context) and with what interventions (including opportunities and resources), from which we can describe iteratively explainable results.
Results
First, the medical students, who spent most of their time in the limited residential areas they lived in, discovered the characteristics of their own community by discovery learning and comparison among peers. Second, they increased their intrinsic interest in the community by discerning specific issues in their familiar community through community diagnosis. Third, they valued community diagnosis by identifying relationships between local data on health issues under their learning responsibility. Fourth, they become more flexible in their thinking and created new knowledge that would fit the local community, and their reflection on themselves was encouraged.
Conclusion
In this online community diagnosis program, medical students learned about the community through four types of learning patterns. Medical students may develop an understanding of community with interest using variation theory as a program development perspective and cognitive flexibility theory surrounding the essential ambiguity and abstraction of community.
... As such, it is not enough for a program theory to describe activities that are expected to lead to outcomes. Instead, the theory must link these activities to outcomes through their causal processes (Astbury & Leeuw, 2000). Mechanisms can be understood as mediating and moderating processes (Chen, 2005) or ideas and opportunities (Pawson & Tilley, 1997) which, if and when appropriately introduced to individuals and groups in the appropriate social and contextual conditions, lead to successful interventions. ...
Program theory reconstruction is an often-overlooked aspect of social intervention research. In this paper, we argue that intervention research benefits if the research design is informed by the specific intervention's program theory (i.e., the idea of how the intervention is supposed to lead to the intended outcomes). The purpose of this paper is to offer a comprehensive and accessible guide to program theory reconstruction in research on social interventions and to provide arguments as to how program theory reconstruction can be used to benefit intervention studies. First, we summarize what program theory is and its role in intervention research. Second, we provide a direct “how-to” for researchers, practitioners, and students who may be unfamiliar with the methods of program theory reconstruction but are interested in undertaking a program theory reconstruction. Finally, we conclude with how program theory reconstruction can benefit intervention research.
... Rather unsurprisingly and in line with an earlier more limited review by Manzano (2016), we find that qualitative methods are most commonly used. This predominance of qualitative research methods reflects what is common in the social sciences and the fact that within realist evaluation mechanisms are often hidden and difficult to measure and thus not easily captured with more quantitative approaches that lack interpretative depth (Astbury & Leeuw, 2010;Bonell, Warren, & Melendez-Torres, 2022;Sayer, 1992). Therefore, the utility of quantitative methods are still debated in realism (Sayer, 2000), especially when they are used within an randomized controlled trial design (Marchal, et al., 2013;Van Belle, et al., 2016). ...
Realist evaluation is becoming increasingly popular as an evaluation methodology. Its main objective is to uncover the mechanisms that lead to observed outcomes following an intervention and the contextual conditions that enabled this. The focus is on explaining why, for whom and in what circumstances an intervention works. It is a theory-driven approach and is explicitly method neutral, meaning that both quantitative and qualitative data collection methods can be used to unearth the underlying mechanisms that cause the intervention outcomes. In this review, we aim to map the methods used in realist evaluation studies, to draw lessons from the findings and to reflect on ways forward. We found that qualitative methods and interviews specifically are most commonly used in realist evaluations; that theory is often absent behind the methods and sampling techniques used; and that more innovative methods remain underexplored. We conclude the review by proposing four ways forward: (1) developing realist surveys, (2) exploring the relevance of innovative methods, (3) increasing the attention paid to sampling procedures and (4) strengthening the theory-driven nature of method. We believe that these four action points can strengthen the practice of realist evaluation and its outcomes.
... The real domain contains entities which are independent of reality, with their causal power being activated in a particular context. The analysis of causation enables the investigation of the generation of events and regularities in these three domains of reality (Astbury and Leeuw, 2010). The researcher viewed the phenomenon of value platform evolution through a critical realist ontological stance, which seeks to uncover the mechanisms that underlie resource-related activities in ecosystems to develop service innovation over time. ...
Advanced technologies assist diverse entities in becoming network actors, exchanging resources and co-creating value together to achieve service innovation. However, tensions emerge when multiple actors have different goals and expectations during the service innovation process. This thesis extends the service ecosystems literature by incorporating the evolution of value platforms in the service innovation process over time. The notion of value platforms facilitates our understanding of the dynamic interactions among actors to co-create value for the development of service innovation. The theoretical lens of institutional logics was applied in this study to explore the dynamic resource-related activities that occur as value platforms evolve. This thesis explores the evolution of value platforms embedded in service ecosystems during the service innovation process. It investigates how the resource-related activities evolve in service ecosystems throughout the process of service innovation and seeks to unravel the mechanism of actor interaction in platform-based service innovation. In particular, the study investigates how value platforms embedded in service ecosystems evolve, what tensions arise throughout the evolution due to the multiple institutional logics of the actors within the ecosystem, and how multiple institutional logics are navigated as value platforms evolve. A critical realist approach is adopted to explore the phenomenon of value platform evolution. A process-based single-case study design with two embedded cases is implemented to investigate value platforms embedded in service ecosystems to develop telematics insurance services. The researcher conducted a two-phased data collection to gather semi-structured interviews and participant-generated drawings as primary data from different actors along with archival documents as the secondary data. A realist evaluation enabled the delineation of the five stages that form the building blocks of the evolution of the value platforms. Moreover, an abductive approach identified three types of process-related tensions and three types of navigating mechanisms that emerge dynamically as value platforms evolve. This research offers theoretical contributions to a processual understanding of value co-creation in service ecosystems by explaining the evolution of tensions resulting from co-existing institutional logics and navigating mechanisms inherent in value platforms. It also highlights how regulatory actors affect service ecosystems during the process of service innovation. Furthermore, the study offers practitioners a processual understanding of tensions that occur in the service innovation process, and the approaches to navigating those tensions in service ecosystems during the service innovation process.
... Realist synthesis seeks to 'uncover the underlying theories that explain these demi-regularities by critically scrutinising the interaction between context, mechanism and outcome in a sample of primary studies' [17], which are commonly expressed as 'context-mechanism-outcome configurations' ('CMOCs'). Mechanisms, defined as 'underlying entities, processes, or structures which operate in particular contexts to generate outcomes of interest' [18], are a defining feature of realist research. They help us understand that it is not the intervention itself which produces outcomes but people's reactions, reasoning and responses to it that are important. ...
Background
Good interpersonal communication is the cornerstone of social work practice, enhancing the outcomes of people in receipt of its services. Social workers’ communication skills are often identified as an area of concern. Communication skills can be developed and refined through training or education. The teaching and learning of communication skills is firmly embedded in many social work qualifying courses; however, considerable heterogeneity exists regarding such complex interventions and the theoretical underpinnings of which have not been made explicit. Realist synthesis can help explain how, why, for whom and in what circumstances an intervention might work, which is an important first step for helping educators to tailor courses to meet the needs of different learner groups and, where applicable, the employing agencies and government departments who fund them.
Methods
Realist synthesis is an interpretive, theory-driven and explanatory approach that aims to explain the interplay between the context, mechanisms and outcomes of interventions. This realist synthesis seeks to understand and explain to what extent, how, why, for whom and in what circumstances complex educational interventions aimed at teaching communication skills to social work students produces its effects. A five-step process will be followed iteratively. In step 1, the initial programme theory will be developed. Step 2 will involve searching for evidence. In step 3, selection and appraisal will take place. Step 4 requires data to be extracted and organised, and in step 5, data will be analysed and synthesised.
Discussion
The teaching and learning of communication skills in social work education is under theorised. The findings from this realist synthesis aim to help policymakers and educators make informed decisions about the design and delivery of complex educational interventions aimed at improving the communication skills of social work students. The realist synthesis will be conducted and reported in accordance with the RAMESES guidelines and standards.
Systematic review registration
The review is registered with the Open Science Framework. https://doi.org/10.17605/OSF.IO/BYHC7
... Realist evaluation is about testing and refining a theory through iterations between theorization and empirical observations using both inductive and deductive reasoning (Astbury and Leeuw, 2010;Pawson and Tilley, 2004). Four key concepts are used in realist evaluation and hence in realist synthesis: 'mechanism', 'context', 'outcome', and Context-Mechanism-Outcome configuration (C-M-O). ...
Payments for environmental services (PES) have been widely studied as one possible way to counter deforestation and support livelihoods. Given the high rates of deforestation in the Amazon region, it is important to study under which conditions PES have positive environmental and social outcomes. We contribute to this debate through a systematic review based on the Realist Evaluation framework. First, we review case studies to identify configurations of social-ecological factors (context), PES design and implementation (resource mechanisms), people's responses (reasoning mechanisms), and the results in terms of forest conservation and people's livelihoods (outcomes). Second, we develop a middle-range theory of how, for whom, and under what conditions PES in the Amazon achieve these results. After screening 972 articles in the SCOPUS, Scielo, and WorldCat databases, we reviewed 13 articles in-depth that contained all the elements (context-mechanism-outcomes) needed to identify these configurations. Our results show that PES in the Amazon generate positive environmental and social outcomes when they: (a) combine cash and in-kind incentives, (b) raise environmental awareness through capacity building, (c) engage socially and ethnically diverse stakeholders through equitable and inclusive approaches, (d) apply transparent spatial targeting, (e) guarantee strong conditionality through robust monitoring of compliance, and (f) guarantee stable delivery of payments.
... Innovation policy evaluation has been dominated by evaluation approaches that try to measure the effects of policy intervention by relying on quantitative indicators to assess programme performance and effects, for example to determine accountability (Molas-Gallart and Davies, 2006). Such approaches, which are usually referred to as "black box evaluations" (Astbury and Leeuw, 2010), are experiment-focused and lack the power to explain how and why outcomes come about (Blamey and Mackenzie, 2007;Rolfe, 2019). Theory-based evaluation emerged as an alternative to such methods-based evaluation, focusing on unpacking the processes between policy intent and policy outcome by revealing causal mechanisms that generate the outcomes as well as the contextual factors that influence them (Rolfe, 2019). ...
In recent years, a new frame for innovation policy has emerged, namely “transformative innovation policy” (TIP), which aims at addressing transformative change or “Grand Challenges”. Such a shift in policy theory should, ideally, be reflected in policy evaluation, but the literature has so far provided little advice on how to address TIP-related evaluation challenges such as directionality and system-level behavioural additionality. This paper discusses how the evaluation of policy interventions targeting system innovation can be designed to address these challenges. Combining the literature on sustainability transitions with policy evaluation, we propose an integrated evaluation framework composed of three main components: (i) programme theory (programme goals, systems boundaries and desired (or accepted) development paths); (ii) system analysis (transformative outcomes); and (iii) synthesis and overall assessment (including revision of programme theory). By integrating the two sets of literature, we provide a bridge between academic research on transitions and current evaluation practices. We briefly illustrate the applicability of the integrated framework in the BioInnovation Strategic Innovation Programme in Sweden.
... Men kontekstualisme og funktionalisme underminerer ikke teoretisk udvikling. En teoretisk base omkring typologiske "familier" af interventionsaspekter kan opbygges (Pawson & Manzano-Santaella, 2012), hvilket over tid reducerer nødvendigheden af teoretisk nyskabelse (Astbury & Leeuw, 2010). Pragmatismen henviser her til, at forskningen i meditation på MBI bør have et anvendelsesorienteret sigte. ...
Meditation is increasingly applied in health promotion and clinical interventions although the influence of meditation during multimodal interventions is unclear. Even among meditation-based interventions (MBI), theories and recommendations concerning meditation vary substantially. Our primary purpose is to discuss from an empirical base if MBI-participants’ amount of meditation practice is a substantial factor for the treatment effects. We investigate this especially by reviewing MBI-research on dose-effect relationships between meditation and outcome changes, and neuroimaging studies of meditators and MBI-participants. The clinical evidence shows that participants’ amount of meditation practice is not consistently related to treatment effects. Neuroscientific studies support this notion, since previous meditation practice has not been consistently or specifically related to structural or functional neural measures for MBI-participants or experienced meditators. The inconsistent dose-effect findings, however, do not lead us to discard meditation as a potentially important aspect of psychosocial interventions. Previous research is methodologically criticizable due to a lack of active control groups, studies investigating the importance of meditation as a function of participant characteristics, and qualitative studies of the experienced significance or quality of the meditation practices. In addition, meta-syntheses of qualitative studies of processes of change during MBI do conclude that meditation is experienced as effective, but in complex, systemic interactions with other treatment aspects. More complex models concerning the significance of meditation practices for different participants and clinical areas are thus needed. We present a contextual, functional and pragmatic approach to the application of meditation and provide recommendations for future research and practice.
... The "C" dimension comprises the setting or social and livelihood system for tree collateral approach implementation as well as the regulatory environment relevant to tree collateral mechanisms (Astbury and Leeuw, 2010). ...
Small-scale forestry requires an integrated environmental rural-development approach. However, individual small-scale forest producers face various constraints, especially regarding finances, as long rotation periods clash with the more immediate financial needs. The "tree-for-loan collateral approach" offers a potential remedy for this financial dilemma wherein trees-or tree-derived products or services-serve as part of the security necessary for loans or other transactions in both formal and informal contexts. This approach can ease access to finance for land users that lack conventional collateral, but it requires innovative lending and valuation procedures. Despite the initial and ongoing dissemination of tree collateral, deliberations in academia and practice are diverse, and relevant knowledge remains largely fragmented. This conceptual and terminological diversity further underlines the necessity of global synthesis and analysis. We conducted a realist synthesis review of literature databases and organizational websites; we also sought guidance through key informant interviews to assess the context, mechanisms, and outcomes of tree collaterals described in the literature. The analyzed mechanisms range from informal or customary approaches to the formal approaches offered by development or commercial banks. Key factors that hinder up-scaling can be overcome through the social aggregation of producers , access to processing and established market outlets, and clear regulations for forest asset collateraliza-tion. The typology classification of the reviewed mechanisms reveals a plethora of innovative intricacies and creditor-debtor constellations, including NGOs and other third parties that secure the transactions. Future economic research on tree collateral should address key parameters and assess the impact and potential conceptual optimization of easing access to finance. We also recommend further research beyond economics to examine the social implications and effects of the tree collateral mechanism as a potential tool for facilitating access to forest resources worldwide, including the financial and technical means for access.
... According to Astbury et al, '"Mechanism is the hidden entities, processes or structures which operate in particular contexts to generate outcomes of interest'. 32 Mechanism can also be defined as the way the programme's resources or opportunities interact with the reasoning of individuals and lead to changes in behaviour. 27 In terms of RE application, findings can be used in making decisions about programmes, using the outcomes to influence how programme and its effect are perceived or using the outcomes to justify decisions about the programme. ...
Introduction
Key populations (KP) living with HIV are underserved and often face social and health system barriers to HIV care. To optimise access to quality HIV services among KP, the WHO recommended community-based approaches to HIV service delivery for KP. However, to inform the successful rollout and scale-up of community-based antiretroviral therapy service delivery models for KP (KP-CBART), there is a need to study the programme implementation. This study aims to evaluate the outcomes of KP-CBART in Benue State Nigeria using a realist impact evaluation approach. Our evaluation question is: what are the mechanisms and context conditions that drive successful community-based implementation and how do these lead to better retention in care, treatment adherence and viral suppression among which categories of KP?
Methods and analysis
This study will be conducted in three phases, relying on a mixed-method design and following the realist evaluation cycle. The first phase is the development of the initial programme theory grounded in a scoping review, programme and policy document review and in-depth interviews with key stakeholders. In phase 2, findings from case studies of KP-CBART programme implementation in one Nigerian state are used to test the initial programme theory and to refine it. The quantitative part is a retrospective cohort study. All HIV-positive KP clients enrolled into the KP-CBART between 2016 and 2020 will be included in the study. While maximum variation and data saturation will inform sample size for the qualitative part, an estimated 90 purposively selected study participants will be interviewed. In phase 3, findings will be synthesised into a middle-range theory through cross-case analysis. The heuristic intervention, context, agents, mechanisms and outcomes (ICAMO) tool will be used to refine the initial programme theory.
Ethics and dissemination
The study protocol was approved by the Institutional Review Boards of APIN Public Health Initiatives (IRB022-FR), Institute of Tropical Medicine Antwerp (1503/21), and the Benue State Ministry of Health and Human Services (MOH/STA/204/VOL1/154). Written informed consent will be obtained from all study participants. Study results will be disseminated through stakeholders meeting, peer-reviewed journals and conferences.
... However, with the increased interest in causal mechanisms, the way they are defined and analysed in evaluation literature and practice has become more diverse. Most of the scholars follow the abovementioned realist accounts of causation in their conceptualisation of mechanisms (Astbury & Leeuw, 2010). Still, some take the minimalist view of a mechanism. ...
Objectives: The aim of the article is to consider the evaluation of public interventions through the prism of evidence-based policy (EBP) as well as, more specifically, its potential to address the problem of how to produce in the process of impact evaluation usable knowledge that can help improve policymaking and policy implementation which can be accumulated over time, where evaluations will not be single endeavours and one-off studies, but will contribute to the growing body of knowledge. Research Design & Methods: The article provides a critical overview of the research literature on evaluation approaches and methods as tools for gathering and apprising evidence relevant for policymaking and policy implementation. Findings: Building upon the identified limitations of the traditional input/output approach to impact evaluation of public interventions, alternative approaches to evaluation are considered that make use of a theory that properly explicates the causal mechanisms linking programme activities with programme outcomes, confronting the mechanisms with empirical observations. As a strategy for synthesising the gained knowledge, the realist synthesis is considered as being more appropriate for reviewing research on complex social interventions (rather than traditional meta-analysis). Implications / Recommendations: The article demonstrates how theory-based evaluation with mechanistic explanation and realist synthesis can contribute to growing evidence for policy needs, identifying also their limitations and practical problems related to their implementation. Contribution / Value Added: The article contributes to the existing pool of knowledge by providing important insights into how to use mechanism-based explanations in impact evaluation to make stronger causal claims and enhance policy-learning.
... Despite growing interest in unpacking the black box of policy implementation, there is scarce research on the social complexity in which implementation is embedded. 1 Tobacco use and its control are complex social phenomena due to the diversity of actors, competing interests, and the politics and economics of sales versus the health costs. 2 The burden of tobacco use and tobaccorelated deaths is disproportionately high in low-and middle-income countries (LMICs). Around 77% of smoking-related deaths and 89% of secondhand smoke-related deaths occur in LMICs. ...
The burden of tobacco use is disproportionately high in low- and middle-income countries (LMICs). There is scarce theorisation on what works with respect to implementation of tobacco control policies in these settings. Given the complex nature of tobacco control policy implementation, diversity in outcomes of widely implemented policies and the defining role of the context, we conducted a realist synthesis to examine tobacco control policy implementation in LMICs. We conducted a systematic realist literature review to test an initial programme theory developed by the research team. We searched EBSCOHost and Web of Science, containing 19 databases. We included studies on implementation of government tobacco control policies in LMICs. We included 47 studies that described several contextual factors, mechanisms and outcomes related to implementing tobacco control policies to varying depth. Our initial programme theory identified three overarching strategies: awareness, enforcement, and review systems involved in implementation. The refined programme theory identifies the plausible mechanisms through which these strategies could work. We found 30 mechanisms that could lead to varying implementation outcomes including normalisation of smoking in public places, stigmatisation of the smoker, citizen participation in the programme, fear of public opposition, feeling of kinship among violators and the rest of the community, empowerment of authorised officials, friction among different agencies, group identity among staff, shared learning, manipulation, intimidation and feeling left out in the policy-making process. The synthesis provides an overview of the interplay of several contextual factors and mechanisms leading to varied implementation outcomes in LMICs. Decision-makers and other actors may benefit from examining the role of one or more of these mechanisms in their particular contexts to improve programme implementation. Further research into specific tobacco control policies and testing particular mechanisms will help deepen our understanding of tobacco control implementation in LMICs.
PROSPERO registration number: CRD42020191541.
... It offered a particular understanding of causation; that phenomena we can experience or observe are generated by events connected at a (usually) non-observable level (Greenhalgh et al. 2017a). At this deeper level exist mechanisms, 'underlying entities, processes, or [social] structures which operate in particular contexts to generate outcomes of interest' (Astbury and Leeuw, 2010). Realist inquiry, by identifying these context-mechanism-outcome (CMO) configurations seeks to develop theory about 'what works for whom in what circumstances and in what respects, and how' (Pawson and Tilley 2004, p. 2). ...
There is currently a paucity of evidence regarding how medical undergraduate training influences students’ attitudes and skills for lifelong learning (LL). This study describes medical graduates’ conceptualisations of LL at the University of Dundee and proposes a provisional theory regarding related curriculum influences. Using a methodology informed by scientific realism, semi-structured interviews were employed to explore six graduating medical students’ conceptualisations of and preparedness for LL. Findings were thematically analysed and aligned with existing educational theories. Graduates characterised LL as a continual process which emphases reflection, responsibility for learning, and maintaining motivation. Their conceptualisations did not translate readily beyond professional medical practice. The strongest curriculum influences were experiences in the authentic clinical environment, use of learning portfolios, and the intercalation of a medical science degree. In the absence of formal teaching, medical students’ conceptualisations of LL develop in accordance with both the implicit prevailing professional narrative and from curriculum interventions which: increase their sense of relatedness with their community of practice; provide autonomy in learning; and engender a sense of responsibility to learn. This provisional theory could be valuable in guiding the teaching and assessment of LL in-line with contemporary graduate outcome frameworks.
... The initial data for the model (5)-(13) do not contain the characteristics of the granulometric composition, strength and other physical and mechanical properties of the material under study. These properties may remain unknown, but their influence is indirectly taken into account by means of and , i.e., the "black box" methodology is used, when instead of the properties and relationships of the components of the system, the reaction of the system as a whole to changing conditions is studied [23]. Thus, the model (5)-(13) is quite universal, but additional research is needed to determine the scope of the rational application of the model. ...
The relevance of problems related to the fracturing of engineering materials and structures will not decrease over time. Fracture mechanics methods continue to be developed, which, combined with numerical methods of computer modeling, are implemented in software packages. However, this is only one facet of the complex of actual problems related to modeling and analyzing the behavior of brittle materials. No less important are the problems of developing not only numerical, but also new analytical models. In this paper, analytical models of only one class are considered, the distinguishing feature of which is that they describe the full load–strain curve using only one equation. However, the determination of model parameters requires tests for which the destruction of the test object is necessary, which may be unacceptable if controlled destruction is technically impossible or economically unreasonable. At the same time, in practice, it is possible to obtain values of stresses and strains caused by loads smaller than the peak load. Pre-peak loads can be used to predict strength using numerical methods, but it is desirable to have a suitable analytical model to extend the capabilities and to reduce the cost of applied research. Such a model was not found in the known literature, which motivated this work, which aims to modify the analytical model to predict strength and the full load–displacement (or stress–strain) curve using only pre-peak loading. This study is based on the analysis of known data and synthesis using mathematical modeling and fracture mechanics. The input data for the model do not include the particle size distribution and other physical and mechanical properties of the components of the material under study. These properties may remain unknown, but their influence is taken into account indirectly according to the “black box” methodology. Restrictions of the scope of the model are defined. The simulation results are consistent with experiments known from the literature.
... Realist evaluation offers this deeper view; it enables the evaluator to examine the underlying mechanisms -the elements that 'intervene between the delivery of a programme and the occurrence of outcomes' (Weiss 1997, p. 46) -to elucidate the 'black box' of intervention, focussing not on the effects of an intervention, but the conditions in which those effects are produced (Astbury and Leeuw 2010). Therefore, realist evaluation affords a more nuanced understanding; it is cognisant to the breadth of the OHD and the multiplicity of its participants; thus, this study presents a valuable contribution to a developing field. ...
This study presents an evaluation of an online game-based student access, retention, progression and attainment (ARPA) initiative at the University of Kent. The initiative, a narrative-based simulation of a condensed student journey from pre-enrolment to graduation, is designed to prepare and support students in their transition to and participation in Higher Education. Student retention continues to be a perennial issue across the Higher Education sector, and studies have indicated that the more knowledgeable and informed students are about their university environment, the less likely they are to leave before completing their studies. Many institutions have developed interventions with the express purpose of addressing these concerns. Recognising the contextualised and subjective nature of such interventions, a realist evaluative framework was adopted to better understand the initiative under scrutiny, asking what works, for whom and in what circumstances. Participant interviews were utilised to assess the efficacy of the initiative in supporting students and in helping them to navigate often unfamiliar institutional cultures, practices and expectations. A revised programme theory is presented, enabling deeper insight into the merit of the initiative and its overall worth as a mechanism for change within the ARPA paradigm.
... This theoretical presentation is in the early phase of field-testing pathways, processes, and intervening variables that form causal mechanisms to measure innovation and change. Causal mechanisms include the choices and capacities which lead to regular patterns of social behavior that are usually hidden and sensitive to variation in context (Pawson & Tilley, 1997;Astbury & Leeuw, 2010). There is a need to look more deeply into the generative causation of successful programs to enable evaluators to make more credible statements on causal links between the contribution of an intervention and the observed effects (Schmitt, 2020). ...
... 15) rather than surmising whether the type of intervention is effective or not (Pawson, 2013). Drawing on the premise that underlying theory provides the answers to why and how interventions work in some circumstances, but not in others (Astbury and Leeuw, 2010) we applied the CIMO-logic (Denyer et al., 2008). The context (C) is defined as the environment or human factors, the intervention (I) as specified in the research question, the mechanisms (M) created by the intervention as the key components for its efficacy, and the outcomes (O) ranging from performance to cost reduction (Denyer et al., 2008) also encompassing potential interactions between respective units of analysis (for an example see Doyle and McDowall, 2019) throughout the review process including the review question, data extraction, quality assessment, and interpretation of findings. ...
Attention Deficit Hyperactivity Disorder (ADHD) is estimated to affect 3.5% of the global workforce. Despite the high prevalence rate, little is known about how best to support adults with ADHD (ADHDers) at work. Relevant research is dispersed across different disciplines such as medicine, health studies and psychology. Therefore, it is important to synthesize interventions aimed at ADHDers to examine what learning can be gleaned for effective workplace support. We conducted a systematic review of relevant interventions framed by realist evaluation and the Context-Intervention-Mechanism-Outcome classification to identify key mechanisms of effectiveness for workplace interventions. We searched 10 databases including a range of journals from medical science to business management applying predetermined inclusion criteria and quality appraisal through a risk of bias assessment for quantitative and qualitative methods. We synthesized 143 studies with realist evaluation. Most studies evaluated the effectiveness of pharmacological interventions highlighting the dominance of the medical approach to supporting ADHDers. Key mechanisms of effectiveness were identified from psychosocial interventions including group therapy, involvement of people in the ADHDers network, and the importance of the client-patient relationship. Overall, there is limited research that examines the effectiveness of workplace interventions for ADHDers. Furthermore, much of the existing research evaluates pharmacological interventions which is difficult to transfer to the workplace context. It is recommended that future research and practice consider the key mechanisms identified in this review when designing interventions as well as barriers to accessing support such as disclosure and self-awareness.
Esta publicación recoge la selección de trabajos presentados con ocasión de las VII Jornadas de Planificación “70 años CEPAL: Planificación para el desarrollo con visión de futuro”, realizadas el 22 y 23 de octubre de 2018 en la sede de CEPAL en Santiago de Chile. Estas Jornadas se han consolidado como un espacio emblemático de encuentro entre especialistas de la región en los temas de la planificación y de la gestión pública en América Latina y el Caribe.
En esta oportunidad, se hizo un llamado a trabajos que tomen como base experiencias y reflexiones interesadas en conocer e informar cómo articular las diferentes fases de la planificación y de la gestión pública, muy especialmente planificación y presupuesto, así como conjugar diferentes sectores, y/o distintos niveles de gobierno. Se seleccionaron 26 ponencias organizadas en seis partes, buscando abordar los procesos de planificación, como una aproximación para resolver los desafíos de multescalaridad, intersectorialidad, pluritemporalidad y participación, en el marco de la Agenda 2030 y los Objetivos de Desarrollo Sostenible. Este abordaje integral es la mejor y más coherente forma de construir las bases del nuevo desarrollo que la humanidad reclama.
This article is a tribute to John Mayne’s work on Contribution Analysis. It focuses on the causal claims Contribution Analysis aims to address, and on how these have evolved since the approach was first published by John in 1999. It first sets out four types of causality with relevance for Contribution Analysis: counterfactual, generative, INUS, and probabilistic causation. It then describes how John integrated the INUS condition and probabilistic elements into the Contribution Analysis approach, followed by how John’s thinking evolved regarding the question of whether the approach could—and should—also address counterfactual questions. The article concludes with observations on how Contribution Analysis can flexibly integrate elements from different causality types.
Introduction
Health visiting services, providing support to under 5s and their families, are organised and delivered in very different ways in different parts of the UK. While there has been attention to the key components of health visiting practice and what works well and how, there is little research on how health visiting services are organised and delivered and how that affects their ability to meet their objectives. The COVID-19 pandemic rapidly disrupted service delivery from March 2020. This realist review aims to synthesise the evidence on changes during the pandemic to identify the potential for improving health visiting services and their delivery.
Methods and analysis
This review will follow the RAMESES (Realist And Meta-narrative Evidence Syntheses: Evolving Standards) quality standards and Pawson’s five iterative stages to locate existing theories, search for evidence, select literature, extract data, synthesise evidence and draw conclusions. It will be guided by stakeholder engagement with practitioners, commissioners, policymakers, policy advocates and people with lived experience. This approach will consider the emerging strategies and evolving contexts in which the services are delivered, and the varied outcomes for different groups. A realist logic of analysis will be used to make sense of what was happening to health visiting services during and following the pandemic response through the identification and testing of programme theories. Our refined programme theory will then be used to develop recommendations for improving the organisation, delivery and ongoing postpandemic recovery of health visiting services.
Ethics and dissemination
General University Ethics Panel approval has been obtained from University of Stirling (reference 7662). Dissemination will build on links to policymakers, commissioners, providers, policy advocates and the public. A range of audiences will be targeted using outputs tailored to each. A final stakeholder event focused on knowledge mobilisation will aid development of recommendations.
PROSPERO registration number
CRD42022343117.
This chapter turns in the direction of cultural explanations in order to make sense of the governance challenges documented Indonesia in previous chapters. In so doing, the chapter emphasizes what, in chapter two, was termed the anthropological perspective. Over the course of several sections, this chapter goes beyond surface level statements of culture to explore its deeper foundations, rooted not only in the governance practices of the New Order regime but also in the extractive and profit-oriented logics that were central to the colonial period. In these sections, the state ideology of Pancasila is discussed along with the “ethos of privatization” that is at the heart of the Indonesian bureaucracy as a semi-peripheral state in the world system. This chapter also advances the concept of “ritual governance,” first, in order to characterize the dynamics that have been witnessed in previous chapters, second, in order to connect back to the notion of mechanisms from the analytic framework in chapter two, and, third, in order to highlight the impossibility of technical solutions. Explicitly, it is ritual governance, and not the other mechanisms espoused by proponents of decentralization (e.g., accountability, participation), that connects with the logics of the Indonesian state and explains the way that the government operates—and to what ends. Through the concept of ritual governance, it is seen that the shortcomings of decentralization are not shortcomings at all; they are essential features of a system that operates according to a particular logic.
This chapter seeks to answer the question of what kind of conceptual framework can enable one, first, to contextualize and to unpack the World Bank’s engagement with a client government around policy change, second, to explain the implementation of, and challenges to, the reforms in practice that have been promoted by the World Bank (and adopted by a national government), and, third, to bring these first two issues into conversation to show how one impacts the other (and doesn’t). The answer to this question is based on a synthesis of five theoretical approaches. These approaches are rooted in: (a) political economy (specifically the “critical,” “international,” and “cultural” variants), (b) the literature on mechanisms of influence in global governance, (c) realist evaluation, (d) systems thinking, and (e) anthropology. In short, the first two of these set the macro framework and dynamics within which the work and influence of the World Bank is understood; the third (realist evaluation) directs attention to the logic of the reforms that will be unpacked; and the fourth (systems theory) and fifth (anthropology) help to flesh out the dimensions of interest when focusing on the context of implementation that is highlighted but not fully explained by realist evaluation. The essential characteristics of each of these approaches are discussed and integrated.
Una de las consecuencias negativas de los procesos de desarrollo que han tenido lugar en la Unión Europea son las disparidades en el desarrollo de las ciudades y zonas urbanas, que afectan tanto a los ciudadanos como al nivel de actividad económica, lo que puede conducir a la degradación del propio espacio. Las disparidades pueden entenderse como el resultado de un fallo del mercado. Por lo tanto, es preciso llevar a cabo intervenciones que guarden relación con un proceso de regeneración. A
fin de abordar este problema, la Comisión Europea decidió presentar la iniciativa de Ayuda Europea Conjunta en Apoyo de Inversiones Sostenibles en Zonas Urbanas (JESSICA), como parte de la política de cohesión ejecutada en el período comprendido entre 2007 y 2013.
JESSICA es un instrumento financiero rotatorio que se centra en el desarrollo sostenible de las ciudades, y 11 Estados miembros decidieron poner a prueba la solución propuesta. El primer país en firmar el acuerdo fue Polonia, que implantó JESSICA en cinco regiones (Pomorskie, Mazowieckie, Śląskie, Wielkopolskie y Zachodniopomorskie). Este enfoque innovador entrañó numerosos desafíos para todas las partes interesadas (actores europeos, regionales y locales procedentes tanto del sectorpúblico como del privado) que participaron en la planificación y la implementación de la iniciativa.
La metodología empleada se basa en el paradigma positivista, que examina las relaciones causales entre las intervenciones públicas (proyectos) y sus efectos. Se ha adoptado un enfoque realista en las técnicas de evaluación, así como en nuestra propia evaluación de todos los proyectos, realizada por expertos, que se han fundamentado a través de numerosas herramientas analíticas como el agrupamiento jerárquico y aglomerativo.
De las principales conclusiones del estudio se desprende que los proyectos ejecutados en Polonia tan solo cumplen parcialmente los supuestos teóricos, y que las imensiones social y horizontal del desarrollo sostenible han estado infrarrepresentadas. Se observan ciertas divergencias a nivel regional en lo que respecta al enfoque modelo, y una región destaca en cuanto a la calidad de los proyectos (Zachodniopomorskie). Los proyectos relacionados con el patrimonio histórico o cultural, seguidos de los que se centran en la reconversión de los terrenos industriales abandonados, son los que mejor reflejan la complejidad del desarrollo urbano sostenible.
Objective
Measurement-based care (MBC) refers to the routine use of symptom rating scales to guide treatment decisions. Although effective, it is an underused approach to enhance patient care. A significant barrier to integration of MBC is insubstantial foundational training. This scoping review aims to survey the literature on MBC educational curricula for mental health trainees.Methods
Investigators searched Ovid Medline, PsycINFO, Embase, Cochrane Central, and Ebsco CINAHL through June 2021 to select records that described studies of MBC educational programs for undergraduate, graduate, or postgraduate learners in mental healthcare.ResultsFrom 1270 unique records, 1263 were excluded in abstract/title and full-text screening. This scoping review included seven articles, of which most were empirical or case studies and took place in the USA. These curricula involved many delivery formats, including lectures and in-service training. Measured learner outcomes include those that are learner-focused (i.e. learner reaction, or attitudinal/behavioral change) and organizational-focused (i.e. increased clinical use of MBC). Mechanisms of positive outcomes are posited to include enhanced stakeholder support and continual curriculum improvement.ConclusionsMBC curricula can be taught in various formats to diverse learners in mental healthcare. Contextual factors, such as dedicated resources, MBC champions, supervisor training, online measurement feedback systems, simple measures, and gathering and disseminating feedback may facilitate curricular success by fostering stakeholder support and continual program improvement. To address literature gaps, future research in MBC education should involve educational frameworks in designing curriculum and address the use of quality improvement approaches in the implementation of MBC education.
Why is it so hard for international development organizations—even ones as well-resourced and influential as the World Bank—to generate and sustain change in the way things are done in those countries where they work? Despite what, in many cases, is decades of investment and effort, why do partner governments continue to engage in those traditional patterns and styles of public service management that international development organizations have sought to supplant with methods that are supposedly more accountable, efficient, and effective? This book provides an answer to these questions. Rather than pathologizing partner governments as the source of the problem—that is, rather than maintaining the distinction between doctor (international development organizations) and patient (partner governments), wherein the patient is seen as unwilling to take their medicine (enacting “good governance” practices)—this book instead reframes the relationship.
The central argument is, first, that the programs and projects of international organizations are introduced into and are constrained by multiple layers of rituals, performative acts, and cultural logics, logics that intersect with and reinforce the political, economic, and social structures in and through which they operate. These dynamics are summarized by the term ritual governance, which is defined as the symbolic and standardized behavior in which actors engage as they participate in the “traditions and institutions by which authority in a country is exercised” (Kaufmann et al., 1999, p. 1). As will be seen, the contextual factors that guide governance practices are largely beyond the reach of the international development organizations; the relevant logics have their roots in state ideology but also extend back to the colonial logics that continue to operate at the heart of the state apparatus.
The second central argument is that international aid organizations and the governments with which they work are engaged in a “ritual aid dance” (Bull, 2005) where each actor plays a part but does not (and cannot) acknowledge the ways that it depends on—or at least uses—the other for its own gain. The ritual aid dance is understood as a form of ritual governance, but one that is specific to the relationship between a given government or governments, on one hand, and any organization offering “aid” in the context of international development work, on the other—though it should be noted that the ritual aid dance can also be analyzed in terms of how international organizations engage with each other in order to preserve the legitimacy of the international development enterprise. This process can be considered a dance because each participant responds to and needs the other, and because both sides do so in ways that are carefully choreographed, with the overall trajectory or contours of the dance being more or less known to the participants.
These arguments are based on research on the World Bank’s efforts over the course of several decades to encourage, through its financing, projects, and technical assistance, the implementation of social sector reform in Indonesia.
Developing and evaluating health interventions for the benefit of patients is notoriously difficult. This also applies to the discipline of nursing, owing to the complexity of nursing interventions. Following significant revision, the updated guidance of the Medical Research Council (MRC) adopts a pluralistic view to intervention development and evaluation, including a theory-based perspective. This perspective promotes the use of program theory, aiming to understand how and under what circumstances interventions lead to change. In this discussion paper, we reflect the recommended use of program theory in the context of evaluation studies addressing complex nursing interventions. First, we review the literature by investigating the question whether and how evaluation studies targeting complex interventions used theory and to what extent program theories may contribute to enhance the theoretical foundations of intervention studies in nursing. Second, we illustrate the nature of theory-based evaluation and program theories. Third, we argue how this may impact theory building in nursing in general. We finish by discussing which resources, skills and competencies are necessary to fulfill the demanding task of undertaking theory-based evaluations. We caution against an oversimplified interpretation of the updated MRC guidance regarding the theory-based perspective, e.g. by using simple linear logic models, rather than articulating program theories. Instead, we encourage researchers to embrace the corresponding methodology, i.e. theory-based evaluation. With the prevailing perspective of knowledge production in crisis, we might be on the verge of a paradigm shift in health intervention research. Viewed through this lens, the updated MRC guidance could lead to a renewed understanding of what constitutes useful knowledge in nursing. This may facilitate knowledge production and, thereby, contribute to improve nursing practice for the benefit of the patient.
Background
This study evaluated an attempt to implement video consultations through a novel approach to telehealth training and implementation in two middle-sized hospitals in Denmark. Three units tested the novel approach along with a regional decision to strengthen multidisciplinary and cross-sectoral collaboration through technology to improve service delivery by making the process more coherent, as well as to save time and resources. This study aims to investigate what contextual factors enable workplace learning, skills acquisition, and utilization of new digital skills to use video consultations in workplace practice.
Methods
This comparative case study draws on the principles of realist evaluation using a qualitative approach concerned with testing and refining program theories by exploring the complex and dynamic interaction among context, mechanism, and outcome. The methods in this study included participant observations, document analysis, semi-structured individual interviews, and focus groups. We performed an interpretive analysis, which explored the context-mechanism-outcome relationship using the guiding question, “what works, for whom, under what circumstances, and why?”
Results
Two broad mechanisms appeared to enable skills acquisition and routinization of video consultations: informal workplace learning and adjusting the use of video consultations to professional judgment. The three units had different approaches to the implementation and training and, as such, had different outcomes. First, the skills acquired in the units differed; therefore, how and with whom they used video consultations varied. Second, video consultation use was more likely to be adjusted to workflows if unit managers were responsive to staff’s professional judgments regarding patients, as was evident in all three units.
Conclusion
Our study shows that a formal training course alone is insufficient to provide healthcare professionals with the skills needed to use video consultations in workplace practice. Informal workplace learning with support on the spot and continuous follow-up seems to equip healthcare professionals with the skills they need to use video consultations. Video consultations are more likely to be used confidently if novel workflows are adjusted to health care professionals' knowledge, skills, and judgment and their concerns regarding patient soundness.
Making key insights from 80 years of systems thinking available. System Dynamics.
Research in the quality and safety field often necessitates an approach that supports the development of an in-depth understanding of how a complex phenomenon occurs, or how an intervention works. Realist review is an increasingly popular form of evidence synthesis that provides a theory-driven, interpretive approach to secondary research. Realist reviews offer quality and safety researchers the opportunity to draw on diverse types of evidence to develop explanatory theory about how, when and for whom interventions ‘work’ or outcomes occur. The approach is flexible, iterative and practical, typically drawing on the experience of policymakers, practitioners and patients throughout the review. With the increasing use of realist reviews, some common misconceptions about the approach have become evident in the literature. This paper introduces what is involved when planning and conducting a realist review, and where the approach can offer most value, as well as outlining common challenges that researchers may face when adopting the approach, and recommended solutions. Our aim is to support researchers who are considering conducting a realist review to understand the key principles and concepts involved, and how they can go about producing high-quality work.
The COVID-19 pandemic resulted in severe supply disruptions and revenue losses, especially for buyers highly dependent on foreign suppliers. Unsurprisingly, scholars and industry experts claim that high dependence on any exchange partner is detrimental to buyers. The literature, however, is ambivalent about whether supply base concentration-the number of sources in a firm's supply base and the degree of importance of each source-benefits or harms buyers. Our study addresses this ambivalence by investigating whether supply base concentration, conceptualized as supplier concentration, supplier country concentration, and carrier concentration, worsens or mitigates the impact of supply disruptions as they unfold over time. We use longitudinal data from ocean bills of lading to examine these relationships in the context of U.S. buyers importing apparel during the COVID-19 pandemic. Our results show that higher pre-disruption supplier concentration and supplier country concentration helped buyers mitigate the impact of supply disruptions related to COVID-19, with diminishing effects once supplier and supplier country concentration exceeded specific inflection points. Conversely, higher pre-disruption carrier diversification helped buyers mitigate the impact of supply disruptions related to COVID-19. Notably, these effects depend on the specific phase of the supply disruption. We discuss implications for research and practice.
Introduction
Objective structured clinical exams (OSCEs) are a cornerstone of assessing the competence of trainee healthcare professionals, but have been criticised for (1) lacking authenticity, (2) variability in examiners’ judgements which can challenge assessment equivalence and (3) for limited diagnosticity of trainees’ focal strengths and weaknesses. In response, this study aims to investigate whether (1) sharing integrated-task OSCE stations across institutions can increase perceived authenticity, while (2) enhancing assessment equivalence by enabling comparison of the standard of examiners’ judgements between institutions using a novel methodology (video-based score comparison and adjustment (VESCA)) and (3) exploring the potential to develop more diagnostic signals from data on students’ performances.
Methods and analysis
The study will use a complex intervention design, developing, implementing and sharing an integrated-task (research) OSCE across four UK medical schools. It will use VESCA to compare examiner scoring differences between groups of examiners and different sites, while studying how, why and for whom the shared OSCE and VESCA operate across participating schools. Quantitative analysis will use Many Facet Rasch Modelling to compare the influence of different examiners groups and sites on students’ scores, while the operation of the two interventions (shared integrated task OSCEs; VESCA) will be studied through the theory-driven method of Realist evaluation. Further exploratory analyses will examine diagnostic performance signals within data.
Ethics and dissemination
The study will be extra to usual course requirements and all participation will be voluntary. We will uphold principles of informed consent, the right to withdraw, confidentiality with pseudonymity and strict data security. The study has received ethical approval from Keele University Research Ethics Committee. Findings will be academically published and will contribute to good practice guidance on (1) the use of VESCA and (2) sharing and use of integrated-task OSCE stations.
In this paper, a backward mapping approach to realist evaluation is used to evaluate the efficacy of an international inclusive education short course program in transforming inclusive perspectives and practices of educators from Nepal who were considered leaders in inclusive education. The program, an Australia Awards South and West Asia (AASWA) short course, delivered by an Australian university, was designed to build on the existing capacity of the participants to lead the implementation of inclusive education reforms and system enhancements within the school system in Nepal. Data collected before, during and after the program provide insights into the ways in which participants responded to the program. Our findings indicate that a deep understanding of context is needed by program planners to identify how and why particular mechanisms are triggered that lead to transformative learning of the participants. We suggest that backward mapping as a methodology is a viable realist evaluation process to evaluate complex programs such as the short-term international aid program on IE. Our paper concludes with recommendations for developers of similar programs in the future.
Objectives
While general practice involves supporting patients to modify their behaviour, General Practitioners (GPs) vary in their approach to behaviour change during consultations. We aimed to identify mechanisms supporting GPs to undertake successful behaviour change in consultations for people with T2DM by exploring (a) the role of GPs in behaviour change, (b) what happens in GP consultations that supports or impedes behaviour change and (c) how context moderates the behaviour change consultation.
Methods
Semi-structured interviews with academic clinicians ( n = 13), GPs ( n = 7) and patients with T2DM ( n = 16) across Australia. Data were analysed thematically using a realist evaluation approach.
Results
Perspectives about the role of GPs were highly variable, ranging from the provision of test results and information to a relational approach towards shared goals. A GP–patient relationship that includes collaboration, continuity and patient-driven care may contribute to a sense of successful change. Different patient and GP characteristics were perceived to moderate the effectiveness and experience of behaviour change consultations.
Discussion
When patient factors are recognised in consultations, a relational approach becomes possible and priorities around behaviour change, that might be missed in a transactional approach, can be identified. Therefore, GP skills for engaging patients are linked to a person-centred approach.
Scientific and Critical Realism attracts increasing attention as a new paradigm of explanation, for many empirical knowledge disciplines. This new approach to explaining our social and material worlds is underpinned by its ‘depth ontology’, encompassing the reality of our senses to the more meta-physical. In this article we introduce and explore this ‘depth ontology’, through rich illustration of these alternative ideas about reality in context of our everyday and early career research experiences. We explain and clarify the realist compromise - between its positivist and constructivist ancestry. We then trace the flow of these philosophical premises into conceptual variation evident around the realist sense of ‘mechanism’, in current evaluation research literature. To further clarify its possible meanings, this synthesis contextualises past and current realist thinking in light of historical ideas of change from Aristotle and Plato, as improvement and degeneration. This article offers a new view on realism and its foundations then, to aid readers’ own understandings and explorations of the natural and social reasons for existence and its changes, sitting in the depths of the universe of the realist.
Introduction
Despite tremendous progress in care, people living with HIV (PLHIV) continue to experience HIV-related stigmatisation by nurses in non-HIV-specialised healthcare settings. This has consequences for the health of PLHIV and the spread of the virus. In the province of Quebec (Canada), only four interventions aimed at reducing the impact of HIV-related stigmatisation by nurses have been implemented since the beginning of the HIV pandemic. While mentoring and persuasion could be promising strategies, expression of fears of HIV could have deleterious effects on nurses’ attitudes towards PLHIV. In literature reviews on stigma reduction interventions, the contextual elements in which these interventions have been implemented is not considered. In order to develop new interventions, we need to understand how the mechanisms (M) by which interventions (I) interact with contexts (C) produce their outcomes (O).
Methods and analysis
Realist synthesis (RS) was selected to formulate a programme theory that will rely on CIMO configuration to describe (1) nursing practices that may influence stigmatisation experiences by PLHIV in non-HIV-specialised healthcare settings, and (2) interventions that may promote the adoption of such practices by nurses. The RS will draw on the steps recommended by Pawson: clarify the scope of the review; search for evidence; appraise primary studies and extract data; synthesise evidence and draw conclusions. To allow an acute interpretation of the disparities between HIV-related stigmatisation experiences depending on people’s serological status, an initial version of the programme theory will be formulated from data gathered from scientific and grey literature, and then consolidated through realist interviews with various stakeholders (PLHIV, nurses, community workers and researchers).
Ethics and dissemination
Ethical approval for realist interviews will be sought following the initial programme theory design. We intend to share the final programme theory with intervention developers via scientific publications and recommendations to community organisations that counter HIV-related stigmatisation.
Truckload carriers face tradeoffs in their effort to build their freight networks from combinations of spot market and contract shipments. These tradeoffs exist because contract freight provides loose volume guarantees but lower linehaul rates, whereas spot freight commands higher linehaul rates while being more stochastic in nature. The issue of carriers’ mix of spot market and contract freight, while of utmost importance in practice, has received limited theoretical and empirical attention. Drawing on dynamic transaction cost theory, we develop middle-range theory that explains why carrier traits such as size and use of team drivers as well as characteristics of the freight markets in which they are embedded impact carriers’ reliance on spot market instead of contract freight. To test our theory, we rely on archival survey data collected by CarrierLists, a firm that specializes in compiling and disseminating motor-carrier information. Our econometric estimates corroborate our theory and remain after robustness testing. Apart from extending theory regarding dynamic transaction costs, our findings have important implications for carriers, brokers, and shippers.
Through exploring the links between policy, evaluation, and gender, using a gender-transformative lens, this article sheds light on policy and evaluation processes of gender equality programmes, such as the political nature of evaluation and power configurations, involving resistance in the change process. Two case studies of gender equality programmes are visited to explore the links between gender policy implementation in specific contexts, and power configurations and resistance as key factors in both implementation and evaluation. We benefitted from the theoretical insights of both the feminist and the theory of change approaches, which helped us to bridge the boundaries between policy, gender scholarship, and evaluation and allowed us to reflect upon systemic factors and underlying mechanisms that are either facilitating or hindering transformation. This article highlights the benefits of applying such an evaluation framework and demonstrates why factoring in power and resistance to evaluations of gender equality programmes is necessary to explain successes or failures.
Preface
"No reality please. We're economists". This heading to a couple of commentaries in a recent edition of The Times Higher Education Supplement (March, 1994), captures an increasingly widely held perception. According to it modern economics is largely irrelevant to an understanding of real world matters. Needless to say this view is not accepted by all economists. And it does seem prima facie implausible as a characterisation of an academic discipline that is often represented as a social science. Yet, my assessment set out below is that it is not wholly mistaken; there is a sense in which social reality is neglected in much of modern economics. I shall indicate furthermore how this neglect is debilitating of the subject, with unfortunate repercussions for the multitude of further activities that draw upon it.
My purpose here, however, is not merely to identify limitations of the discipline but also to work towards transcending them. My overall aim, if somewhat unfashionably, is to bring reality (or more of it) back into economics. Of course, in formulating my goal in this way, and in acknowledging some validity to the perception of the discipline noted above, I am not wanting to suggest that modern economics is devoid of a conception of reality altogether. Indeed, it could not be. For the adoption of any method, practice or goal, presupposes a world-view, even if it is left implicit and unexamined. The feature of modern economics which is remarkable here is the minimal concern that is shown either for tailoring methods to insights available regarding the nature of the social world, or for explicitly determining the sorts of conditions under which chosen methods would be appropriate. This is despite a dearth of successes according to criteria which the discipline sets itself. Rather, methods and procedures are formulated according to the nature or degree of their technical sophistication, or their conformity with a priori conceptions of proper practice, or some such. The majority of economists seem merely to take it for granted that if ever real world problems or situations are to be studied the chosen methods will necessarily `fit'. It is in this sense that reality is neglected in modern economics. And we will see that it is just the presumption that methods of economic analysis can be fashioned without explicit regard to the nature of social phenomena that accounts for the subject's numerous failings.
In short, contemporary economics is marked by an effective neglect of ontology, by a lack of attention to elaborating the nature of (social) being or existence. And the project reported here is motivated by a desire to compensate for this neglect. The objective is to determine a sustainable account of natural and social reality with a view both to explaining and resolving numerous problems which beset the discipline and to gaining an informed vantage-point on the sort of practices that are likely to bear fruit. Theories which address these sorts of concerns and involve commitments to the reality of features elaborated are usually collected under the heading of philosophical (or ontological) realism. The position sustained in this book, then, is an explicitly realist one. The book constitutes a realist theory of and for economics.
This article is intended to stimulate a dialogue between program theory evaluation and action science for the purposes of cross-fertilization and mutual enrichment. Both program theory evaluation and action science use the concept of implicit “theories of action” as a central construct in the study of social practice. However, an action science approach suggests a wider understanding of program theory that (1) specifies the links between individual reasoning and behavior to program implementation, and (2) accounts for how programs deal with dilemmas, conflict, and error. This paper begins with a systematic, though not exhaustive, comparison of program theory evaluation and action science. It analyzes an exemplar of program theory evaluation from an action science perspective to illustrate a subtheory, “designed blindness,” and its impact on both program implementation and the evaluation itself. It then offers a theory for overcoming designed blindness. Finally, this article argues that action science concepts and skills can enable program theory evaluators to be more effective in confronting defensiveness and in facilitating learning among stakeholders when there is a gap between “espoused” program theory and “theory-in-use.”
This paper builds on Clarke and Homel's (in press) expansion of the situational crime prevention model, which includes new techniques for making,the potential offender feel guilty or ashamed,about their contem- plated crime. In place of Clarke and Homel's single category of "inducing guilt or shame," two separate categories involving the manipulation of internal controls (guilt) and social controls (including shame) are proposed. The addition of these categories expands the repertoire of available crime preven- tion techniques by giving fuller recognition to the subtleties and complexities of the motivations to commit crime implicit in the rational choice perspective. It is argued that the new strategies also "soften" the narrow, target-harden- ing image of the situational approach, and may help researchers avoid counterproductive situational crime prevention effects. In a recent revision of Clarke's (1992) classification of situational crime
Past literature has identified several putative precursors of use, as well as alternative forms of use. However, important shortcomings still exist in previous work on use. In particular, inadequate attention has been given to the underlying processes that may mediate the effects of evaluation on attitude and action. In essence, a key part of the theory of change for evaluation itself is missing. To help fill this gap, we describe a framework designed to capture key mechanisms through which evaluation may have its effects. The framework includes change processes that have been validated in various social science literatures. It identifies three levels of analysis (individual, interpersonal and collective), each with four kinds of processes (general influence, attitudinal, motivational and behavioral). With a more comprehensive view of the mechanisms underlying evaluation’s influence, the field can move forward in relation to its understanding and facilitation of evaluation’s role in the service of social betterment.
Social mechanisms are theoretical cogs and wheels that explain how and why one thing leads to another. Mechanisms can run from macro to micro (e.g., explaining the effects of organizational socialization practices or compensation systems on individual actions), micro to micro (e.g., social comparison processes), or micro to macro (e.g., how cognitively limited persons can be aggregated into a smart bureaucracy). Explanations in organization theory are typically rife with mechanisms, but they are often implicit. In this article, the authors focus on social mechanisms and explore challenges in pursuing a mechanisms approach. They argue that organization theories will be enriched if scholars expend more effort to understand and clarify the social mechanisms at play in their work and move beyond thinking about individual variables and the links between them to considering the bigger picture of action in its entirety.
The concept of mechanism is analyzed in terms of entities and activities, organized such that they are productive of regular changes. Examples show how mechanisms work in neurobiology and molecular biology. Thinking in terms of mechanisms provides a new framework for addressing many traditional philosophical issues: causality, laws, explanation, reduction, and scientific change.
This essay examines a mechanisms-oriented approach to theory building in organization studies and identifies conditions under which such an approach is likely to be fruitful. Mechanisms-oriented theorizing is a promising path to reviving organization theory but has the potential pitfalls of overly reductionist models of organizations and the complete abandonment of grand theorizing. These undesirable side effects can be reduced when researchers act as playful pragmatists and treat mechanisms as toolkits for working on theoretical puzzles. The article describes some tactics for accomplishing this goal.
The advancement of social theory requires an analytical approach that systematically seeks to explicate the social mechanisms that generate and explain observed associations between events. These essays, written by prominent social scientists, advance criticisms of current trends in social theory and suggest alternative approaches. The mechanism approach calls attention to an intermediary level of analysis in between pure description and story-telling, on the one hand, and grand theorizing and universal social laws, on the other. For social theory to be of use for the working social scientist, it must attain a high level of precision and provide a toolbox from which middle range theories can be constructed.
The mandate for evaluators to ‘look inside the black box’ of an intervention has become a familiar and heeded cry. So whether it is process evaluators with their logic maps, or theories-of-change researchers with their intervention stepping stones, or realists with their context, mechanism outcome configurations, searchlights have been aimed into the gloom. So much so that a contemporary sounding riposte, ‘been in there, done for that’ might be deemed to reflect the current state of play in evaluation research.
This article, nonetheless, warns against complacency. It peers even further into the darkest reaches of the inky blackness, and urges other researchers to follow. It throws light on the presence of an overlooked set of program mechanisms, so deeply buried that they are almost invisible. The processes I have in mind are missed because they are tacit, mundane, over-familiar, and taken for granted and, as result, they often overlooked. And yet they are often responsible for a goodly part of impact of a goodly number of interventions.
As such, they deserve a sustained program of research and this article sets out a brief agenda for such inquiries.
In this new edition of his critically acclaimed book, Jon Elster examines the nature of social behavior, proposing choice as the central concept of the social sciences. Extensively revised throughout, the book offers an overview of key explanatory mechanisms, drawing on many case studies and experiments to explore the nature of explanation in the social sciences; an analysis of the mental states - beliefs, desires, and emotions - that are precursors to action; a systematic comparison of rational-choice models of behavior with alternative accounts, and a review of mechanisms of social interaction ranging from strategic behavior to collective decision making. A wholly new chapter includes an exploration of classical moralists and Proust in charting mental mechanisms operating ‘behind the back’ of the agent, and a new conclusion points to the pitfalls and fallacies in current ways of doing social science, proposing guidelines for more modest and more robust procedures.
Over the past few decades serious reservations have been expressed about the explanatory power of sociological theory and research. In this important book, leading social theorist Peter Hedström outlines the foundations of an analytically oriented sociology that seeks to address this criticism. Building on his earlier influential contributions to contemporary debates, Professor Hedström argues for a systematic development of sociological theory so that it has the explanatory power and precision to inform sociological research and understanding. He discusses various mechanisms of action and interaction and shows how strong links can be forged between the micro and the macro, and between theory and empirical research. Combining approaches to theory and methodology and using extensive examples to illustrate how they might be applied, this clear, concise and original book will appeal to a broad range of social scientists.
This book asks, and I trust answers, a direct and seemingly simple question-can sociology be substantiated? What I want to do is referee the fight between those in the red corner for whom sociology is a substantive discipline, substantiated by valid and reliable evidence about the real world versus those in the blue corner, who would have it that sociological knowledge is discursive, based on conceptual building blocks and tied together by convention and conviction.
* Introduction Models Of Explanation * Causal Analysis * Rational Choice Theory * Interpretation Theory Variations And Elaborations * Functional and Structural ExplanationMaterialismEconomic AnthropologyStatistical Analysis Current Controversies * Methodological IndividualismRelativismToward Methodological Pluralism
PART ONE: INTRODUCING CRITICAL REALISM Introduction Key Features of Critical Realism in Practice A Brief Introduction PART TWO: POSTMODERN-REALIST ENCOUNTERS Introduction Realism for Sceptics Postmodernism and the Three 'PoMo' Flips Essentialism, Social Constructionism and Beyond PART THREE: Social Science and Space Introduction Space and Social Theory Geohistorical Explanation and Problems of Narrative PART FOUR: CRITICAL REALISM: FROM CRITIQUE TO NORMATIVE THEORY Introduction Critical Realism and the Limits to Critical Social Science Ethics Unbound For a Normative Turn in Social Theory
This article is intended to stimulate a dialogue between program theory evaluation and action science for the purposes of cross-fertilization and mutual enrichment. Both program theory evaluation and action science use the concept of implicit "theories of action" as a central construct in the study of social practice. However, an action science approach suggests a wider understanding of program theory that (1) specifies the links between individual reasoning and behavior to program implementation, and (2) accounts for how programs deal with dilemmas, conflict, and error. This paper begins with a systematic, though not exhaustive, comparison of program theory evaluation and action science. It analyzes an exemplar of program theory evaluation from an action science perspective to illustrate a subtheory, "designed blindness," and its impact on both program implementation and the evaluation itself. It then offers a theory for overcoming designed blindness. Finally, this article argues that action science concepts and skills can enable program theory evaluators to be more effective in confronting defensiveness and in facilitating learning among stakeholders when there is a gap between "espoused" program theory and "theory-in-use.".
Disputes in educational research over the past few decades have resulted in part from an inadequate conception of the nature of science itself. Developments in the philosophy of science have led to a new understanding–scientific realism–that has promise of resolving many longstanding dilemmas. At the core of the "standard view" of science is the incorrect Humean notion of causation, which has had devastating effects on research in the social sciences. An adequate notion of causation may provide a framework for research that is at once scientific, that incorporates the perceptions and intentions of participants, and that advances critical values such as social justice.
The Crime Reduction Programme began in April 1999 and was largely finished by March 2002. It went through a number of major changes. It also incorporated an ambitious initial evaluation component and a range of forms of evaluation activity. This article traces the changing theories in and of the programme as a whole and the changing forms of evaluation that were conducted under its auspices. It attempts to explain how and why the programme metamorphosed and how and why provisions for evaluation activity evolved as they did. It tries to distil realistic lessons for the conduct of informative evaluation in the context of large-scale government programmes. It also attempts to locate the potential for realist evaluation within such programmes.
The idea of theory-based evaluation (TBE) is plausible and cogent, and it promises to bring greater explanatory power to evaluation. However, problems beset its use, including inadequate theories about pathways to desired outcomes in many program areas, confusion between theories of implementation and theories of programmatic action, difficulties in eliciting or constructing usable theories, measurement error, complexities in analysis, and others. This article explores the problems, describes the nature of potential benefits, and suggests that the benefits are significant enough to warrant continued effort to overcome the obstacles and advance the feasibility of TBE.
The aim of this article is to elucidate the notions of explanation and mechanism, in particular of the social kind. A mechanism is defined as what makes a concrete system tick, and it is argued that to propose an explanation proper is to exhibit a lawful mechanism. The so-called covering law model is shown to exhibit only the logical aspect of explanation: it just subsumes particulars under universals. A full or mechanismic explanation involves mechanismic law statements, not purely descriptive ones such as functional relations and rate equations. Many examples from the natural, biosocial, and social sciences are examined. In particular, macro-micro-micro-macro social relations are shown to explain other wise puzzling macro-macro links. The last part of the article relates the author's progress, over half a century, toward understanding mechanism and explanation.
After an extraordinarily promising beginning in 1937 with The Structure of Social Action, Talcott Parsons abandoned his attempt to ground social theory in a theory of purposive action. The functionalism that resulted moved in one direction, while social research has progressively moved in an individual-behavioristic direction, resulting in an ever-widening divergence between research and theory. This paper describes paths in research and in theory development, that will reconstitute relevance of each for the other. The essential elements are two. The first is use of a theory of purposive action as a foundation for social theory; this entails acceptance of a form of methodological individualism and rejection of holism. The second is a focus in social research and theory on the movement from the level of individual actions to macrosocial functioning, that is, the level of system behavior.
This paper has two parts. The first provides a general discussion of the nature of theories—and the reasons why some understanding of this abstract subject is of value in the practice of evaluation. The second part looks more closely at the dependence of good practice on a minimalist theory of evaluation, and at what such a theory comprises.
British criminology has evolved its own distinctive division of labour. Theorists theorize, ethnographers empathize, statisticians soothsay, and so on. Unlike other areas of scholarship, this arrangement allows little time or gain for anyone bent on fundamental reflection on the nature of criminological research. Whilst (thankfully) this has had the effect of avoiding the unproductive epistemological brawls which characterize neighbouring disciplines, it can lead to a directionless research strategy founded on a taken-for-granted pragmatism. This paper examines the case of 'evaluation research' in the criminal justice area and attempts to inject some strategic thinking into this, the dourest, most matter-of-fact corner of the literature. On offer is a brief (and at this length) polemical case, arguing that the quasi-experimental paradigm has resulted in moribund evaluation, being itself a contributing factor to the 'nothing works' lament. Rescue is at hand in the form of a 'scientific realist' approach to evaluation. © 1994 The Institute for the Study and Treatment of Delinquency.
The abstract for this document is available on CSA Illumina.To view the Abstract, click the Abstract button above the document title.
Several authors have claimed that mechanisms play a vital role in distinguishing between causation and mere correlation in the social sciences. Such claims are sometimes interpreted to mean that without mechanisms, causal inference in social science is impossible. The author agrees with critics of this proposition but explains how the account of how mechanisms aid causal inference can be interpreted in a way that does not depend on it. Nevertheless, he shows that this more charitable version of the account is still unsuccessful as it stands. Consequently, he advances a proposal for shoring up the account, which is founded on the possibility of acquiring knowledge of social mechanisms by linking together norms or practices found in a society.
The author argues that evaluation efforts such as outcome studies can be profitably undertaken in the absence of knowing how or why a program works. This is labeled black box evaluation. In gray box evaluation, one may discern the components of an intervention but not their principles of operation. And in clear box evaluation, the inner workings of an intervention are fully revealed. The professional imperative of the evaluator is to evaluate. To do this we do not need to know how the programs work or why they fail to work or even what their components are. (Abstract prepared by the Research on Social Work Practice editor.)
Program Theory-Driven Evaluation Science fills the gap between 21st century literature on evaluation and what is happening in practice. it features detailed examples of how evaluations actually unfold in practice to develop people, programs, and organizations. Commonly accepted strategies for practicing evaluation are outlined, followed by comprehensive accounts of how those strategies have played out in the face of the complexities and challenges of “real world” settings. in so doing, the book illustrates the authentic challenges of implementing an evaluation approach in practice.
Mechanisms in a theory are defined here as bits of theory about entities at a different level (e.g., individuals) than the main entities being theorized about (e.g., groups), which serve to make the higher-level theory more supple, more accurate, or more general. The criterion for whether it is worthwhile to theorize at lower levels is whether it makes the theory at the higher levels better, not whether lower-level theorizing is philosophically necessary. The higher-level theory can be made better by mechanisms known to be inadequate in the discipline dealing with the lower level. Conditions for the usefulness of lower-level theorizing are proposed, with many examples from various social and physical sciences.
The advancement of social theory requires an analytical approach that systematically seeks to explicate the social mechanisms that generate and explain observed associations between events. These essays, written by prominent social scientists, advance criticisms of current trends in social theory and suggest alternative approaches. The mechanism approach calls attention to an intermediary level of analysis in between pure description and story-telling, on the one hand, and grand theorizing and universal social laws, on the other. For social theory to be of use for the working social scientist, it must attain a high level of precision and provide a toolbox from which middle range theories can be constructed.
This article addresses the following problems: What is a mechanism, how can it be discovered, and what is the role of the knowledge of mechanisms in scientific explanation and technological control? The proposed answers are these. A mechanism is one of the processes in a concrete system that makes it what it is — for example, metabolism in cells, interneuronal connections in brains, work in factories and offices, research in laboratories, and litigation in courts of law. Because mechanisms are largely or totally imperceptible, they must be conjectured. Once hypothesized they help explain, because a deep scientific explanation is an answer to a question of the form, “How does it work, that is, what makes it tick—what are its mechanisms?” Thus, by contrast with the subsumption of particulars under a generalization, an explanation proper consists in unveiling some lawful mechanism, as when political stability is explained by either coercion, public opinion manipulation, or democratic participation. Finding mechanisms satisfies not only the yearning for understanding, but also the need for control.