Article

Evaluating complex interventions: A theory-driven realist-informed approach

Authors:
To read the full-text of this research, you can request a copy directly from the authors.

Abstract

There is a growing recognition that programs that seek to change people’s lives are intervening in complex systems, which puts a particular set of requirements on program monitoring and evaluation (M&E). Developing complexity-aware M&E systems within existing organizations is difficult because they challenge traditional orthodoxy. Little has been written about the practical experience of doing so. This article describes the development of a complexity-aware evaluation approach in the CGIAR Research Program on Aquatic Agricultural Systems. We outline the design and methods used including trend lines, panel data, after action reviews, building and testing theories of change, outcome evidencing and realist synthesis. We identify and describe a set of design principles for developing complexity-aware M&E. Finally, we discuss important lessons and recommendations for other programs facing similar challenges. These include developing evaluation designs that meet both learning and accountability requirements; making evaluation as part of a program’s overall approach to achieving impact; and, ensuring evaluation cumulatively builds useful theory as to how different types of program trigger change in different contexts.

No full-text available

Request Full-text Paper PDF

To read the full-text of this research,
you can request a copy directly from the authors.

... On the other hand, the environments these programs operate in are characterized by uncertainty and unpredictability; for example, resulting from violent conflict, political turbulence, or erratic weather patterns. This implies feedback loops, multiple related causes, and lengthy time frames, inter alia (Douthwaite et al., 2017;Mayne and Stern, 2013;Pawson, 2013). The coupled complexity of programs and contexts means that pathways to impact are difficult if not impossible to anticipate, and interventions are designed, steered, and adapted along the way (Ling, 2012;Ripley and Jaccard, 2016). ...
... Programs that face double complexity are messy to understand. M&E approaches within these programs should anticipate outcomes that are unknown, untangle multiple related pathways to impact, and incorporate dynamic contextual contingencies (Douthwaite et al., 2017). At the same time, M&E systems must remain actionable and practicable, while being constrained by available resources. ...
... They need to capture the wide variety of intervention logics of partnerships, while going into depth to make sense of complex, multifaceted, and non-linear pathways of change. M&E systems that credibly capture double complexity are therefore likely to be complex and extensive themselves, requiring numerous resources (Douthwaite et al., 2017). An acceptable M&E system from the perspective of credibility would thus require an extensive and robust set-up. ...
Article
Full-text available
Evaluation in complex programs assembling multiple actors and combining various interventions faces contradictory requirements. In this article, we take a management perspective to show how to recognize and accommodate these contradictory elements as paradoxes. Through reflective practice we identify five paradoxes, each consisting of two contradicting logics: the paradox of purpose—between accountability and learning; the paradox of position—between autonomy and involvement; the paradox of permeability—between openness and closedness; the paradox of method—between rigor and flexibility; and the paradox of acceptance—between credibility and feasibility. We infer the paradoxes from our work in monitoring and evaluation and action research embedded in 2SCALE, a program working on inclusive agribusiness and food security in a complex environment. The intractable nature of paradoxes means they cannot be permanently resolved. Making productive use of paradoxes most likely raises new contradictions, which merit a continuous acknowledging and accommodating for well-functioning monitoring and evaluation systems.
... The affluent are able to net the benefits of intensification and are less disadvantaged by any loss of ecosystem services than more vulnerable groups who generally have a higher dependence on these services for their livelihoods. These examples illustrate that the pathways to impact are highly uncertain, a characteristic that Douthwaite, Mayne, McDougall, and Paz-Ybarnegaray (2017) attribute to the typical complexity of programs across the international aid and R4D sectors (including agricultural research). ...
... Purely disciplinary methodologies fail to capture this broader context and are inadequate for understanding the complex phenomena that emerge in rural communities in developing countries, namely poverty, gender and social inequality, and food insecurity (Butler et al., 2017). Recognising this, integration sciences are progressively being used to explore solution spaces for complex social-agroecological problems (Aravindakshan et al., 2021;Butler et al., 2017;Douthwaite, Mayne, et al., 2017;Ha, Bosch, Nguyen, & Trinh, 2017). With this orientation, integrative research methods and framings draw from multiple disciplines and inter-or-transdisciplinary interactions within the project team and with others who inform, use or are impacted by the research (Blundo-Canto et al., 2019). ...
... Although the project team did not explicitly set out to embed the IA and MEL, the synergies of these processes emerged over time (Figure 3). The IA process in the SIAGI project contributed to evaluation and monitoring steps in the Theory of Change for Monitoring and Evaluation of Douthwaite, Mayne, et al. (2017). ...
Article
Full-text available
Large investments in Research-for-Development (R4D) have occurred around agricultural intensification to improve social and economic outcomes for poor small and marginal farmer households. Mixed evidence for sustained and socially just impacts from these investments reflects that projects aimed at achieving social change are inherently complex and the pathways from intervention to impact are deeply uncertain. R4D projects are increasingly drawing on integrative approaches to explore solution spaces for these complex social-agroecological problems; albeit integration science is not yet mainstream in R4D. We reflect on one approach (integrated assessment, IA) in a project on socially inclusive agricultural intensification, namely on how the project team embraced integration tools and research approaches, translated knowledge and learnings of the community and broader research team into systems frameworks, and ensured that social inclusion and justice concepts were central to the IA tools and process. IA was valued for its participatory focus and for lessening ‘silo thinking’ in the design of community interventions and research activities. We argue that complexity-aware integration approaches like IA are needed to support the design, monitoring and evaluation of R4D projects to enhance outcomes and achieve sustained impact.
... Realist evaluation starts from the premise that 'Nothing will work everywhere all of the time'. 31 The fundamental question that researchers working with a realist perspective ask is how do certain causal mechanisms (e.g. GP-led advocacy) operating in particular circumstances (e.g. a HHDS) create certain changes or outcomes (e.g. ...
... Within this view, it is acknowledged that programmes and interventions do not change people, rather it is how people interpret and use what the programme provides that changes things. 31 Within a realist evaluation a context describes the conditions in which programmes and interventions are introduced, acknowledging that mechanisms will be active only in particular circumstances. Context includes cultural norms, economic conditions, public policy, etc. Mechanism describes what it is about programmes and interventions that bring about any effects (i.e. ...
... We included a small number of additional articles that considered homeless health or hospital discharge more generally, but only where they raised questions about the need for intermediate care. 26,[30][31][32][33][34][35][36][37] Each source was read by at least two members of the research team. Papers were assessed based on the same realist 'quality' criteria utilised by Pearson et al. 25 This makes distinctions between those that are 'conceptually rich' (with well-grounded and clearly elucidated theories, ideas and concepts), 'thick' (a rich description of a programme, but without explicit reference to theory underpinning it) and 'thin' (weaker description of a programme, where discerning a programme theory would be problematic). ...
Article
Full-text available
Background In 2013, 70% of people who were homeless on admission to hospital were discharged back to the street without having their care and support needs addressed. In response, the UK government provided funding for 52 new specialist homeless hospital discharge schemes. This study employed RAMESES II (Realist And Meta-narrative Evidence Syntheses: Evolving Standards) guidelines between September 2015 and 2019 to undertake a realist evaluation to establish what worked, for whom, under what circumstances and why. It was hypothesised that delivering outcomes linked to consistently safe, timely care transfers for homeless patients would depend on hospital discharge schemes implementing a series of high-impact changes (resource mechanisms). These changes encompassed multidisciplinary discharge co-ordination (delivered through clinically led homeless teams) and ‘step-down’ intermediate care. These facilitated time-limited care and support and alternative pathways out of hospital for people who could not go straight home. Methods The realist hypothesis was tested empirically and refined through three work packages. Work package 1 generated seven qualitative case studies, comparing sites with different types of specialist homeless hospital discharge schemes ( n = 5) and those with no specialist discharge scheme (standard care) ( n = 2). Methods of data collection included interviews with 77 practitioners and stakeholders and 70 people who were homeless on admission to hospital. A ‘data linkage’ process (work package 2) and an economic evaluation (work package 3) were also undertaken. The data linkage process resulted in data being collected on > 3882 patients from 17 discharge schemes across England. The study involved people with lived experience of homelessness in all stages. Results There was strong evidence to support our realist hypothesis. Specialist homeless hospital discharge schemes employing multidisciplinary discharge co-ordination and ‘step-down’ intermediate care were more effective and cost-effective than standard care. Specialist care was shown to reduce delayed transfers of care. Accident and emergency visits were also 18% lower among homeless patients discharged at a site with a step-down service than at those without. However, there was an impact on the effectiveness of the schemes when they were underfunded or when there was a shortage of permanent supportive housing and longer-term care and support. In these contexts, it remained (tacitly) accepted practice (across both standard and specialist care sites) to discharge homeless patients to the streets, rather than delay their transfer. We found little evidence that discharge schemes fired a change in reasoning with regard to the cultural distance that positions ‘homeless patients’ as somehow less vulnerable than other groups of patients. We refined our hypothesis to reflect that high-impact changes need to be underpinned by robust adult safeguarding. Strengths and limitations To our knowledge, this is the largest study of the outcomes of homeless patients discharged from hospital in the UK. Owing to issues with the comparator group, the effectiveness analysis undertaken for the data linkage was limited to comparisons of different types of specialist discharge scheme (rather than specialist vs. standard care). Future work There is a need to consider approaches that align with those for value or alliance-based commissioning where the evaluative gaze is shifted from discrete interventions to understanding how the system is working as a whole to deliver outcomes for a defined patient population. Funding This project was funded by the National Institute for Health Research (NIHR) Health Services and Delivery Research programme and will be published in full in Health Services and Delivery Research ; Vol. 9, No. 17. See the NIHR Journals Library website for further project information.
... When agents respond, they create a discrete path in which they actively interact with the characteristics of the proposition, adapting them to their own situation. The PEDR framework fits with evaluation methods that trace the complex pathway of the contribution of agricultural research to technological change (Douthwaite et al. 2017;Pierre-Benoit Joly and Matt 2017;Springer-Heinze et al. 2003). Guided by the PEDR framework, we adapted the evaluation method proposed by Faure et al. (2020) which reconstructs the history of an innovation process, the endogenous and exogenous factors and the dynamics of the interactions shaping it, and its impact pathway. ...
... Focusing our research on an area marginalised by policy, development and research, and characterised by one-off interventions by isolated stakeholders, enables us to highlight the importance of systemic analyses. Our study follows the recent revival in the agricultural research literature of evaluation methods that address the challenges of complexity and systems thinking (Barjolle, Midmore, and Schmid 2018;Douthwaite et al. 2017;Joly et al. 2015). Like other applications of the PEDR framework (Glover et al. 2021), our results support the case for systematic monitoring and evaluation systems focused on learning during and after the end of projects that target technological change. ...
... The CGIAR Research Program on Aquatic Agricultural Systems (AAS) began in 2011 with the goal "to improve the wellbeing of poor people dependent on aquatic agricultural systems by putting in place the capacity for communities to pull themselves out of poverty" (Apgar and Douthwaite 2013;Douthwaite et al. 2017a: 295). Operating across five countries (Bangladesh, Cambodia, Philippines, Solomon Islands, and Zambia), among the key emphases of the program were to use a participatory action research approach, a gender transformative approach that aimed at challenging inequitable gender norms, and an underlying philosophy of "research in development", which aimed to contrast with more conventional research 'for' development approaches alluded to above (Douthwaite et al. 2017b: esp. Figure 2). Originally intended to operate for 12 years, the program ended ahead of time in 2016 following significant funding cuts to CGIAR and negative external evaluations that were conducted according to conventional approaches for measuring impact (Douthwaite et al. 2017a). ...
... Although the AAS program shut down, in many of the locations where AAS worked, communities and partners continue to use the knowledge and skills they acquired to shift mindsets and improve the performance of conventional agricultural research programs (Douthwaite et al. 2017a). Among the documented benefits of this approach included significant levels of ownership by local stakeholders in the research program, identification of new opportunities that emerged over time, and the ability to incorporate these findings into the research process in an iterative manner (Douthwaite et al. 2017b). Researchers were able to develop critical analyses of governance that addressed difficult, intractable problems such as representation, authority, and accountability (Ratner et al. 2013;Apgar et al. 2017a). ...
Article
Organisations working on conservation and community-based natural resource management (CBNRM) projects with communities have sometimes damaged the wellbeing of those communities. The social and political dynamics between organisations funding or implementing projects and the communities in which they work might be a factor causing this damage. This review paper explores the literature for evidence of and methods for evaluating impacts on community wellbeing from social relations in conservation and natural resource management projects. We found 101 papers addressing social connections in the human wellbeing-conservation nexus, acknowledging the damage done by colonising project relations and detailing proposals for or examples of more equitable relationality, and also evaluations of social equity in conservation/CBNRM work. However, we found few explicit evaluations of how the social, economic, and political relations of projects impact the wellbeing of participating communities. We call on researchers to address this gap, especially those working in evaluating project outcomes. To advance this agenda, we present literature that sheds light on what more equitable project relations look like, and how project relationality might be evaluated. We finish with ideas for how organisations can diagnose internal relationality problems likely to affect project outcomes, and how to transform those.
... However, despite the rhetoric and wide use of the terms food or agrifood systems in recent debates and mission statements (CGIAR, 2020;FAO, 2018), as well as practical attempts to incorporate complexity thinking into research and development Ramalingam, 2015), complexity remains poorly understood and has yet to inform the emergence of a distinctive body of development practice (Cholez et al., 2023;Douthwaite and Hoffecker, 2017). Challenging the "orthodoxy in much of mainstream research and evaluation practice" (Mayne et al., 2017), complexity-aware approaches remain poorly implemented in interventions (Foran et al., 2014;Hambloch et al., 2022). ...
... Table 1 presents the sources of complexity and how mainstream approaches (and earlier framings of interventions) usually engage with them, based on the review of the literature conducted above. Then, it compares this "orthodox" way of dealing with complexity (Mayne et al., 2017) with the principles emerging from an agri-food system perspective as illustrated by (Hertz et al., 2021;Thompson et al., 2007;Thompson and Scoones, 2009). ...
Article
CONTEXT Complexity has long been recognised as a key feature of agri-food systems. Yet, it remains largely theoretical or poorly addressed in practice, hampering the potential of international development projects to address agriculture and food-related challenges in the Global South. OBJECTIVE The paper identifies and examines six sources of complexity that can manifest in projects, namely: unpredictability; path dependencies; context-specific dynamics; power relations; multiple temporal and spatial scales. It then proposes and tests six agri-food system principles that could be drawn upon to more successfully navigate this complexity. The aim of the paper is to illustrate how these principles could help projects respond to the changing circumstances and unpredictable turns of agri-food systems contexts in a different way, which flexibly embraces complexity. This flexibility is essential in an age of uncertainty and transformation. METHODS Comparative case study analysis of six projects implemented by the CGIAR: aflatoxin control in groundnuts in Malawi (1), pigeonpea in Eastern and Southern Africa (2), sorghum beer in Kenya (3), sweet sorghum for biofuel in India (4), precooked beans in Uganda and Kenya (5), Smart Foods in India and Eastern Africa (6). The projects aimed to either increasing smallholder farmers' incomes or addressing food and nutrition security, or both. They were specifically selected as all they were affected by some of the sources of complexity, which hampered the projects to different extents. This makes the cases relevant for not only illustrating manifestations of complexity, but also help reflect on alternative strategies to tackle it. RESULTS AND CONCLUSION The analysis of the case studies reveals how complexity can frustrate objectives of development interventions under several aspects. It also serves to discuss how complexity can be more successfully navigated (within but also beyond the selected cases) by applying the set of proposed agri-food system principles. The principles are also presented as ways future interventions could avoid clinging to what is “known to work” and instead venture into more powerful pathways of change. SIGNIFICANCE The following complexity-aware principle are proposed: Welcome surprises and openly discuss trade-offs; Shun orthodoxies; Engage with context-specificity; Expose patterns of power; Embrace the lengthy nature of change; Understand the multi-scale (in terms of space and time) nature of agri-food systems contexts. These principles could be used by project designers and implementors to cope with the complexity and uncertainty that will inevitably be encountered in agri-food system interventions, and can no longer be ignored.
... Theory of change models have been used to describe and evaluate complex aspects of interventions in other contexts (Douthwaite et al., 2020;Rogers, 2008). Benefits of engaging in the development of theory of change models include articulating assumptions of intended pathways and providing a mechanism for discussions of additional pathways (Douthwaite et al., 2017). Implementation of a theory of change model may, for example, generate data collection instruments suitable for various stakeholders as part of a multimethod evaluation (Cooksy et al., 2001). ...
... A complexity-informed evaluation approach was essential for generating evidence for our CTL self-study and to more accurately demonstrate the complex impact of our ED services as part of a required quality assurance review. Conceptualizing ED services as a complex whole operating within dynamic conditions means causal links between intervention and impact are inherently uncertain and emergent (Douthwaite et al., 2017;Patton, 2011), and that we must therefore evaluate complex outcomes as greater than the sum of their individual components. Our approach to the evaluation of complex outcomes of ED work is thus unique in three ways: 1) It serves multiple reinforcing purposes for enhancing, measuring, and showcasing ED outcomes useful for multiple audiences: centre, institution, and external reviewers. ...
Article
Full-text available
Increasing instructional quality in higher education is a key goal of educational development (ED) work, yet demonstrating complex outcomes remains challenging and lacks practical guidance. Evaluating ED services often relies on a reductionist approach characterized by linear assumptions of causal pathways to measure the extent to which instructional outcomes have been achieved and uses proxies such as short-term participant satisfaction. This paper advances a complexity-informed approach for guiding the evaluation of complex outcomes of ED services across individuals and activities within institutions that is adaptable across institutional contexts. To do this, we position the need for innovation in evaluation approaches within current ED literature and practice, and outline key implications of four complexity principles for guiding our approach. Then we describe an iterative process for developing and implementing the evaluation approach within a larger Centre for Teaching and Learning self-study. We describe the transferability of the evaluation approach to contexts beyond the study, and conclude with theoretical, practical, and methodological implications for evidence-based decision-making and strategic planning of ED work.
... In our application of impact pathways for ex ante evaluation, the strategies to achieve the outcomes are identified by addressing the existing obstacles and opportunities related to these changes (Blundo Canto et al., 2020). The literature contains reflections on the application of this approach to agricultural research for development (Blundo-Canto et al., 2023;Douthwaite et al., 2017;Thornton et al., 2017). ...
Article
Full-text available
Evaluators widely utilize theory of change or impact pathway approaches to design and evaluate interventions. In research settings, there is increasing demand for ex ante impact pathways that are place‐based, coherent, and plausible, in order to foster more impactful interventions. Foresight approaches, which enable a collaborative, structured, and systematic way to anticipate, prepare, and affect the course of change, can help achieve this by exploring diverse future scenarios, the consequences of different assumptions, or how to overcome threats and leverage opportunities for change. Through three case examples, we show how foresight tools used prior to developing impact pathways can free participants from preconceived notions of the intervention's context, actors, and impacts, and thus support creative and systemic analyses of the future to rethink the present. They extend the analysis of how different actors can shape the future, identifying effects that might otherwise be overlooked or marginalized during the planning and evaluation of the intervention, while also helping to identify the conditions necessary for desired impacts. Nonetheless, their use entails evaluative judgments about the potential impacts to be prioritized in intervention design and evaluation. This is especially relevant in the case of research where scientific or technical aspects are usually prioritized. Finally, there is an entry cost to foresight tools for evaluators, in particular for the facilitation of strategic thinking about the future. However, we recommend them as a worthwhile addition to the evaluator's toolbox to broaden the scope of design and evaluation.
... (Pawson & Tilley, 1997). The evaluator seeks to identify the underlying mechanisms that explain 'how' outcomes were caused and the influence of context thereupon (compare with Douthwaite et al., 2017). • Differently, contribution analysis starts with an articulation of the theory of change, but this may be complex rather than simple. ...
... Traditional measures of success are largely quantitative (e.g. measuring transformation in terms of crop yields and productivity increases (IPES 2016)) continue to be used in determining how transformation is be tracked and evaluated (Douthwaite et al. 2017;Partidario 2020). These traditional measures and metrics while necessary valuable are alone insufficient to capture the complex and long-term nature of transformation (Caniglia et al. 2021;Conti et al. 2024b;Hambloch et al. 2022). ...
Article
Full-text available
System innovation is a signature feature of agri-food system transformation. Such system innovation often occurs in niches. However, how the "green shoots" of transformation can be detected and appraised through time remains ambiguous. This paper proposes, applies and tests a framework that could be used as a ‘transformation assessment tool’ to evaluate the level of system innovation in a domain of change. The framework is tested against a case study of a Non-Pesticide Management initiative in South India. The framework helps to reveal how, over 20 years, the initiative triggered a number of system innovations that opened a new development pathway, more aligned to environmental sustainability, equity and social inclusion. A critical enabling factor identified for the expansionand "blossoming" of this green shoot was its capacity to flexibly respond and adapt to emergent and largely unknowable agri-food systems dynamics. In its conclusions, the paper sheds light on the ongoing tensions around the defining benchmarks or thresholds for assessing the ‘transformativeness’ of initiatives and change processes. Finding a way of combining qualitative assessments of system changes with quantitative measures of social, economic, and environmental impact could be a valuable vein of research to enhance our understanding of transformative processes and how to enable them.
... Finally, the ToC process is limited by its lack of signifcant input or revision from local partnering agencies during the visioning and implementation stages. Tere is a need to develop more "complexity-aware" or "complexity-sensitive" ToCs and evaluation tools [47,48] that emphasize the interplay between complex systems and account for complexity through feedback loops or backward mapping approaches [49][50][51]. Additionally, uncertainty and emergence in complex evaluation settings necessitate the use of developmental evaluation methods [52], where both implementers and evaluators seek optimal solutions to strategize dynamic realities in complex systems, for example, utilizing ToC as both an outcome and a process. ...
Article
Full-text available
Refugee newcomers resettled in the United States face numerous challenges impacting their mental health and overall well-being. Despite extensive research on clinical interventions, multimodal interventions with peer-based group models in resettlement contexts remain scarce. Adopting a realist evaluation approach, this study aims to conduct a comprehensive implementation evaluation of a complex community-based mental health and psychosocial support (CB-MHPSS) group intervention, examining its mechanisms and processes while considering the interplay among context, implementing agents, and community settings. Qualitative and quantitative data on the implementation process were collected from 11 refugee agencies, involving trained resettlement staffers (n = 32) and refugee community facilitators (n = 31) who implemented the 31 CB-MHPSS psychosocial groups in 2021. The analysis included fidelity reports, process reflections, and follow-up survey responses, utilizing a structured template based on the CB-MHPSS Theory of Change (ToC) and Medical Research Council (MRC) guidance. Results of an in-depth exploration of complex relations between context and implementation processes reveal the vital role that adaptability to socio-ecological circumstances during the global coronavirus pandemic played in achieving successful implementation of psychosocial group interventions. Striking a balance between fidelity and refinement of intervention of the intervention curriculum emerged as crucial factors for aligning facilitation goals with newcomer needs. This approach maintained the integrity and appropriateness of the intervention, requiring implementers to utilize local knowledge of community resources and their understanding of lived experience of forced migrants and their within the community. Furthermore, exploring intervention barriers and facilitators revealed merits aligned with program change theories and areas for adjustment, while cataloged guidelines addressed community needs, showcasing competence beyond standard checkboxes. Strategic dissemination guidance provided by the central office in a supportive and nonintrusive manner facilitated uptake in a participatory and context-specific manner. This study emphasizes the importance of leveraging community partnerships and local knowledge to result in sustained improvements in the mental health and well-being of refugees.
... Based on the stages of knowledge utilization (Landry et al., 2001), it is developed as follows: (1) Recognition: whether the results produced by SE increase the awareness of the related problem and the need for action (Wiek et al., 2014;Fritz et al., 2019;Tribaldos et al., 2020;Chambers et al., 2021); (2) Discussion: whether the results produced by SE is taken up into dialogue, even used to produce new knowledge (Polk, 2014;Wiek et al., 2014;Luederitz et al., 2017;Fritz et al., 2019); (3) Test: whether the results produced by SE is used to test, including scenario analysis (Schneider and Rist, 2014); (4) Application: whether the results produced by SE is used to change processes and apply as a technology (Polk, 2014;Wiek et al., 2014;Luederitz et al., 2017;Fritz et al., 2019). (5) Continuation: whether the results produced by SE include long-term adoption or changes, even follow-up or take-over by other institutions (Douthwaite et al., 2017;Wyborn et al., 2019). Since the results of SE may have multiple different depths, each degree is multiplied by a weighted factor and merged to form the final depth of results (Jahn et al., 2022). ...
Article
Full-text available
Energy transition is one of the demands for resource management towards sustainability. It is increasingly recognized that this transition would not justly happen without effective stakeholder engagement (SE). Assessing the impact of SE and its relationship with SE configuration is imperative for understanding dynamics that may promote or hinder stakeholders' contribution to the transition. This paper systemically and quantitatively investigates whether and how the configuration of SE affects its impacts with globally scattered 1119 cases of SE in resources management including clean energy. It was found that 1) most participation with minimal or no meaningful impact in terms of both depth and scale; 2) the phase element in configuration, especially participants involved in a more empowered stage, have the strongest influence on the impacts. As a relatively recent phenomenon, stakeholders in clean energy governance were often involved in the early planning and designing phase, far from being impactful. 3) the purpose, participants, and methods had a less prominent impact. These findings can inform SE in the energy transition by configuring large-impact and high-justness SE elements.
... Realist informed approach to evaluate complex interventions (Douthwaite, Mayne, McDougall, & Paz-Ybarnegaray, 2017) Evaluation to lead to learning and accountability, evaluation to be a part of the project approach for impact, build theory for evaluation to bring change Projects aimed at improving people's lives work in complex systems; there is a need for developing complexity-aware project monitoring and evaluation. ...
Thesis
Full-text available
How a project is perceived by its stakeholders affects how it is implemented, and how the outcomes of the project are interpreted by the stakeholders influences the impact project can have on those stakeholders. Often this diversity of perspective is considered an impediment to the effectiveness of the project in meeting its goals. Standard project evaluation techniques dependent on linear and conventional methods to assess and present outputs and outcomes from projects fail to consider the complexity in projects. Complexity in a project arises from the involvement of multiple stakeholders from diverse disciplines, backgrounds, and geographies, and having varied perceptions, expectations, and understanding of the project and its aspects. The overall aim of this PhD is to improve the understanding of evaluation of complex projects by studying the projects from the perspectives of the multiple stakeholders involved in them. The first objective is to explore and understand the approaches to evaluation drawing on perspectives from literature, and observations from the field. The second objective is to understand the perspectives of stakeholders operating at various levels of a complex project on different aspects of the project such as its nature, approach, outputs, and outcomes. The third objective is to relate outcomes at various levels in the project to processes used, as well as associate outputs with outcomes. The fourth objective is to develop an integrated approach to evaluate complex multi- stakeholder projects, which enhances a project’s outcomes and enables learning for the stakeholders involved. With the aim of improvement in the existing knowledge on evaluating complex projects, the methodological approach is developed from a combination of theories and practices on evaluation. Central themes of the methodology are methodological pluralism, multiple perspectives, systems thinking, and appreciation and learning. To facilitate flexibility in navigating through a variety of theories and perspectives to enable both change and enhancement, the PhD is undertaken and presented as an action research. Three complex projects with stakeholders from diverse backgrounds and disciplines are examined in two stages of this thesis. These projects situated on the Chotanagpur Plateau in India with different intervention areas are, i) an agricultural research for development (AR4D) project, ii) a project to develop the skills of community youth to impart education, and iii) a Corporate Social Responsibility initiative. Data are collected from 82 project participants chosen by purposive sampling in the form of narratives, through semi-structured questionnaires. Findings from examining multiple perspectives were similar across the three studied projects. Stakeholders interpreted the nature and outcomes of the project uniquely. This study confirmed the existence of diverse stakeholder perspectives that were not captured or acknowledged in the evaluation of the three projects. These perspectives, however, were important for the stakeholders in how they identified with the project, how they functioned in it, and eventually, how it impacted their lives. Moreover, largely, there was no cognisance of this diversity in the stakeholders of the project. In instances where the stakeholders were aware of the multiple views, there was no mechanism for interaction of, or sharing those perspectives. Neither did the project stakeholders learn to acknowledge and work with varied perspectives, nor did they learn from multiple views in the project which were different from theirs. Besides the standard outputs and outcomes from the project, the project stakeholders outlined long-term personal changes. In particular, the learning which they underwent was considered profound and significant. The subtle shifts in learning and development of capabilities in project stakeholders were capabilities that enhance their sense of agency and change their worldviews, which they may further utilise to impact the project, themselves, and others. In considering these findings and addressing the challenge of incorporating complexity in project evaluation, the thesis develops a framework to evaluate complex projects. The framework is complexity-appreciative which acknowledges, appreciates, and integrates multiple perspectives in the design and evaluation of projects. Evaluation frameworks are always dependent on the contexts in which they are applied, and on those who design and use them, and the kind of boundary judgements they make. Hence, the framework provided in this PhD is not a tool to be used at the end of a project to measure its outcomes; rather, it is a process that must be part of a project from inception as a feedback tool to enhance outcomes. The framework can become a means to create spaces and processes in a project to enable stakeholders to share perspectives, listen to others, understand the diversity in the project, and acknowledge, appreciate and learn from each other’s perspectives as well as each other’s process of learning. Such a space will also allow stakeholders to find their voice and purposes in the project, to help each other do the same, and to further develop those purposes
... R4D programmes that involve multiple disciplines and stakeholders pose challenges for applying randomized control trials in their evaluation, because interventions occur in multiple emergent change processes, which are not always replicable Hughes 2020, Adler et al. 2018). Consequently, qualitative and theory-based approaches to evaluation that trace the contributions of research activities, outputs, outcomes, and impacts have gained traction in evaluations of R4D , Douthwaite et al. 2017. Similar concerns have been raised in public health, and both research fields now use more qualitative as well as theory-based approaches to evaluation that combine different methods depending on the context of meeting a variety of evaluation needs Hughes 2020, Jagosh 2019). ...
Article
To address complex societal problems, transdisciplinary approaches are increasingly being employed in research to achieve both scientific and societal effects. Comparing experiences of different impact evaluation approaches enables mutual learning across research fields. We provide an overview of the key characteristics of different approaches to assess the impact of transdisciplinary research across the fields of public health, development, and sustainability; uncover commonalities and challenges in applying these approaches; and suggest how they can be overcome by drawing on examples from specific approaches and fields. We find commonalities in terms of conceptual framing as well as data collection and analysis from which we derive the following key challenges: 1. evidencing causal claims, 2. including multiple perspectives on effects, and 3. sustaining continuous monitoring and evaluation. We conclude that impact evaluation of transdisciplinary research must capture the interplay and effects of multiple actors, processes, and impact pathways to promote learning and empirical rigour and suggest how funders can support this endeavour.
... Such methods often rely on systematic analyses of the theory of change and the impact pathway(Bamberger & Mabry 2019).Qualitative comparative analysis (QCA) is a methodological approach to contribution analysis that examines patterns in the data to identify necessary and sufficient conditions for relationships between interventions and outcomes without performing any tests of statistical significance(Ragin 2008;Pattyn et al. 2017). Other methods, such as "most significant change" or "outcome harvesting", support the reconstruction of the impact pathway to identify and assess policy outputs, outcomes and impacts by considering the contributions of multiple stakeholders, programmes and contextual factors(Alvarez et al. 2010;Blundo-Canto et al. 2017;Douthwaite et al. 2017). To trace the trajectory of outcomes, qualitative data-collection methods include key-informant interviews, focus groups, ethnographic research, or simulation games with participant observation, among others(Stern 2012;Hennink et al. 2020).Qualitative research also allows for inductive analysis (i.e. to go from specific cases to the general) and for generating hypotheses regarding the underlying mechanisms of policy impacts. ...
Article
Full-text available
The goal of this paper is to provide insights into how scientific evidence can be used for policy making and to put evidence-based agriculture and food policies at the top of research and policy agendas. We illustrate how scientific evidence can be used in a targeted manner for better policy making and present an overview of the rich set of ex-ante and ex-post evaluation methods and tools that agricultural economists use for evaluating agricultural policies to provide evidence for policy decisions. We present insights into both established and new/emerging methods and approaches, including their advantages and disadvantages, and discuss their potential use for policy evaluation. We also discuss how methods and approaches should be combined and could be better targeted towards decision makers. The paper also discusses the crucial role of high-quality data in supporting the science–policy interface. Finally, we present an overview of papers in this special issue titled ‘Evidence-Based Agricultural and Food Policy: The Role of Research for Policy Making’.
... We assume a theory-driven Realist-informed perspective, similar to that described in Douthwaite et al. (2017). ...
Article
Full-text available
Influencing policy is an important scaling mechanism. However, if a program is to plausibly claim that it has or can influence policy, it needs to explain how. This is not straightforward because of the complex nature of policy change. Scholars suggest the use of theory to help answer the ‘how’ question. In this article, we show how, in practice, a middle-range policy change theory—Kingdon’s Policy Window theory—helped us model the workings of four outcome trajectories that produced agricultural policy outcomes in four cases. By providing a common framework, the middle-range theory helped accumulate learning from one evaluation to the next, generating specific and generalizable insights in the process. Accumulation learning in this way can help organizations become more convincing in the proposals they write to donors, more accountable and better able to identify and deliver on their goals.
... Douthwaite et al., 2017;Wyborn et al., 2019) Transfer of Results(Transfer) The effect of transfer includes the transfer of learning from the project to another geographical or thematic focus.(Fritz et al., 2019;Luederitz et al., 2017) ProductsAcademic Outputs (O_Academic)Academic outputs include publications, presentations at conferences, or other ways of spreading produced knowledge through academic channels.(Mitchell ...
Article
Full-text available
Transdisciplinary research (TD) integrates knowledge from different scientific disciplines, as well as from research and practice. Research and practice therefore describe TD as well-suited for addressing complex sustainability challenges. However, the effects of TD on sustainable development are difficult to assess, as such projects produce manifold, interconnected effects through nonlinear processes, contingent on different contexts. In this article, we use a systematic literature review of 101 TD projects to assess the different effects of TD projects and their interconnections. We distinguish between North-South TD projects and TD projects within the global North. Due to differences in terms of historical development and context, we expect to observe differences in the effects they achieve. We find that North-South projects scored higher for societal effects and uptake of knowledge, while projects in the global North produced more tangible outputs, such as academic publications. In terms of interconnections of effects, N-S projects emphasize inclusion more strongly than global North projects, due to an increased awareness of differences between different project participants. However, effects related to uptake of knowledge, learning, and societal effects are often interconnected in both types of projects. This article improves our understanding of the prominence of different effects of TD projects, the interconnections between effects they produce, and the differences between N-S and North projects. Acknowledging this diversity of effects is important, not least for evaluating the efficacy of TD projects.
... 6 We seek to build upon some recent, realist contributions in the literature. Douthwaite et al. (2017) use realist synthesis and other techniques to develop design principles for monitoring and evaluation for complex interventions. ...
Article
Full-text available
Billions of dollars continue to be provided in foreign aid each year. However, few scholarly studies have examined whether the outcomes from foreign aid interventions are sustained after donor funding has ceased. This paper examines current approaches to assessing this issue before arguing that a realist evaluation approach is ideally suited to understand why and how sustained outcomes are—or are not—achieved. It contributes to the existing literature by presenting three new frameworks to examine the sustainability of outcomes in international development as well as some Context‐Mechanism‐Outcome statements. Implications for governments, communities, households and donor/implementing organisations are discussed.
... Our response to the combination of internal diversity and unpredictability was application of complexity-aware evaluation approaches (e.g. Douthwaite et al. 2017;Patton 2010;. Such approaches argue for regular revisiting of assumptions about how change is unfolding (though real time feedback loops) coupled with the use of goal independent evaluations that capture change as it emerges rather than through tracking predetermined indicators. ...
... Understanding this moderating influence of context is a key challenge that has been faced and addressed by evaluators of programs and interventions, including collaboration as an intervention, in particular in the field of realist evaluation (Pawson and Tilley 1997). Such evaluation recognizes that interventions are embedded in open, complex systems characterized by dynamic, non-linear interactions and uncertainties (Douthwaite et al., 2017). This approach is well suited to research on collaboration in socialecological systems as in the case studies we present here. ...
Article
Full-text available
The increasing scale and interconnection of many environmental challenges – from climate change to land use – has resulted in the need to collaborate across borders and boundaries of all types. Traditional centralized, top-down and sectoral approaches to governance of single-issue areas or species within social-ecological systems often have limited potential to alleviate issues that go beyond their jurisdiction. As a result, collaborative governance approaches have come to the forefront. A great deal of past research has examined the conditions under which collaborative efforts are likely to achieve desired outcomes. However, few studies have analyzed how the means to achieve successful collaborative outcomes differ based on context when examined across multiple studies. In this research, we begin to chart a means for doing this. Building onto a Context-Mechanism-Outcome (CMO) Framework, we provide a coding manual to analyse how contextual variables mediate the effects of mechanism variables on outcomes of the collaborative governance of social-ecological systems. Through the examination of four cases, we provide a proof-of-concept assessment and show the utility of the CMO framework and coding manual to draw comparisons across cases for understanding how collaborative outcomes are contingent on the social-ecological context in which they occur.
... These programmes anticipate health-related changes in varying (social) settings, which by definition involves the interaction of the initiatives with varying contexts at multiple levels and involving multiple stakeholders. Complexity is often perceived as a challenge when it comes to evaluation (Pawson, 2004;Stame, 2004;Douthwaite et al., 2017;Moore et al., 2019); however, complex systems potentially also comprise a lot of relevant information about health-related change in social settings. This could be assessed through the design and use of suitable evaluations. ...
Article
Full-text available
Under the umbrella of the Healthy Futures Nearby programme, 46 small-scale projects were funded to promote changes in health-related behaviours (smoking, alcohol, diet and exercise) and to improve perceived health among vulnerable families in the Netherlands. The evaluation of these health-related multiple project programmes is often based on funder-defined outcomes and strategies. However, within the funded projects, assumptions about improving the health of vulnerable families based on local knowledge and experiences will also shape the project outcomes and strategies. These additional outcomes and strategies are project-specific interpretations of effective health promotion. Knowing these interpretations is crucial for the policy related and scientific relevance of the evaluation. Therefore, we aimed to determine the interpretations of each project and how they translate into relevant inputs for the overall evaluation of the programme. Based on 46 semi-structured group interviews with local project stakeholders, we produced a list of assumptions about what health promotion for vulnerable families should look like and then identified five main clusters: (i) strategies of offering pre-defined, health (behaviour)-related activities to families, (ii) actively involving vulnerable families in the initiative, (iii) assumptions about how health promotion should start with or include non-health-related topics, (iv) assumptions on how one should build on what already exists in the local context of the families and (v) assumptions on the role of the (health) professional in health promotion among vulnerable families. These project interpretations of effective health promotion provide inputs and priorities for the HFN programme’s overall evaluation.
... However, evaluations focusing on the attributes of technologies and adopters (Wigboldus et al., 2016) and adoption by numbers (Woltering et al., 2019) are common in agricultural research (Maredia et al., 2014). One answer to this challenge comes from complexity science, which is increasingly promoted as an appropriate approach for the analysis of innovations supported by research in dynamic and adaptive systems (Douthwaite et al., 2017;Faure et al., 2018;Joly et al., 2015). A complexity-informed approach examines the diversity of actors, interests, roles, the changes they bring about and/or experience in their environment, as well as the contextual factors that enable these changes. ...
Article
Full-text available
In the West African Sahel, climate variability and climate change pose huge challenges to food security and health, particularly for poor and marginalised population groups. Co-production of actionable climate information between scientists and users has been advocated to increase its use in adaptation to climate change. Consequently, Weather and Climate Services (WCS) co-production models have been expanding, but there have been few evaluations of their effects, and those that exist mostly focus on the end user. The empirical contribution of this paper is an evidence based evaluation of the scaling of WSC co-production models and its enabling factors. The methodological contribution is a systemic and iterative evaluation method involving multiple analytical tools. The scaling of WCS in Senegal involved at least 161 actors and resulted in five axes of transformation: 1) continuous improvement of WCS, 2) emergence and consolidation of WCS facilitators, 3) inclusion of WCS in action planning, 4) active mobilisation to sustain WCS scaling, and 5) empowerment of actors. New users and uses emerged beyond agriculture, involving the fisheries, water and energy sectors, producing changes in institutional communication strategies, operational planning, and in coordination between actors. Enabling factors for scaling included capacity strengthening, knowledge-sharing and action platforms, interaction opportunities , and financial and political support. However, reduced precision of forecasts over time is perceived. New challenges are emerging including improving delivery and finer grain information, getting the private sector involved, and building capacity and trust at a large scale, to keep pace with the increase in uses and users.
... This ubiquity renders the system outcomes unpredictable without a thorough understanding of systems elements and their nature of interactions. Studies of complex systems suggest that informed systems intervention is possible only if the elements of the systems are precisely mapped, and their interactions are measured (van Mil et al., 2014;Douthwaite et al., 2017). Developing this understanding is extremely challenging for smallholder agriculture that embodies huge systems diversity in terms of their elements, interactions and contexts within which such interactions take place (Tittonell et al., 2007). ...
Article
The shock of Coronavirus Disease 2019 (COVID-19) has disrupted food systems worldwide. Such disruption affecting multiple systems interfaces in smallholder agriculture is unprecedented and needs to be understood from multi-stakeholder perspectives. The multiple loops of causalityy in the pathways of impact renders the system outcomes unpredictable. Understanding the nature of such unpredictable pathways is critical to identify present and future systems intervention strategies. Our study aims to explore the multiple pathways of present and future impact created by the pandemic and “Amphan” cyclonic storm on smallholder agricultural systems. Also, we anticipate the behaviour of the systems elements under different realistic scenarios of intervention. We explored the severity and multi-faceted impacts of the pandemic on vulnerable smallholder agricultural production systems through in-depth interactions with key players at the micro-level. It provided contextual information, and revealed critical insights to understand the cascading effect of the pandemic and the cyclone on farm households. We employed thematic analysis of in-depth interviews with multiple stakeholders in Sundarbans areas in eastern India, to identify the present and future systems outcomes caused by the pandemic, and later compounded by “Amphan”. The immediate adaptation strategies of the farmers were engaging family labors, exchanging labors with neighboring farmers, borrowing money from relatives, accessing free food rations, replacing dead livestock, early harvesting, and reclamation of waterbodies. The thematic analysis identified several systems elements, such as harvesting, marketing, labor accessibility, among others, through which the impacts of the pandemic were expressed. Drawing on these outputs, we employed Mental Modeler, a Fuzzy-Logic Cognitive Mapping tool, to develop multi-stakeholder mental models for the smallholder agricultural systems of the region. Analysis of the mental models indicated the centrality of “Kharif” (monsoon) rice production, current farm income, and investment for the next crop cycle to determine the pathways and degree of the dual impact on farm households. Current household expenditure, livestock, and soil fertility were other central elements in the shared mental model. Scenario analysis with multiple stakeholders suggested enhanced market access and current household income, sustained investment in farming, rapid improvement in affected soil, irrigation water and livestock as the most effective strategies to enhance the resilience of farm families during and after the pandemic. This study may help in formulating short and long-term intervention strategies in the post-pandemic communities, and the methodological approach can be used elsewhere to understand perturbed socioecological systems to formulate anticipatory intervention strategies based on collective wisdom of stakeholders.
... Dans ce contexte ont émergé de nouvelles méthodes d'évaluation de l'impact de la recherche en agriculture et en alimentation sur la société et les écosystèmes, en mettant l'accent sur : (i) la multi-dimensionnalité de l'impact de la recherche (Joly et al., 2015a, b) ; (ii) la temporalité et la sérendipité des chemins d'impacts (Ruegg et Feller, 2003) ; (iii) la complémentarité entre des approches qualitatives et quantitatives de caractérisation des impacts (Donovan, 2011 ;Naudet et al., 2012) ; (iv) la compréhension de la dimension systémique des processus d'innovation (Spaapen et van Drooge, 2011 ;Mayne, 2012 ;Bornmann, 2013 ;Temple et al., 2016 ;Douthwaite et al., 2017) ; enfin (v) la participation des acteurs de l'innovation dans le processus d'évaluation (Wiek et al., 2014). La métaphore de « chemins d'impact » au centre de ce renouvellement de méthodes repose en soi sur deux idées principales : i) un enchaînement de causalité entre les inputs (moyens et connaissances) mobilisés pour mettre en oeuvre la recherche, les produits et résultats de cette recherche (outputs), les changements liés à l'appropriation des résultats (que nous qualifieront par la suite fréquemment d'outcomes) et les impacts sur des indicateurs de développement et ii) la diversité de ces enchaînements qui interagissent au sein d'un même processus d'innovation. ...
Article
Full-text available
In light of the evolution of issues and development models in the agriculture and food sector, research organizations are questioning the methods to assess the impact of research activities. Indeed, conventional methodological frameworks used since the 1950s are increasingly less adapted to societal expectations and realities of agricultural research. This thematic issue of Cahiers Agricultures. "Evaluation of the impacts of agricultural research on society: tools, methods, case studies" presents new methodological approaches developed in recent years by INRA, CIRAD and CGIAR. These methods build on common theoretical underpinnings that emphasize the systemic dimensions of the innovation process, multiple perspectives and longer temporality of impact generation. They also differ in certain dimensions, such as the level of participation of stakeholders in the assessment of impacts. One part of this thematic issue is dedicated to the empirical test of these approaches in developing countries. The issue is enriched by papers providing cross-cutting analyses of the case studies, or presentation of different methods and geographical contexts. Finally, some articles identify the limitations of these methods for future research. This synthesis article presents knowledge on the understanding of the impacts of research activities, and invites readers to reconsider the role of research in the innovation process.
... Whilst theory driven approaches to evaluation are not common within evaluations of European rural development initiatives, they are frequently deployed within the evaluations of, and used to develop, rural policy in developing countries (Nayiga et al. 2014;Breuer et al. 2016;Apgar et al. 2017;Douthwaite et al. 2017;Thornton et al. 2017;Higgins et al. 2018;Maru et al. 2018;Richardson et al. 2018). Others have shown how theory driven approaches can help evaluations of urban renewal projects (Mehdipanah et al. 2015) in order to reveal the complexity that formal evaluations struggle to accommodate (Stame 2004). ...
Article
Rural policy evaluation helps to understand the extent to which policies have met pre‐defined objectives, achieve value for money and learn from implementation failures. However, there is increasing debate over the quality of policy evaluation and the extent to which its methods can fully contribute to an understanding of rural policy. Responding to these calls, this paper employs a theory driven approach to policy evaluation to assess the social impacts of attempts to reduce animal disease on farms in England. Popular in other policy arenas, theory driven evaluation relies on developing a theory of change to examine the interactions between policy contexts and mechanisms and policy outcomes and determine what works for whom. Drawing on longitudinal qualitative and quantitative research, the paper identifies two mechanisms of change to evaluate the Badger Vaccine Deployment Project (BVDP) in England to reduce incidence of bovine Tuberculosis (bTB) in cattle. The papers shows how these mechanisms – ‘seeing is believing' and ‘practice similarity' – are triggered by different contextual factors leading to the failure to deliver expected policy outcomes. We also consider the advantages and limitations to theory based evaluation, and the contribution it can make to the evaluation of other rural development programmes.
... There is a growing consensus in the literature about the need to understand such cooperation processes as complex interventions in complex systems (Douthwaite et al. 2017;Mayne and Stern 2013;Pawson 2013). Indeed, these paths include different level of analysis (farm, operational group, supply-chain and sector levels) and timing for investigations and assessments. ...
Article
Purpose: This study explores the potential of the Farm Accountancy Data Network (FADN) to assess the technical, economic and environmental effects of cooperative innovation projects at farm level. Design/Methodology/Approach: The analysis of the FADN potential relies on the classification of innovations and the co-identification, through the Most Significant Change (MSC) approach, of the most relevant indicators for tracking and assessing the performances of innovations and attributing to them the effects of a specific innovation. Findings: The study shows that the FADN has a certain range of useful indicators and that they have a good coverage of the different types of innovation. Furthermore, the farm visits let emerge the importance of participatory approaches to capture the different changes and interplays occurred in farming processes. Practical implications: The paper lays the foundations for the adaptation of the current methodology for data collecting and provides useful insights concerning the overcoming representativeness claims, costs’ issues, and problems related to the observation time limit. Theoretical implications: The paper reveals the importance of participatory monitoring and evaluation approaches in helping the collection of more robust and relevant account data on farm, as well as in attributing certain results to a specific innovation and recognising synergies and side-effects of cooperative processes. Originality/Values: The paper provides some recommendations on how to enlarge the scope of the FADN survey in order to be used effectively in the analysis of the performance of cooperative innovations at farm level.
... There are increasing efforts to do evaluation which is 'complexity-aware', evident for example in USAID evaluation and monitoring guidelines (Britt, 2016) (Douthwaite, Mayne, McDougall, & Paz-Ybarnegaray, 2017). But in practice this awareness amounts to a minor sophistication of the idea of 'context' (Pawson & Tilley, 1997), recognising that the context contains interdependencies, is dynamic and is liable to change its behaviour. ...
Technical Report
Full-text available
This report was commissioned by the Swedish Agency for Growth Analysis to respond to the requirement to evaluate much larger and more complex research and innovation programmes than were tackled in the past, a requirement that is partly driven by the need to address the so-called ‘societal challenges’. The report could be of interest to policy-makers, as it analyses complex innovation programmes, such as national innovation strategies, and programmes that address the need to make transitions in socio-technical systems, such as removing fossil fuels from the electricity supply system.
... Middle-range theory is an important concept in Realist Evaluation (Pawson, 2013), located in between social science theories and program-specific ToCs. As such, middle-range theories apply to families of programs (Stachowiak, 2007) A monitoring, evaluation and learning system for the program to use in its basins was developed (Douthwaite et al., 2017a). Together with researchers from others systems CRPs, work was done on developing indicators and the conceptual for underpinning "capacity to innovate," (Leeuwis et al., 2014), a key intermediate outcome along the impact pathway for this research (see outcome 7 in Fig. 2). ...
Chapter
All research for development programs wish to achieve impact, but understanding how to plan for and document this has been challenging. One of the newest and most popular approaches is the use of theories of change (ToC). This paper looks at how ToCs can be used in agricultural research for development (AR4D) programs. ToCs have been widely used in evaluation of development programs. In this chapter, we will describe their use in international AR4D and in CGIAR.
... The theory of change is built on critical assumptions and relationships among various programmatic and non-programmatic factors that help in understanding how outcomes will be achieved [56]. Logical framework analysis, on the other hand, coins together "the strategic, planning and operational needs of the organization" in order to achieve outputs, outcomes, and impacts [57]. Our analysis points out that CSR companies usually do not have a well-articulated theory of change and logical framework analysis, which is indicative of them usually lacking a robust program design in their CSR initiatives. ...
Article
Full-text available
This study examines the continuum of sustainability and corporate social responsibility (CSR) policies, and analyzes broad patterns that have emerged with respect to monitoring and evaluation practices in the CSR programs of Indian companies under new CSR regulations. Under these regulations, the Indian firms are mandated to spend at least 2% of their profits on social and development sectors. We specifically analyze (i) how Indian companies have conceptualized the idea of sustainability in their annual sustainability reports, and how these ideas get reflected in their CSR policies, and (ii) the monitoring and evaluation practices in CSR interventions. The study uses both primary and secondary data sources, and employs text network analysis and narratives-based content analysis to analyze the data. We find that the conceptualization of sustainability is a largely rhetoric and customary exercise that does not take into account variations in firms’ businesses. This approach toward sustainability initiatives presents serious challenges to sustainability, including social sustainability. The study also finds that there is lack of ‘willingness’ and ‘readiness’ among Indian companies to measure and monitor the outcomes of CSR interventions, which is arguably one of the most robust ways to signal their commitment toward corporate sustainability. Although mandatory CSR spending is a recent phenomenon in India, our study establishes that it is only through the design of effective CSR policies that the best practices for Indian business community can emerge in the near future.
... Dans ce contexte ont émergé de nouvelles méthodes d'évaluation de l'impact de la recherche en agriculture et en alimentation sur la société et les écosystèmes, en mettant l'accent sur : (i) la multi-dimensionnalité de l'impact de la recherche (Joly et al., 2015a, b) ; (ii) la temporalité et la sérendipité des chemins d'impacts (Ruegg et Feller, 2003) ; (iii) la complémentarité entre des approches qualitatives et quantitatives de caractérisation des impacts (Donovan, 2011 ;Naudet et al., 2012) ; (iv) la compréhension de la dimension systémique des processus d'innovation (Spaapen et van Drooge, 2011 ;Mayne, 2012 ;Bornmann, 2013 ;Temple et al., 2016 ;Douthwaite et al., 2017) ; enfin (v) la participation des acteurs de l'innovation dans le processus d'évaluation (Wiek et al., 2014). La métaphore de « chemins d'impact » au centre de ce renouvellement de méthodes repose en soi sur deux idées principales : i) un enchaînement de causalité entre les inputs (moyens et connaissances) mobilisés pour mettre en oeuvre la recherche, les produits et résultats de cette recherche (outputs), les changements liés à l'appropriation des résultats (que nous qualifieront par la suite fréquemment d'outcomes) et les impacts sur des indicateurs de développement et ii) la diversité de ces enchaînements qui interagissent au sein d'un même processus d'innovation. ...
Article
Full-text available
In light of the evolution of issues and development models in the agriculture and food sector, research organizations are questioning the methods to assess the impact of research activities. Indeed, conventional methodological frameworks used since the 1950s are increasingly less adapted to societal expectations and realities of agricultural research. This thematic issue of Cahiers Agricultures "Evaluation of the impacts of agricultural research on society: tools, methods, case studies" presents new methodological approaches developed in recent years by INRA, CIRAD and CGIAR. These methods build on common theoretical underpinnings that emphasize the systemic dimensions of the innovation process, multiple perspectives and longer temporality of impact generation. They also differ in certain dimensions, such as the level of participation of stakeholders in the assessment of impacts. One part of this thematic issue is dedicated to the empirical test of these approaches in developing countries. The issue is enriched by papers providing cross-cutting analyses of the case studies, or presentation of different methods and geographical contexts. Finally, some articles identify the limitations of these methods for future research. This synthesis article presents knowledge on the understanding of the impacts of research activities, and invites readers to reconsider the role of research in the innovation process.
... This article's contribution aims to renew the pool of impact assessment methods for research activities (Må rtenssona et al., 2016) and, ultimately, gain support of national and international donors for the use of evaluation methods that embrace complexity (Douthwaite et al., 2017). Moreover, by gaining a better understanding of how to build desired outcomes to achieve impacts, researchers shall be better able to frame research questions, implement research protocols, and anticipate strategies to increase relevant interactions all along the impact pathway. ...
Article
Full-text available
Over the past decade, renewed societal demands on public research have been structured by various generic issues, while others are specific to the context of developing countries. In the first part of this article, after reviewing those issues, we examine how they reshape the analytical frameworks that structure the understanding of causal relationships between research activities, innovation processes, and the consequences of both for development. We used an impact pathway framework to assess innovation processes by looking at 13 case studies on research in agricultural and food sectors of developing countries. The results show the diversity of outcomes related to human capital, social capital, and knowledge infrastructure. Moreover, they show the systemic interaction between outputs, outcomes, and impacts. Based on this assessment, we demonstrate that the way impact pathways are framed and analysed needs to be improved to better consider the complex interactions between the diverse actors involved in innovation processes. Through a discussion of our results, we propose an analytical framework to help improve impact assessment methods for research activities.
... Even formative feedback recognizes that objectives may need to be modified for practice. The complexity and uncertainty of real-life initiatives with unclear objectives trying to meet multiple demands through many sites makes developmental evaluation especially appropriate (Patton, 2006(Patton, , 2011Gamble, 2008;Douthwaite et al., 2017). ...
Article
Full-text available
Developmental evaluation supports grant-funded initiatives seeking innovation and change. Programs born from aspirational grant language and plans often need guidance as they work toward creating workable models for social innovation. This article describes the challenge of designing and implementing complex programs and presents a case that illustrates how a program moves from proposal to practice. The Massachusetts Charter Public School Association Capacity Building Network is a complex intervention, funded by the US federal government, aimed at raising school capacity to serve students with disabilities and English language learners. Developmental evaluation served to bridge the gaps between an aspirational proposal, an ambitious and ambiguous program plan, and emerging practices to serve this population of students. Jointly reviewing our experience in this developmental evaluation, the evaluation team and a program director share important thematic lessons learned about the developmental evaluation approach.
Preprint
Despite a broad consensus on the necessity of fundamental change, endeavors to transform food systems appear to have reached an impasse. Greater engagement with the uncertainty of food systems could open up new ways of triggering transformation directed towards achieving more sustainable and inclusive outcomes. As a way of reorienting current food system change efforts to better embrace uncertainty, we propose a framework for a transformative learning system that serves two aims. First, the framework highlights the importance of locally led action, experimentation, and learning, providing a way of focusing on the core capacities and skills needed to act in the face of uncertainty. Second, it outlines the different types of learning functions that need to operate at different scales of food systems to trigger disruptive, coordinated, and more democratic change processes. The operationalization of this framework necessitates shifts in roles and ways of working across the landscape of food system interventions. The discussion will address the who and how of this potential change, as well as its subsequent impact on the operational modalities of individuals, the process of change itself, and the structures and institutions involved in the process. We argue that embracing uncertainty and the focus on learning has the potential to facilitate a more agile and locally relevant change process. This would allow actors to learn from decentrally pursued food systems reforms, leading to the emergence of diverse pathways that complement on-going efforts and potentially accelerate transformation efforts.
Article
Full-text available
In this concluding article, we take stock of the diverse and stimulating contributions comprising this special issue. Using concept mapping, we identify eight evaluation themes and concepts central to John Mayne’s collective work: evaluation utilization, results-based management, organizational learning, accountability, evaluation culture, contribution analysis, theory-based evaluation, and causation. The overarching contribution story is that John’s work served to bridge the gaps between evaluation practice and theory; to promote cross-disciplinary synergies across program evaluation, performance auditing, and monitoring; and to translate central themes in evaluation into a cogent system for using evaluative information more sensibly. In so doing, John left a significant institutional and academic legacy in evaluation and in results-based management.
Article
In recent years, the productive interactions approach has gained increasing prominence as a foundation for studying the socioeconomic impact of science. So far, however, it has been deployed primarily as a heuristic for analyses in the context of social sciences research at universities. The majority of impact studies of basic research infrastructures, in contrast, have remained focused on these facilities' long-term contribution to science, their role as economic agents or the learning effects induced through their collaboration with suppliers. This paper demonstrates that the array of productive interactions at basic research infrastructures can be substantially wider. First, it identifies a number of productive interactions relevant for this context based on workshops with diverse infrastructures' managements. Beyond qualified procurement, these include on-demand experiments, scientific collaboration with industrial partners, the provision of datasets, the development of technical equipment and external communication. Second, our survey of the DESY synchrotron's users reveals that productive interactions with external partners are prevalent and tend to be embedded in particular project types. Among such projects, we find those most associated with concrete impact that involve external partners in a formal manner.
Article
The complex ways in which food security actions lead to nutrition and other health outcomes make it important to clarify what programs work and how, with theory-driven evaluation emerging as a promising approach to evaluate complex programs. However, it is unclear how and why theory-driven evaluation is applied in food security contexts. Our objective is to examine the development and use of Theory of Change and Realist Evaluation to support food security programs globally. Using a systematic search and screening process, we included studies that described a food security program, used a Theory of Change or Realist Evaluation, and presented original research or evaluations. We found a total of 59 relevant Theory of Change studies and eight Realist Evaluation studies. Based on our analysis, Theories of Change arose in response to three main problems: 1) the need to evaluate under complexity; 2) challenges with evaluation; and, 3) information gaps surrounding a program. In contrast, Realist Evaluation was reported to be developed primarily to understand a program's outcomes. Reflecting on the problem to be addressed in the evaluation would help improve understandings of the evaluation context, which would then inform the choice and design of an evaluation approach.
Article
Full-text available
Facilitated learning approaches are increasingly being used as a means to enhance climate and sustainability collaborations working across disciplines, regions, and scales. With investments into promoting and supporting inter-and transdisciplinary learning in major programs on complex global challenges like climate change on the rise, scholars and practitioners are calling for a more grounded and empirical understanding of learning processes and their outcomes. Yet, methodologies for studying the interplay between learning and change in these initiatives remain scarce, owing to both the "hard to measure" nature of learning and the complexity of large-scale program implementation and evaluation. This paper proposes a new method for studying social learning in the context of large research programs. It aims to analyze the social learning of researchers and practitioners engaged in these programs and assess the contributions of this learning to the resilience of the natural and social systems that these programs seek to influence. We detail the theoretical basis for this new approach and set out six steps for developing multi-layered contribution pathways and contribution stories with stakeholders to document both the process and outcomes of social learning. The proposed method, we argue, can strengthen our analytical capacity to uncover the structural drivers and barriers to social learning that are often masked by the complexity of large-scale programs. An illustrative example, drawn from a large-scale climate adaptation research program, provides evidence on how this method might advance our methodological strategies for studying learning in these programs. We conclude by highlighting two key methodological contributions brought about through this approach, and by reflecting on opportunities for further methodological development. Enriching our understanding of learning and change processes, we argue, is an important avenue for understanding how we can pursue transformations for sustainability.
Article
Full-text available
Inclusive innovation as a strategy for inclusive development has received increased attention from development policymakers, practitioners, and scholars in recent years. What these processes entail in practical terms, however, remains contested and under-theorized. This paper addresses the scarcity of mid-level analysis and models of inclusive innovation processes within complex systems, which are needed to enable a coherent empirical research agenda and to inform program theory-building, implementation, and evaluation. Looking to smallholder-oriented agricultural systems in the Global South, where the majority of inclusive innovation implementation and research has been located, this paper proposes that it is possible to identify the essential features and causal logic of these processes to create an empirically-derived, middle-range model with cross-context applicability. Drawing on methods from realist evaluation and social inquiry, I conducted a theory-driven, cross-case synthesis of three studies of inclusive innovation processes in agricultural systems, with one case each from South America, Southeast Asia, and Africa. I find that despite significant diversity in project designs, facilitation approaches, and local contexts, the three inclusive innovation processes unfolded in strikingly similar ways, and that this modus operandi can be modeled as a middle-range theory of change. In each case, I find that a consistent set of activities and processes changed the local context for the inclusive innovation initiative. These altered contextual factors interacted with ongoing programmatic activities in consistent ways to trigger processes of social learning, social capital strengthening, collective cognition, and consensus formation, which acted as causal mechanisms responsible for producing the intermediate outcomes that led to technical, organizational, and institutional system innovation. The middle-range model enables cross-context insights into how inclusive innovation processes work and what capacities are needed to facilitate them. It can also guide the adaptive management and assessment of these processes, while offering testable hypotheses to guide future empirical work and evaluation.
Chapter
This Chapter applies the lens of ‘public value’ to governance reform within the British context. To frontload subsequent debates within the book, the chapter addresses key themes and issues relating to public value, governance and reform in the British context. It starts by considering new public management (NPM) in the context of British governance and then focuses on major themes associated with public value, including complexity, networks, outcomes, evaluation, co-production and trust. The chapter ends by discussing key issues with regards to measuring and establishing public value. Overall, the chapter calls for analyses of governance reform to avoid over-claiming that NPM is over; rather change is incrementalism, but not unidirectional; and that cumulative changes over time expand our understanding of NPM ideas, rather than replace them.
Article
Full-text available
The concept of Theory of Change (ToC) is well established in the evaluation literature, underpinning substantial research and practice eff orts. However, its ability to facilitate learning has been increasingly debated. The objective of this paper is to identify, characterize, and evaluate concerns over the use of ToCs based on a review of relevant studies. Seven concerns are found: distinguishing ToCs from other evaluation approaches, conceptual vagueness, underdeveloped ToCs, under-contribution to theoretical knowledge, uncertainty in stakeholder engagement, neglecting context, and overlooking complexity. Priority areas for improvement include integrating context and complexity throughout the ToC process, contributing theoretical knowledge, and engaging stakeholders as appropriate.
Article
Full-text available
There are limited approaches available that enable researchers and practitioners to conduct multiple case study comparisons of complex cases of collaboration in natural resource management and conservation. The absence of such tools is felt despite the fact that over the past several years a great deal of literature has reviewed the state of the science regarding collaboration. Much of this work is based on case studies of collaboration and highlights the importance of contextual variables, further complicating efforts to compare outcomes across case-study areas and the likely failure of approaches based on one size fits all generalizations. We expand on the standard overview of the field by identifying some of the challenges associated with managing complex systems with multiple resources, multiple stakeholder groups with diverse knowledges/understandings, and multiple objectives across multiple scales, i.e., multifaceted collaborative initiatives. We then elucidate how a realist methodology, within a critical realist framing, can support efforts to compare multiple case studies of such multifaceted initiatives. The methodology we propose considers the importance and impact of context for the origins, purpose, and success of multifaceted collaborative natural resource management and conservation initiatives in social-ecological systems.
Article
Full-text available
The basic ideas behind contribution analysis were set out in 2001. Since then, interest in the approach has grown and contribution analysis has been operationalized in different ways. In addition, several reviews of the approach have been published and raise a few concerns. In this article, I clarify several of the key concepts behind contribution analysis, including contributory causes and contribution claims. I discuss the need for reasonably robust theories of change and the use of nested theories of change to unpack complex settings. On contribution claims, I argue the need for causal narratives to arrive at credible claims, the limited role that external causal factors play in arriving at contribution claims, the use of robust theories of change to avoid bias, and the fact that opinions of stakeholders on the contribution made are not central in arriving at contribution claims.
Article
Full-text available
A common and driving assumption in agricultural research is that the introduction of research trials, new practices and innovative technologies will result in technology adoption, and will subsequently generate benefits for farmers and other stakeholders. In Lao PDR, the potential benefits of introduced technologies have not been fully realised by beneficiaries. We report on an analysis of a survey of 735 smallholder farmers in Southern Lao PDR who were questioned about factors that influenced their decisions to adopt new technologies. In this study, we have constructed measures or states of adoption which identify key elements of an adoption decision-making nexus. Analysis was conducted to statistically group explanatory factors of adoption. The key explanatory factors represented attributes of the farmer, the factors considered when undertaking production decisions and elements of the agricultural value chain that present as opportunities or constraints. We describe the combination of farmer’s personal attributes, perceptions of the value chain, and the introduction of new technologies by external actors as an “agricultural research value chain”, where agricultural research activities intervene to derive greater benefits for local farmers. A generalised linear model, via Poisson (multiple) regression analysis on the identified explanatory factors, was applied to explore how they influence adoption measures and we found several significant relationships.
Article
International research-for-development initiatives seeking to tackle complex problems present a range of challenges and responsibilities. Meeting standards of ethical research practice is one of these. While practitioners may identify as being committed to ethical practice, the reality is far more complicated. This situation is particularly apparent in agricultural research for development (AR4D). This article reflects on the changing research landscape before using the experiences of a unique partnership to demonstrate how the authors moved beyond compliance-focused tasks towards collective reflection and planning for a broad range of ethical challenges. It concludes with suggestions for integrating ethics into the planning and implementation of development initiatives.
Article
International development assistance is increasingly being seen as operating within complex adaptive systems. Evaluators have been developing new methods and approaches that are compatible with the dynamic and unpredictable realities of complex adaptive systems, which are composed of many separate but interacting agents and groups. However, these new methods are not easily implemented in a conventional commissioned end-of-program evaluation. This article builds on what is known about how complex adaptive systems’ properties affect program performance to propose eight evaluation questions. The findings from the questions will reveal if implementers were aware of and responded effectively to complex adaptive systems’ properties. The eight questions can be incorporated within a conventional evaluation to create a plausible narrative for program impact and inform the design and implementation of other programs.
Article
Full-text available
The Challenge Program on Water and Food pursues food security and poverty alleviation through the efforts of some 50 researchfor-development projects. These involve almost 200 organizations working in nine river basins around the world. An approach was developed to enhance the developmental impact of the program through better impact assessment, to provide a framework for monitoring and evaluation, to permit stakeholders to derive strategic and programmatic lessons for future initiatives, and toprovide information that can be used to inform public awareness efforts. The approach makes explicit a project's program theory by describing its impact pathways in terms of a logic model and network maps. A narrative combines the logic model and the network maps into a single explanatory account and adds to overall plausibility by explaining the steps in the logic model and the key risks and assumptions. Participatory Impact Pathways Analysis is based on concepts related to program theory drawn from the fields of evaluation, organizational learning, and social network analysis.
Article
Full-text available
Managing large-scale water resources and ecosystem projects is a never ending job, and success should be measured in terms of achieving desired project performance and not just meeting prescriptive requirements of planning and constructing a project simply on time and within budget. Success is more than studying, planning, designing, or operating projects. It is developing the right plan, getting it implemented, and seeing that it is operated and performs properly. Success requires all of these, and failing any of these results in wasted resources and potential for doing great harm. Adaptive management can help make success possible by providing a means for solving the most complex problems, answering unanswered questions, and, in general, reducing uncertainty. Uncertainties are the greatest threats to project success. Stakeholder support and political will are ultimately essential in achieving project success. Project success is often impossible to achieve if uncertainties persist. Resolving uncertainties quickly and efficiently facilitates the greatest forward progress in the shortest possible time. Uncertainties must be reduced or resolved to a sufficient level, not over-resolved or under-resolved. Over-resolving presents a value trade-off between additional knowledge and the cost of getting it. Under-resolving trades greater risks of failure for cost savings. Resolutional sufficiency varies from uncertainty to uncertainty, and applying risk-based logic is helpful in determining what is sufficient. Adaptive management can bring great efficiency and produce high returns on investment. Project-stopping uncertainties get resolved, and resources are spent wisely. Organizational governance must understand adaptive management and value it. Adequate time and money must be provided. Adaptive management must be integrated into other organizational processes such as project management and project delivery. Integrating adaptive management requires a new mind-set, individually and organizationally. The resources provided must be scaled, oriented, and tuned to meet the challenges being addressed.
Technical Report
Full-text available
Natural resource management research (NRMR) has a key role in improving food security and reducing poverty and malnutrition in environmentally sustainable ways, especially in rural communities in the developing world. Demonstrating this through impact evaluation poses distinct challenges. This report sets out ways in which these challenges can be met. NRMR combines technological innovation with real-world changes in agricultural practice that involve many stakeholders at farm, community, scientific and policymaking levels. These programs generally seek to integrate multiple inputs or interventions—scientific, institutional, human and environmental; engage participatively with beneficiaries and other implicated parties; and mobilise stakeholders, both to support innovative programs and to carry lessons learned into the future. Simple attribution of productivity and socioeconomic outcomes to NRMR interventions is difficult when NRMR itself is a ‘package’ of different actions adapted to diverse settings by farmers and other stakeholders, often over extended periods. This report outlines impact evaluation strategies that accept that NRMR is likely to be a ‘contributory cause’ rather than the sole cause of program results. It builds on recent reports that demonstrate that, in many development settings, impact evaluation should be seen as contributing to an adaptive learning process that supports the successful implementation of innovative programs. Change is nearly always the result of a ‘causal package’ and for an NRMR intervention to make a contribution it must be a necessary part of the package. This contrasts with an ‘impact assessment’ perspective that is mainly concerned with forms of accountability that measure and attribute impacts to particular programs or interventions. Starting from a learning perspective, impact evaluation still addresses accountability by demonstrating that NRMR programs make a difference by contributing to outcomes and impacts, and improve performance through continuous learning. The proposed evaluation strategy pays special attention to the causal links between NRMR programs and intended outcomes. As these programs are expected to produce generalised answers that can be replicated and scaled up to tackle global problems, evaluation also has to be able to explain why and under what circumstances programs are effective. This is why the proposed evaluation strategy includes approaches to explanation, and why theories of change are an essential part of the proposed approach. A theory of change both helps to unpick the assumptions about how programs bring about change and takes into account the way programs are implemented. Such a theory-based approach also allows programs to be tested against what is known from wider research literatures and, at the same time, allows evaluation results to contribute to these literatures. Against this background, an overarching evaluation framework is put forward that aims to answer impact evaluation questions by selecting appropriate evaluation designs that take into account NRMR program ‘attributes’ or characteristics. The report argues that, in a complex program setting, an evaluation must begin with appropriate evaluation questions that interest policymakers, donors and other stakeholders. Key evaluation questions should be about what difference the program is making (i.e. the contribution being made), about understanding the progress being made and why results are occurring, and about the learning that is taking place. This is distinguishable from the kinds of evaluation questions that are appropriate for more straightforward interventions such as: ‘Did our program cause the intended change?’ The evaluation questions to be considered are broader than those dealing solely with causality, and include questions of rationale and implementation, and of measuring results, in terms of both their sustainability and transferability. The report suggests a framework for defining evaluation questions that takes account of both the outcomes and processes of change, and tries to explain how change occurs in different settings and can be generalised or scaled up. A broad range of different evaluation designs and methods is considered, including theory-based, case- based and participatory approaches. However, although not specifically discussed in this report, more traditional approaches such as experimental and statistical methods are not dismissed—they will often be valuable as part of an overall ‘nested’ evaluation strategy. The attributes of NRMR programs also pose evaluation challenges and have consequences for impact evaluation design. These challenges and consequences are reviewed. For example, multi-stakeholder programs require methods capable of assessing collective action, and time-extended programs require iterative and longitudinal methods. The approaches laid out in the report have been ‘walked through’ and refined in relation to several specific programs including: the CGIAR Research Program on Aquatic Agricultural Systems, the CGIAR Challenge Program on Water and Food’s Ganges Basin Development Challenge, and the CSIRO–AusAID African Food Security Initiative. The report proposes a ‘general evaluation framework’ that would allow the evaluation design principles outlined to be turned into an overall operational plan, and suggests what activities are necessary to put together such a plan. It concludes with summary recommendations, appendixes giving sample evaluation questions and an example of a mixed methods statistical design evaluation, and details of literature cited.
Article
Full-text available
Theory-based evaluations have helped open the ‘black box’ of programmes. An account is offered of the evolution of this persuasion, through the works of Chen and Rossi, Weiss, and Pawson and Tilley. In the same way as the ‘theory of change’ approach to evaluation has tackled the complexity of integrated and comprehensive programmes at the community level, it is suggested that a theory-oriented approach based on the practice of realistic cumulation be developed for dealing with the vertical complexity ofmulti-level governance.
Article
Full-text available
In this introductory article, a brief history and introduction to contribution analysis is provided to lay the stage for the articles that follow. At the heart of contribution analysis is the aim to be able to make credible causal claims about the contribution an intervention is making to observed results. The key role that theories of change play is noted, and what a useful theory of change ought to contain is discussed. The article then makes a link between the philosophical discussions on causality and contribution analysis through a discussion of contributory causes. It is argued that such causes, which on their own are neither necessary nor sufficient, represent the kind of contribution role that many interventions play: where there are a number of other influencing events and conditions at work in addition to the intervention of interest. Contribution analysis is an approach to confirming that an intervention is a contributory cause.
Article
Full-text available
OCIAL SCIENCE HAS PROVEN ESPECIALLY inept in offering solutions for the great problems of our time—hunger, violence, poverty, hatred. There is a pressing need to make headway with these large chal- lenges and push the boundaries of social inno- v a t ion t o m a ke re a l prog re s s . T he ver y possibility articulated in the idea of making a major difference in the world ought to incorpo- rate a commitment to not only bring about sig- nificant social change, but also think deeply about, evaluate, and learn from social innova- tion as the idea and process develops. However, because evaluation typically carries connota- tions of narrowly measuring predetermined out- comes achieved through a linear cause-effect intervention, we want to operationalize evalua- tive thinking in support of social innovation through an approach we call developmental evaluation. Development al evaluation is designed to be congruent with and nurture developmental, emergent, innovative, and trans-
Article
Full-text available
Evidence-based policy is a dominant theme in contemporary public services but the practical realities and challenges involved in using evidence in policy-making are formidable. Part of the problem is one of complexity. In health services and other public services, we are dealing with complex social interventions which act on complex social systems-things like league tables, performance measures, regulation and inspection, or funding reforms. These are not ‘magic bullets‘ which will always hit their target, but programmes whose effects are crucially dependent on context and implementation. Traditional methods of review focus on measuring and reporting on programme effectiveness, often find that the evidence is mixed or conflicting, and provide little or no clue as to why the intervention worked or did not work when applied in different contexts or circumstances, deployed by different stakeholders, or used for different purposes. This paper offers a model of research synthesis which is designed to work with complex social interventions or programmes, and which is based on the emerging ‘realist’ approach to evaluation. It provides an explanatory analysis aimed at discerning what works for whom, in what circumstances, in what respects and how. The first step is to make explicit the programme theory (or theories) - the underlying assumptions about how an intervention is meant to work and what impacts it is expected to have. We then look for empirical evidence to populate this theoretical framework, supporting, contradicting or modifying the programme theories as it goes. The results of the review combine theoretical understanding and empirical evidence, and focus on explaining the relationship between the context in which the intervention is applied, the mechanisms by which it works and the outcomes which are produced. The aim is to enable decision-makers to reach a deeper understanding of the intervention and how it can be made to work most effectively. Realist review does not provide simple answers to complex questions. It will not tell policy-makers or managers whether something works or not, but will provide the policy and practice community with the kind of rich, detailed and highly practical understanding of complex social interventions which is likely to be of much more use to them when planning and implementing programmes at a national, regional or local level.
Book
This book is an expanded and revised edition of the author's critically acclaimed volume Nuts and Bolts for the Social Sciences. In twenty-six succinct chapters, Jon Elster provides an account of the nature of explanation in the social sciences. He offers an overview of key explanatory mechanisms in the social sciences, relying on hundreds of examples and drawing on a large variety of sources - psychology, behavioral economics, biology, political science, historical writings, philosophy and fiction. Written in accessible and jargon-free language, Elster aims at accuracy and clarity while eschewing formal models. In a provocative conclusion, Elster defends the centrality of qualitative social sciences in a two-front war against soft (literary) and hard (mathematical) forms of obscurantism.
Book
Described by the philosopher A.J. Ayer as a work of ‘great originality and power’, this book revolutionized contemporary thinking on science and knowledge. Ideas such as the now legendary doctrine of ‘falsificationism’ electrified the scientific community, influencing even working scientists, as well as post-war philosophy. This astonishing work ranks alongside The Open Society and Its Enemies as one of Popper’s most enduring books and contains insights and arguments that demand to be read to this day. © 1959, 1968, 1972, 1980 Karl Popper and 1999, 2002 The Estate of Karl Popper. All rights reserved.
Article
Agricultural innovation systems (AIS) are increasingly recognized as complex adaptive systems in which interventions cannot be expected to create predictable, linear impacts. Nevertheless, the logic models and theory of change (ToC) used by standard-setting international agricultural research agencies and donors assume that agricultural research will create impact through a predictable linear adoption pathway which largely ignores the complexity dynamics of AIS, and which misses important alternate pathways through which agricultural research can improve system performance and generate sustainable development impact. Despite a growing body of literature calling for more dynamic, flexible and “complexity-aware” approaches to monitoring and evaluation, few concrete examples exist of ToC that takes complexity dynamics within AIS into account, or provide guidance on how such theories could be developed. This paper addresses this gap by presenting an example of how an empirically-grounded, complexity-aware ToC can be developed and what such a model might look like in the context of a particular type of program intervention. Two detailed case studies are presented from an agricultural research program which was explicitly seeking to work in a “complexity-aware” way within aquatic agricultural systems in Zambia and the Philippines. Through an analysis of the outcomes of these interventions, the pathways through which they began to produce impacts, and the causal factors at play, we derive a “complexity-aware” ToC to model how the cases worked. This middle-range model, as well as an overarching model that we derive from it, offer an alternate narrative of how development change can be produced in agricultural systems, one which aligns with insights from complexity science and which, we argue, more closely represents the ways in which many research for development interventions work in practice. The nested ToC offers a starting point for asking a different set of evaluation and research questions which may be more relevant to participatory research efforts working from within a complexity-aware, agricultural innovation systems perspective.
Article
There have been repeated calls for a ‘new professionalism’ for carrying out agricultural research for development since the 1990s. At the centre of these calls is a recognition that for agricultural research to support the capacities required to face global patterns of change and their implications on rural livelihoods, requires a more systemic, learning focused and reflexive practice that bridges epistemologies and methodologies. In this paper, we share learning from efforts to mainstream such an approach through a large, multi-partner CGIAR research program working in aquatic agricultural systems. We reflect on four years of implementing research in development (RinD), the program’s approach to the new professionalism. We highlight successes and challenges and describe the key characteristics that define the approach. We conclude it is possible to build a program on a broader approach that embraces multidisciplinarity and engages with stakeholders in social-ecological systems. Our experience also suggests caution is required to ensure there is the time, space and appropriate evaluation methodologies in place to appreciate outcomes different to those to which conventional agricultural research aspires.
Article
Although theories of change are frequently discussed in the evaluation literature and there is general agreement on what a theory of change is conceptually, there is actually little agreement beyond the big picture of just what a theory of change comprises, what it shows, how it can be represented, and how it can be used. This article outlines models for theories of change and their development that have proven quite useful for both straightforward and more complex interventions. The models are intuitive, flexible, and well-defined in terms of their components, and they link directly to rigorous models of causality. The models provide a structured framework for developing useful theories of change and analyzing the intervention they represent.
Article
This article describes the development and use of a rapid evaluation approach to meet program accountability and learning requirements in a research for development program operating in five developing countries. The method identifies clusters of outcomes, both expected and unexpected, happening within areas of change. In a workshop, change agents describe the causal connections within outcome clusters to identify outcome trajectories for subsequent verification. Comparing verified outcome trajectories with existing program theory allows program staff to question underlying causal premises and adapt accordingly. The method can be used for one-off evaluations that seek to understand whether, how, and why program interventions are working. Repeated cycles of outcome evidencing can build a case for program contribution over time that can be evaluated as part of any future impact assessment of the program or parts of it.
Article
Many rural poor and marginalized people strive to make a living in social-ecological systems that are characterized by multiple and often inequitable interactions across agents, scale and space. Uncertainty and inequality in such systems require research and development interventions to be adaptive, support learning and to engage with underlying drivers of poverty. Such complexity-aware approaches to planning, monitoring and evaluating development interventions are gaining strength, yet, there is still little empirical evidence of what it takes to implement them in practice. In this paper, we share learning from an agricultural research program that used participatory action research and theory of change to foster learning and support transformative change in aquatic agricultural systems. We reflect on our use of critical reflection within participatory agricultural research interventions, and our use of theory of change to collectively surface and revisit assumptions about how change happens. We share learning on the importance of being strengths-based in engaging stakeholders across scales and building a common goal as a starting point, and then staging a more critical practice as capacity is built and opportunities for digging deeper emerge.
Book
In this new edition of his critically acclaimed book, Jon Elster examines the nature of social behavior, proposing choice as the central concept of the social sciences. Extensively revised throughout, the book offers an overview of key explanatory mechanisms, drawing on many case studies and experiments to explore the nature of explanation in the social sciences; an analysis of the mental states - beliefs, desires, and emotions - that are precursors to action; a systematic comparison of rational-choice models of behavior with alternative accounts, and a review of mechanisms of social interaction ranging from strategic behavior to collective decision making. A wholly new chapter includes an exploration of classical moralists and Proust in charting mental mechanisms operating ‘behind the back’ of the agent, and a new conclusion points to the pitfalls and fallacies in current ways of doing social science, proposing guidelines for more modest and more robust procedures.
Article
The idea of theory-based evaluation (TBE) is plausible and cogent, and it promises to bring greater explanatory power to evaluation. However, problems beset its use, including inadequate theories about pathways to desired outcomes in many program areas, confusion between theories of implementation and theories of programmatic action, difficulties in eliciting or constructing usable theories, measurement error, complexities in analysis, and others. This article explores the problems, describes the nature of potential benefits, and suggests that the benefits are significant enough to warrant continued effort to overcome the obstacles and advance the feasibility of TBE.
Article
Agricultural development is fundamentally a social process in which people construct solutions to their problems, often by modifying both new technologies and their own production systems to take advantage of new opportunities offered by the technologies. Hence, agricultural change is an immensely complex process, with a high degree of non-linearity. However, current ‘best practice’ economic evaluation methods commonly used in the CGIAR system ignore complexity. In this paper we develop a two-stage monitoring, evaluation and impact assessment approach called impact pathway evaluation. This approach is based on program-theory evaluation from the field of evaluation, and the experience of the German development organization GTZ (Deutsche Gesellschaft für Technische Zusammenarbeit GmbH). In the first stage of this approach, a research project develops an impact pathway for itself, which is an explicit theory or model of how the project sees itself achieving impact. The project then uses the impact pathway to guide project management in complex environments. The impact pathway may evolve, based on learning over time. The second stage is an ex post impact assessment sometime after the project has finished, in which the project's wider benefits are independently assessed. The evaluator seeks to establish plausible links between the project outputs and developmental changes, such as poverty alleviation. We illustrate the usefulness of impact pathway evaluation through examples from Nigeria and Indonesia.
Program proposal Available at: pubs.iclarm.net/resource_centre
  • Aas
  • Cgiar
[AAS] CGIAR Research Program on Aquatic Agricultural Systems (2011) Program proposal. Available at: pubs.iclarm.net/resource_centre/WF_2936.pdf (accessed 2 July 2015).
Extension proposal 20152016 Available at: https://goo.gl/8hQT8I
  • Aas
  • Cgiar
[AAS] CGIAR Research Program on Aquatic Agricultural Systems (2014) Extension proposal 20152016. Available at: https://goo.gl/8hQT8I (accessed 8 August 2016).
Complexity, adaptation, and results. Global Development: Views from the Center
  • O Barder
  • B Ramalingam
Barder O and Ramalingam B (2012) Complexity, adaptation, and results. Global Development: Views from the Center. Available at: blogs.cgdev.org/globaldevelopment/2012/09/complexity-andresults.php (accessed 11 September 2015).
Bureau for Policy, Planning and Learning. Discussion note: Complexity aware monitoring
  • H Britt
Britt H (2013) US Agency for International Development (USAID), Bureau for Policy, Planning and Learning. Discussion note: Complexity aware monitoring. Available at: https://usaidlearninglab. org/library/complexity-aware-monitoring-discussion-note-brief (accessed February 2017).
Available at: www.cgiar.org/resources/strategy-and-results-framework
  • Cgiar Consortium
CGIAR Consortium (2015) CGIAR Strategy and Results Framework 2016-2030. Available at: www.cgiar.org/resources/strategy-and-results-framework/ (accessed September 2015).
Monitoring and Evaluations Strategy Brief
  • B Douthwaite
  • M Apgar
  • C Crissman
Douthwaite, B., Apgar, M., Crissman, C. (2014). Monitoring and Evaluations Strategy Brief. CGIAR Research Program on Aquatic Agricultural Systems. Penang, Malaysia. Program Brief: AAS-2014-04.
Impact evaluation of natural resource management research programs: a broader view
  • J Mayne
  • E Stern
Mayne J and Stern E (2013) Impact evaluation of natural resource management research programs: a broader view. ACIAR 84: 79.
Informed by Knowledge: Expert Performance in Complex Situations
  • D Snowden
Snowden D (2010) Informed by knowledge: Expert performance in complex situations. Psychology Press Ltd, pp.223-234.
Realist Impact Evaluation: An Introduction. London: ODI Annual Reports
  • G Westhorp
Westhorp G (2014) Realist impact evaluation: An introduction. London: ODI Annual Reports.
Outcome Harvesting. Ford Foundation. Accessed, 6
  • R Wilson-Grau
  • H Britt
Wilson-Grau, R., & Britt, H. (2012). Outcome Harvesting. Ford Foundation. Accessed, 6, 2012.
Draft ISPC Comments on the revised extension proposal of the CRP Aquatic Agricultural systems (AAS) for
ISPC (2014) Draft ISPC Comments on the revised extension proposal of the CRP Aquatic Agricultural systems (AAS) for 2015-2016. Available at: https://goo.gl/KaBX6A (accessed 31 July 2016).
Outcome harvesting. Ford Foundation
  • R Wilson-Grau
  • H Britt
Wilson-Grau R and Britt H (2012) Outcome harvesting. Ford Foundation. Available at: http://www. outcomemapping.ca/resource/outcome-harvesting (accessed June 2012).
Ybarnegaray is an agricultural engineer with 20 years' experience designing, implementing and strengthening planning, monitoring, evaluation and learning systems particularly in America and Asia, but also in the Pacific and Africa
  • Rodrigo Paz
Rodrigo Paz-Ybarnegaray is an agricultural engineer with 20 years' experience designing, implementing and strengthening planning, monitoring, evaluation and learning systems particularly in America and Asia, but also in the Pacific and Africa.
Available at: www.agropolis.org/cooperation/headquarters-cgiar-consortium
  • Agropolis International
Agropolis International (2015) CGIAR Consortium headquarters in Montpellier. Available at: www.agropolis.org/cooperation/headquarters-cgiar-consortium.php (accessed 11 September 2015).
Boru Douthwaite is an agricultural engineer, technology policy analyst and independent evaluator who has 25 years' experience working in Africa
  • Latin America
Boru Douthwaite is an agricultural engineer, technology policy analyst and independent evaluator who has 25 years' experience working in Africa, Asia and Latin America. His research is aimed at understanding how research output and process can be used to catalyse and bolster rural innovation processes, in particular, how the practice of doing research for development can be improved.
Available at: blogs.cgdev.org/globaldevelopment/2012/09/complexity-and-results
  • O Barder
  • B Ramalingam
Barder O and Ramalingam B (2012) Complexity, Adaptation, and Results. In : Global Development: Views from the Center. Available at: blogs.cgdev.org/globaldevelopment/2012/09/complexity-and-results.php Sep 2012 (accessed 11 September 2015).
An Approach to Evaluating Complex Interventions Accepted for publication in Evaluation poor
  • Mayne Douthwaite
Douthwaite, Mayne et al.: An Approach to Evaluating Complex Interventions Accepted for publication in Evaluation poor. London: ODI Annual Reports.
Evaluation for Accountability: Reality or Myth?
  • J Mayne
Mayne, J. (2007). Evaluation for Accountability: Reality or Myth? In Making Accountability Work: Dilemmas for Evaluation and for Audit. M.-L. Bemelmans-Videc, J. Lonsdale and B. Perrin, Eds: 63-84. New Brunswick, NJ: Transaction Publishers.
CGIAR Strategy and Results Framework 2016-2030
  • Cgiar Consortium
Outcome Mapping: Building Learning and Reflection into Development Programs. Ottowa, Canada: International Development Research Centre (IDRC)
  • S Earl
  • F Carden
  • T Smutylo