John Mayne

Theories of change (ToCs) are models of how an intervention is expected to bring about desired changes in results of interest. Robust ToCs spell out the conditions under which the intervention should 'work', i.e., bring about or contribute to the desired results. But rarely does the ToC address issues of the future sustainability of the benefits fr...

The basic ideas behind contribution analysis were set out in 2001. Since then, interest in the approach has grown and contribution analysis has been operationalized in different ways. In addition, several reviews of the approach have been published and raise a few concerns. In this article, I clarify several of the key concepts behind contribution...

This is an update from the March 2018 version. Some terms and concepts have been clarified and a discussion on the role and usefulness of feedback loops added.

While contribution analysis provides a basis for making causal claims and understanding how and why change occurs, it does not on its own estimate the relative importance, much less the size, of the causal factors at work. In this CDI Practice Paper written by John Mayne, we discuss ways of assessing the relative importance of such causal factors,...

An update on the previous September 2018 uploaded article

The use of theories of change (ToCs) is a hallmark of sound evaluation practice. As interventions have become more complex, the development of ToCs that adequately unpack this complexity has become more challenging. Equally important is the development of evaluable ToCs, necessary for conducting robust theory-based evaluation approaches such as con...

A note on key terms and concepts in contribution analysis.

Template for Building a COMB Model Theory of Change The COMB model of behaviour change provides a useful framework for building a theory of change (ToC). Many interventions involve several pathways to impact. In what follows, the template is set out for a specific pathway (the 'intervention'). It sets out the results expected along the pathway and...

While contribution analysis provides a step-by-step approach to verify whether and why an intervention is a contributory factor to development impact, most contribution analysis studies do not quantify the 'share of contribution' that can be attributed to a particular support intervention. Commissioners of evaluations, however, often want to unders...

While contribution analysis provides a step-by-step approach to verify whether and why an intervention is a contributory factor to development impact, most contribution analysis studies do not quantify the 'share of contribution' that can be attributed to a particular support intervention. Commissioners of evaluations, however, often want to unders...

An example illustrating the COM-B model, nested ToCs, contribution analysis, ToC Analysis and elements of process tracing

An revised version of a similar paper posted last year.

It is frequently assumed that evaluation should play a significant role in budgeting and more specifically in expenditure reviews. That evaluation should play such a role is often not contested and seen as one of several key aims of evaluation. This article argues otherwise: that evaluation is not fit nor designed to play such a role. Rather, if th...

The COMB Theory of Change Model Note: This is an updated and revised version of the paper posted in 2016. It reflects more recent experience with the COMB model, and clarification of some terms and concepts.

All research for development programs wish to achieve impact, but understanding how to plan for and document this has been challenging. One of the newest and most popular approaches is the use of theories of change (ToC). This paper looks at how ToCs can be used in agricultural research for development (AR4D) programs. ToCs have been widely used in...

While contribution analysis provides a basis for making causal claims and understanding how and why change occurs, it does not on its own estimate the relative importance, much less the size, of the causal factors at work. In this paper we discuss ways of assessing the relative importance of such causal factors, while arguing that there are no quan...

Bien que les théories du changement soient souvent l’objet d’articles en évaluation et qu’il y ait un consensus sur ce qu’est, conceptuellement, une théorie du changement, il n’y a, en réalité, au-delà des principes généraux, que peu de consensus sur ce qu’elles comprennent, sur ce qu’elles devraient mettre en valeur ainsi que de la manière dont el...

Models for theories of change vary widely as do how they are used. What constitutes a good or robust theory of change has not been discussed much. This article sets out and discusses criteria for robust theories of change. As well, it discusses how these criteria can be used to undertake a vigorous assessment of a theory of change. A solid analysis...

There is a growing recognition that programs that seek to change people’s lives are intervening in complex systems, which puts a particular set of requirements on program monitoring and evaluation (M&E). Developing complexity-aware M&E systems within existing organizations is difficult because they challenge traditional orthodoxy. Little has been w...

An organization with a strong evaluative culture engages in self-reflection, evidence-based learning and experimentation. It sees evidence as essential for managing well, but building such a culture is challenging. Community service organizations seek to provide effective services for their clients. To build an evaluative culture, they need to acqu...

Theories of change are increasingly being discussed and referenced in development evaluation even while the elements of what a theory of change consist of differ widely among applications. Equally, examples of actual use of theories of change other than as overview illustrations of interventions are rare. In this article, we present generic models...

Although frequently discussed in the evaluation literature and general agreement on what a theory of change is conceptually, there is actually little agreement beyond the big picture of just what a theory of change comprises, what does it show, how it can be represented and how it can be used. This article outlines models for theories of change and...

This book focuses on assumptions underlyingmethods choice in program evaluation. Credible program evaluation extends beyond the accuracy of research designs toinclude arguments justifying the appropriateness of methods. An important part of this justification is explaining the assumptions made about the validity of methods. This book provides a fra...

This article proposes a combination of a popular evaluation approach, contribution analysis (CA), with an emerging method for causal inference, process tracing (PT). Both are grounded in generative causality and take a probabilistic approach to the interpretation of evidence. The combined approach is tested on the evaluation of the contribution of...

Natural resource management research (NRMR) has a key role in improving food security and reducing poverty and malnutrition in environmentally sustainable ways, especially in rural communities in the developing world. Demonstrating this through impact evaluation poses distinct challenges. This report sets out ways in which these challenges can be m...

Brief points • Natural resource management research (NRMR) programs are complex, involving multiple interventions, actors, levels, and locations; the need to integrate social, environmental, and economic systems; and emerging understanding of the pathways to results. • Credibly linking NRMR actions with intended benefits in terms of reduced poverty...

In this introductory article, a brief history and introduction to contribution analysis is provided to lay the stage for the articles that follow. At the heart of contribution analysis is the aim to be able to make credible causal claims about the contribution an intervention is making to observed results. The key role that theories of change play...

An ongoing challenge in evaluation is the need to make credible causal claims linking observed results to the actions of interventions. In the very common situation where the intervention is only one of a number of causal factors at play, the problem is compounded – no one factor 'caused' the result. The intervention on its own is neither necessary...

An ongoing challenge in evaluation is the need to make credible causal claims linking observed results to the actions of interventions. In the very common situation where the intervention is only one of a number of causal factors at play, the problem is compounded – no one factor 'caused' the result. The intervention on its own is neither necessary...

A group of 12 evaluation practitioners and observers takes stock of the state of program evaluation in Canada. Each contributor provides a personal viewpoint, based on his or her own experience in the field. The selection of contributors constitutes a purposive sample aimed at providing depth of view and a variety of perspectives. Each presentation...

As many reviews of results-based performance systems have noted, a weak evaluative culture in an organization undermines attempts at building an effective evaluation and/or results management regime. This article sets out what constitutes a strong evaluative culture where information on performance results is deliberately sought in order to learn h...

As many reviews of results-based performance systems have noted, a weak evaluative culture in an organization undermines attempts at building an effective evaluation and/or results management regime. This article sets out what constitutes a strong evaluative culture where information on performance results is deliberately sought in order to learn h...

Questions of cause and effect are critical to assessing the performance of programmes and projects. When it is not practical to design an experiment to assess performance, contribution analysis can provide credible assessments of cause and effect. Verifying the theory of change that the programme is based on, and paying attention to other factors t...

Public service quality is a recent trend in the context of new public management. During the 1990s, public service quality initiatives have been launched in numerous countries at various levels of government. These initiatives have involved a variety of approaches: professional standard setting, service excellence based on users' perception of qual...

Integrating performance information into budgeting, managing and reporting has become a common component of good public and not-for-profit management. In many jurisdictions, efforts to do so have been under way for many years, yet progress is usually seen as slow at best. It is also clear that, while much has been learned, many challenges remain; f...

Typically, a good measurement strategy to support results-based management includes both ongoing performance measures and periodic evaluations. It is argued that this is a too limited set of measurement tools, resulting, not infrequently, in less useful and costly ongoing performance measures. It is proposed that, in addition to ongoing performance...

Audit and evaluation play important roles in public management. There can be confusion and debate, however, over what each of these functions cover and indeed what roles they should play. This article reviews and compares each in relation to public management and the key challenges each face in today's public sector. Audit and evaluation should pla...

The authors contend that developments in the public and not-for-profit sectors over the last decade or so have profound implications for the profession of evaluation, implications that are not being adequately addressed. They argue that evaluation needs to transform itself if it wishes to play a significant role in the management of organizations i...

This article addresses ways to enhance the quality of evaluations with weak designs through a variety of quality assurance practices. Many types of evaluations are restricted in the types of designs they can use. Evaluations of development programs with widely dispersed projects in different countries are often a case in point, where the design use...

There is now a large supply of evaluative information in the forms of evaluation, performance reporting and performance auditing. Relatively little attention has been paid to assuring the quality of this information. The article explores the origins, practice and consequences of evaluative information quality assurance in light of the political and...

Program evaluation has developed in Canada to address the concerns of accountability for management of public funds in a context of economic restraint.

Ces dernières années, on s’est beaucoup préoccupé de la gestion de l’information dans les organisations, ainsi que les défis pour que celles-ci puissent s’en servir et s’adapter. Les organisations sont censées attacher de l’importance à l’information, pouvoir tirer les leçons de l’expérience et s’adapter aux circonstances nouvelles. Si la plupart d...

Results, and more particularly outcomes, are at the centre of public management reform in many jurisdictions, including Canada. Managing for outcomes, including setting realistic outcome expectations for programs, and credibly reporting on what was achieved are proving to be challenges, perhaps not unexpectedly, given the challenges faced in evalua...

In recent years, considerable attention has been focused on the importance of managing information in organisations, as well as the challenges for organisations to make use of and adapt from it. Organisationsare expected to value information, to be able to learn from the past and to adapt to changing circumstances. While much of the literature has...

The changing culture of public administration involves accountability for results and outcomes. This article suggests that performance measurement can address such attribution questions. Contribution analysis has a major role to play in helping managers, researchers, and policymakers to arrive at conclusions about the contribution their program has...

Les systèmes d'information sur la performance des programmes n'ont pas donné, dans bien des cas, les résultats attendus, et cela parce qu'on n'a pas mis le soin voulu pour déterminer quels étaient les renseignements nécessaires et qui avait besoin de ces renseignements, et pour établir le lien entre ces éléments et les diverses caractéristiques des...

The Edmonton-Calgary corridor transportation study is an example of multimodal intercity passenger transportation planning in low density corridor. This paper discusses a number of unique approaches used in the study that were the result of the need to deal with the low-density aspects, the need to consider both regional and inter-city travel, and...

Integrating performance information into budgeting, managing and reporting has become seen as an essential part of good public management. In many jurisdictions, efforts to do so have been underway for many years, yet progress is usually seen as slow at best. It is also clear that while much has been learned, many challenges remain; few organizatio...