Content uploaded by Steve Powell
Author content
All content in this area was uploaded by Steve Powell on Nov 13, 2018
Content may be subject to copyright.
Value in Theories of Change p. 1
Theories of Change: making value explicit
Abstract
Background:
This article addresses two problems:
The Flexibility Problem: One advantage of traditional logic models, in which the
variables are ordered into a neat system of layers (“inputs”, “outputs” etc) in a strict
hierarchical format, is that it is easy to see which variables we intervene upon (namely,
all those with no arrows pointing to them) and which variables we value (the ones we
care about, the ones we want to influence – namely, all those without arrows leading
away from them). However, this kind of format, with its neat layers, is too restrictive
to be useful for accurately modelling a wide variety of project theories. So, the
problem arises: if we are to use a more flexible format for theories of change, how can
we show which variables we value, and which we intervene on?
The Definition Problem: What is the difference between a theory showing the causal
influences within and around a project and, more specifically, a theory of change for
the project? Can we provide definitions of “Theory of Change” and “Theory” which
show how the two are related?
Purpose:
To solve the Flexibility Problem and the Definition Problem.
Value in Theories of Change p. 2
Setting:
Any situation in which theories and theories of change are employed, specifically within
social and development projects and programmes but also beyond.
Intervention:
N/A
Data Collection and Analysis:
N/A
Findings:
A definition of “Theory of Change” is introduced, based upon a definition of “Theory”
together with two symbols to mark variables we value (“♥”, or any suitable alternative
symbol) and variables we intervene on (“▶”, or any suitable alternative symbol). These two
definitions and the two symbols together answer both the Flexibility Problem and the
Definition Problem, and have some interesting side-effects as follows. Firstly, they suggest
that it is the task of evaluators to model how stakeholders value aspects of a project just as
much as it is to model the causal chains within a project. Secondly, evaluators (and others)
are, using these ideas, able to model the fact that stakeholders may value variables which are
not at the end of a causal chain, which throws a new light on the debate between results-
based and principles-based programming. Thirdly, they open up a way to understand the
behaviour of stakeholders and stakeholder groups in terms of their own theories of change –
the way they view the world and how they can get what they want – rather than from the kind
of more traditional behaviourist perspective more familiar to most evaluators.
Value in Theories of Change p. 3
The Flexibility Problem: going beyond logic models
In most traditional logic models and logical frameworks, the items are ordered into a neat
system of layers (“inputs”, “outputs” etc) in a strict hierarchical format (see for example
Coleman, 1987; DFID, 2011).
Before going any further, it will be useful to introduce some technical terms. We will call the
items within a logic model, theory of change, etc., "variables“
1
. Also, we will use terms based
on those used by Pearl (2000, p. 13) to refer to variables according to their place within a
network of variables, such as the network shown in Figure 1 and the other diagrams in this
article:
any variable which has an arrow pointing to another variable can be called a “parent”
of the latter;
any variable which has an arrow pointing to it from another variable can be called a
“child” of the latter;
any variable in a Theory which has no “parents”, i.e. there are no arrows pointing to it
can be called a “no-parent”
2
variable;
any variable which has no “children”, i.e. it has no arrows pointing away from it can
be called a “no-child” variable;
any variable in a Theory which is neither a “no-parent” variable nor a “no-child”
variable can be called an “intermediate” variable.
1
Are the boxes even in a traditional logic model really variables? Yes. Even boxes with very simple labels like
“Workshops held”, “Teaching improved”, etc., can be thought of as binary false/true variables (“Teaching
improved – false/true”, etc.), so that the links between them can be thought of as causal hypotheses – if
workshops are held (false/true), then teaching is improved (false/true). In other cases, the boxes may represent
other kinds of variable – for example, “counting number” variables such as the number of visits to a website.
2
I prefer the words “no-parent” and “no-child” for these (purely structural) purposes; network theorists more
usually speak of “leaves” and “roots” – see Pearl (2000) – , but these terms, though well-suited to describing
networks in general, can be counterintuitive when applied to causal networks in which the leaves happen before
the roots.
Value in Theories of Change p. 4
In a traditional logic model, the “no-parent” variables are just exactly the ones we intervene
upon and the “no-child” variables are just exactly the ones we value. Often these models are
presented graphically in a bottom-up sense, in which case the no-parent variables are all those
at the bottom, usually given names like “Inputs” or “Activities”, and all the variables which
we value (the ones we care about, the ones we want to change) are at or near the top – usually
given a name like “Goal”.
Value is a central part of evaluation, perhaps its very core, as Scriven (2012) has argued
persuasively. When looking at a Theory of Change, the question of what we value is just:
which are the things we want to influence, the things which motivate us, without which we
wouldn’t bother with the whole project. When we look at the graphical representations of
project theories, we find that the underlying, implicit notion of value represented is a simple
hierarchical one (even if the causal theory represented is not hierarchical). Some authors such
as Dhillon & Vaca (2018, p. 8) do provide symbols for “result” or “outcome” whose use is not
necessarily dictated by position within the causal network, but more generally we do not have
the differentiated tools we need to represent such an important concept as value within our
theories. The method nearly always presented for constructing a theory of change is
“backwards mapping” (Anderson, 2005, p. 12): to start with the variable(s) which we want to
influence and work backwards from them. This procedure tends to pre-empt the question of
the value of the intervening steps. Intermediate variables are characterised merely as
“preconditions”, getting their value, if they are valued at all, automatically, because they serve
as means to an end.
Rigid planning formats, popular as they are, have been widely criticised (Chambers and Pettit,
2004; Davies, 2004, p. 104; Earle, 2002, p. 2) as too restrictive to be useful for accurately
Value in Theories of Change p. 5
modelling a wide variety of project
3
theories and the broad range of different factors which
may influence them. Specific criticisms are levelled at the use of a fixed number of “layers”
(Inputs, Outputs, Outcomes etc); at the insistence that one variable may have only one “child”
(Davies, 2004, p. 111); at the exclusion of factors beyond the control of the project (Mayne,
2015, p. 224); amongst other issues. A variety of more flexible templates have been
introduced which address some of these issues, some of them called “theories of change”, for
example, (Anderson, 2005).
This gives rise to a problem which I will call “The Flexibility Problem”. If we are to use a
more flexible format for theories of change than for traditional logic models, one in which, in
particular, we can no longer assume that only the no-child variables are valuable, and that we
intervene on all the no-parent variables, how then can we show which variables
4
we value,
and which we intervene on? There is no systematic, accepted way to do either of these things,
although individual project models sometimes employ various symbols, colours etc. for these
purposes. Dhillon & Vaca (Dhillon and Vaca, 2018, p. 80), in a recent paper on the graphical
presentation of theories of change, suggest the use of a special symbol for “activities”,
although they use it not only for “no-parent” variables but also for intermediate variables.
More generally, if we provide a way to explicitly mark which variables are valued, rather than
assuming without discussion that they get their value automatically purely due to their
position within a project diagram, can this help facilitate practical and theoretical discussion
about value within theories of change and within evaluation more broadly? For example,
Davidson (2015, p. iii), following Scriven, gives a good summary of the argument that it is
not enough in an evaluation to simply report scores on variables, or even differences made by
3
I use the word “project” loosely to also include broader, longer-term programmes and to include sector- or
system-level strategies
4
We will use the word “variable” for both the symbols in the written or graphical theory and the variables in the
world, to which they refer.
Value in Theories of Change p. 6
interventions to scores on variables, but to evaluate those scores: simply put, is this score
good enough? Following this line of argumentation, is it possible to represent within a theory
of change not only which variables are valued but which levels of achievement on those
variables count as, for example, inadequate, acceptable or outstanding?
The Definition Problem: from Theory to Theory of Change
The second problem addressed by this article, “the Definition Problem”, is as follows: What
is the difference between a theory showing the causal influences within and around a project
and, more specifically, a theory of change for the project? Can we provide definitions of
“Theory of Change” and “Theory” which show how the two are related?
Theories about how projects or programmes are supposed to work (Chen, 1990; Weiss and
Others, 1995) have often been hailed as a central, or even the central, concept within
evaluation, and for good reason. But there is no universal or even dominant definition of
what, in evaluation, constitutes a theory, or of the difference between a theory and a theory of
change – if, indeed, there is one. These two terms are far from enjoying the kind of consensus
(Vogel, 2012, p. 3) which is the case with, say, the OECD-DAC definitions of
“effectiveness”, etc.
Weiss (1995) defined a theory of change as “a theory of how and why an initiative works”,
which is not so different from that given by Scriven (1981): “a hypothesis about the way that
a program brings about its effects.” These are a good start. Can we go on to relate “theory of
change”, “logic model”, and “theory” to one another in a more satisfying and useful way?
Some authors treat the traditional logic model as just one part of a theory of change, namely
the part which concerns only the intervention variables and their direct causal consequences,
ignoring other influences. For an overview of this issue, see Blamey & Mackenzie, (2007, p.
Value in Theories of Change p. 7
445). Mayne (Mayne, 2015, p. 3), also referring to Patton (Patton, 2008, p. 336) says that
what turns a logic model into a theory of change is just precisely the addition of “no-parent”
variables beyond the control of the project, which he calls “causal assumptions”. Dhillon &
Vaca (2018) on the other hand suggest that including such assumptions is one factor which
turns an ordinary theory of change into a “strong” theory of change.
In the present article I intended to identify, as far as possible, general distinctions and
definitions based on structural characteristics, in the belief that these should prove more
robust than those driven by specific evaluation contexts, issues or agendas. So for example,
Mayne (Mayne, 2015, p. 122) explores the specific layers which many real-life theories of
change will usually include – “behavioural changes”, “direct benefits” etc. These distinctions
are useful in many contexts but may not be applicable in every case.
Limitation
In this article I will focus only on the formal, structural aspects of what constitutes a Theory
of Change and not on the many important political and practical questions such as who
constructs theories of change, for whom
5
, why, and using what processes. Having said that,
providing ways to make value visible within Theories of Change would certainly facilitate
political, practical and ethical discussion about who values what – within and between
stakeholder groups. I will also ignore issues of measurement and how one could, or should,
measure the variables within a theory of change, or their value. Finally, my driving interest is
in the concepts underlying theories of change and not primarily in the (also important) issue
of how exactly we should or could visualise them graphically, as for exampe discussed by
Dhillon and Vaca (2018) and Vaca and Vidueira (2016). The “heart” and “action” symbols
introduced here are basic “Unicode” symbols (available to copy and paste as text on different
5
See for example (Shaw and Crompton, 2003) for a discussion of the moral question of what is an appropriate
(normative) theory for public health.
Value in Theories of Change p. 8
computing platforms) which are generic and easy to draw by hand. In real-life applications,
more attractive realisations of these ideas could be used. The actual symbols are unimportant
for the focus of this article, which is to make value explicit within Theories of Change.
Definition of “Theory”
The definition of “Theory of Change” which I present in the next section below is, quite
literally, theory-based: a Theory of Change is a special type of (programme or project)
Theory. So first it will be necessary to provide a definition of “Theory”. This definition is
based loosely on the work of Judea Pearl (Author, in press; Pearl, 2000; Pearl and Mackenzie,
2018), who has provided a formal and mathematically rigorous (yet non-parametric) treatment
of the kind of causal networks which may underlie theories of change, as well as a set of
related tools to reason about causality.
I introduce the term “Theory” (with a capital “T”) as a new concept which is intended to be
close enough to most existing usages of the familiar word “theory”, as follows:
A “Theory”, in some particular context, is a model (e.g. a belief, claim or
hypothesis; a description, map or picture) presenting how two or more variables
(causally) influence one another: what leads to what.
Some more details on this definition are given in the Note on p. 51.
Figure 1 shows a simplified Theory about the influence of a training course for some
hypothetical new teaching method, first upon teachers’ creativity, and then upon student
achievement.
Figure 1 about here
Value in Theories of Change p. 9
Definition of “Theory of Change”: a Theory plus value (♥) and
intervention (▶)
Now at last, we will (loosely) define a “Theory of Change”:
A Theory of Change is somebody’s Theory which shows how their intervention
on one or more variables (marked with “▶”) causally influences variables which
they value (marked with “♥”). In other words, it shows how they can get what
they want.
This definition highlights what makes an ordinary Theory into a Theory of Change. I use
capital letters to clarify that “Theory of Change” is another new concept: I am not claiming
that this is the definition which everyone meant or should have meant all along when they
talked about “theory of change”. But this new concept is close enough to previous concepts of
“theory of change” to be a useful possible replacement.
Figure 2 shows that we intervene on one variable (Amount of training which teachers receive
on the new teaching method, marked with a “▶”); our intervention sets this variable to
“complete package delivered” rather than “nothing” and it shows what we value (Student
academic achievement, marked with a “♥”; and it shows how these two are connected.
This definition is formulated in terms of a specific “someone” – an agent who believes, values
things, and intervenes. So “a Theory of Change” is, at least from a linguistic point of view,
just like “a plan” – a plan has to be somebody’s plan, and if they don’t believe it, it isn’t their
plan
6
. This leaves open the possibility of separate or even overlapping theories which include
6
Perhaps the only real difference between a Theory of Change and a plan – following the way we usually use the
word “plan” – is that a plan can (perhaps) also be pointless, without any goal – you can plan to stand up, and sit
down, and stand up again at various times during the day, for no special reason. This plan is not a Theory of
Change.
Value in Theories of Change p. 10
different agents who may differ in what they believe, control or value. For the moment, to
keep things simple, we will just assume that it is “we” who do the believing, intervening and
valuing. Nevertheless the important point is that constructing or reconstructing a Theory of
Change involves two difficult modelling tasks: modelling not only (somebody’s) causal
theory but also (somebody’s) “valuation theory” – at this point, “the valuation theory” is
simply the set of variables marked as being valuable, but I generalise this idea below.
Figure 2 about here
As defined here, if a four-year-old child plans (whether implicitly or explicitly) to whine and
whine until her father buys her ice cream, she has a Theory of Change
7
. The conception of
“Theory of Change” presented here is completely agnostic about the kinds of Theory involved
and also about the kinds of steps which are likely to be involved in a particular case, and as
such is very different from that taken by Mayne (Mayne, 2015, p. 122). My aim is to find
some general principles for modelling how people plan and implement behaviour, so there is
no need to restrict our understanding of “Theory of Change” to, for example, only projects
which are intended to benefit society, improve human rights, etc. I have also made no use of
the semi-technical terms “output”, “intermediate result” etc. in the present article as these
terms are often useful in specific circumstances but can be very difficult to use generically.
For many organisations, a “Theory of Change” is often primarily a broad and inspirational
overview. If we can understand such an overview as a map of what influences what and how
we can intervene to get what we want, I would still understand it as a Theory of Change. If it
is nothing but an inspirational picture, I would not.
Value, intervention and Theory of Change mutually define one another: what someone values
7
And we could also say that she is implementing a project.
Value in Theories of Change p. 11
is what they try to maximise, given the opportunities they have to intervene and the theory
they have about what leads to what. If you know the Theory of Change which a particular
agent has, you have a good chance of working out what they value simply by observing where
and how they intervene in different circumstances
8
.
Valued variables should be of the form “more is better”
I suggest only marking a variable as valued, with the “♥” symbol (or some other suitable
symbol), if it is of the form “more is better”: it should be ordered (i.e., we should be able to
distinguish more of it from less of it), and more of it should always be better in terms of the
way we value it. So in a specific context we might mark “number of required vaccinations
actually received by the child” with a “♥” because more of it is better, whereas we would not
do this with “body-weight of the child”; although the latter is an important variable in many
contexts, more of it is not always better.
Possible extensions to this rule are discussed below.
Predicting the difference made by an intervention
When we intervene on a binary variable like “the grant is awarded, false/true”, an
intervention
9
simply sets it to “true” (the state of the variable with the intervention, aka
“factual” or “intervention” state) rather than “false” (the state of the variable without the
intervention, aka “base” or perhaps “counterfactual” state). Similarly, we can think of an
8
As formulated here, “value” is, at least formally, close to the idea of “utility” as variously used in economics
and is open to much of the same controversy. For example, what if we are not good at working out what is good
for us? Chen (1990) addresses this problem with the promotion of “Normative Outcome Evaluation”: the
analysis of whether the actual implementation is optimal compared to other possible implementations. Another
problem: how to deal with the case when our ability to act (to get what we value) is constrained? A prisoner
might in principle value many things which are in their current context hardly relevant. So their everyday Theory
of Change might not even include, say, the wellbeing of a beloved child – though they would take opportunities
(which they don’t currently have) to defend that wellbeing (on which they currently have little influence), and a
more general Theory of Change would have to include both those opportunities and that wellbeing.
9
The word “intervention” is often used as synonymous with “project” or “program”, e.g. (Hansen and Vedung,
2010, p. 297). Here, the focus is specifically on the act of intervening: for making the project happen, rather than
not happen.
Value in Theories of Change p. 12
intervention on a continuous variable as setting it to one level rather than another – but as the
variable has more than two levels, it will be necessary to highlight which level is the
“intervention” level and which is the contrasting or “base” level – for example, holding 20
training workshops rather than zero workshops. Generally, a single intervention will set the
levels of several “no-parent” variables in this way, and it might use feedback from
downstream variables to adjust the release of resources in response to need; and these
variables might stretch or repeat over time, and so forth.
Real-life theories of change nearly always show not only, trivially, the difference made by the
implementation on the intervention variables themselves but also the likely consequences on
all the variables downstream of them which can be deduced using causal inference. Figure 3
does this, though in a rather vague and limited way.
Figure 3 about here
In Figure 3, the variable names include very minimal information about the differences made
under implementation (“receive”, “increase”, “improve” …) – so minimal that it would be
quite hard to falsify this Theory (how small does an improvement have to be in order to still
count as an improvement?). In general, it would be desirable to estimate more precisely the
difference made to the downstream variable(s) by the intervention, given what we know about
the relevant causal influence.
The concept of making a difference was central to the work of the philosopher Daniel Lewis,
who revolutionised ideas about causation: “We think of a cause as something that makes a
difference, and the difference it makes must be a difference from what would have happened
without it.” (Lewis, 1974, p. 167). Naively, we can think of this “difference made” as a
change over time, as the difference between a baseline and an endline score on a variable. In
Value in Theories of Change p. 13
practice of course, baseline scores often change anyway, even without our intervention: the
real “difference made” which should interest evaluators is the difference between the actual or
“factual” score and the “base” score which would have been observed if the intervention (had)
never happened. “Theory of Change” is actually a misnomer – we should be talking about
“Theory of Difference-Making” – but it is probably too late to do anything about this now.
In practice, Theories of Change rarely explicitly mention the strength or nature of the causal
connections or the evidence for them. Instead, the process is usually reversed: we simply
guess, with or without evidence, when constructing the Theory, that a certain difference made
to one variable (e.g. giving the complete package of training rather than nothing) should be
enough to cause some given difference to another (say, 15% more creative techniques),
resulting in diagrams like Figure 4. So, working backwards from such a diagram, our beliefs
about the power and nature of the causal connections can be deduced from the variable labels,
which include phrases like “X rather than Y” or “Z% improvement” – in terms of differences
which we hope will be made to them.
Figure 4 about here
In some cases, we can treat this “difference” as a literal subtraction of numbers. If the
percentage of children vaccinated is 90% following some intervention, and we’d expect only
70% without it, we can say the intervention made a difference of 20%. So we might see a
variable labelled like this: “Percentage of children vaccinated improved by 20%”. In other
cases, the difference made cannot be literally expressed as a subtraction and we might prefer
to specifically mention both the intervention state and the contrasting base state, perhaps
using the phrase “rather than”, e.g. “teachers frequently use a whole range of creative
techniques (rather than hardly ever using any)”. Most real-life theories of change gloss over
these niceties and simply combine the name of the variable and the difference made into
Value in Theories of Change p. 14
something as vague as “Improved creativity”, leaving any details for the more boring parts of
the planning documentation.
To summarise the ideas in this section: interventions consist in setting the levels of one or
more variables, marked with a “▶” symbol (or other suitable symbol), to one level (“the
intervention level”) rather than another (“the base level”, which they would have had without
the intervention). Often, intervention variables are in any case expressed in binary false/true
form, so the intervention simply sets them to true rather than false (“do this” rather than “do
not do this”). Usually, when Theories of Change are presented, the consequences of this
intervention on all the downstream variables (according to what we know about the nature,
strength etc. of the causal links) is actually noted as part of the variable label, in more detail
(e.g. “20% increase in vaccinations” or even “12,000 rather than 10,000 vaccinations”) or less
detail (“increased vaccinations”). In particular we are interested in differences made to the
variables we value, which are marked with a “♥” symbol (or some other suitable symbol).
Where do valued variables and intervention variables appear in a
Theory of Change?
Marking some “no-child” variables as intervention variables
Within logical frameworks or logic models, all the “no-parent” variables are also intervention
variables – they are the points where we can intervene. Theories of change are distinguished
from such models in part because they explicitly include “no-parent” variables which are not
intervention variables, at least not by “us”.
The suggestion made here is to formalise this distinction by marking intervention variables
with the “▶” symbol and leaving other “no-parent” variables unmarked, whether we think of
these as completely external influences such as the weather, or as those parts of an explicit
Value in Theories of Change p. 15
“causal package” (Mayne, 2015, p. 124) which we ourselves do not intervene on.
We will only use the “▶” symbol on “no-parent” variables, and not on intermediate
10
variables. This is because there is something incoherent about saying “We decided to do X;
but also, our decision was caused by factors A, B and C”
11
.
Intermediate variables can be valued
As mentioned above, it is usually assumed without discussion that only no-child variables can
be ultimately valuable. If intermediate variables are treated as valuable at all, it is only in
virtue of their being links in a chain, and in proportion to their distance from the end of it.
Project management may be criticised for focusing “merely” on intermediate variables. There
may be good reasons for this kind of criticism, such as when someone has been trying to
imply that project success consists in, say, attendance at endless workshops. It is often right to
ask, about an intermediate variable, “what is the real use of that?”. But it is wrong to express
this point by banning anyone from recording intermediate variables as valued. There really
are times when we need to be able to say “Alongside improving student outcomes, we also
want the teachers to use creative techniques (an intermediate variable); this is simply
something we value in its own right”, as shown in Figure 5. Perhaps we have a conception of
teacher professional development in which this is a key factor; perhaps we just love all things
creative. In other words, we might value it because it is the kind of thing which contributes to
things like the valued variable causally downstream, or perhaps because it might causally
contribute to some other valued variable which has not been mentioned, but (most
interestingly) for no reason at all: we simply value it.
10
This restriction does not mean that “no-parent” variables necessarily only feature at the earliest time within a
project or programme; interventions can be made at any point in time, and also stretched out or repeated over
periods of time.
11
This is consistent with Pearl’s (2000, p. 347) conception of conducting an intervention on a variable as
deleting all the causal arrows pointing to that variable. By all means we can do further research, in retrospect, on
the reasons why we did in fact intervene as planned. But that research then involves a new Theory of Change
which is based upon the old one, now extended backwards.
Value in Theories of Change p. 16
It is surprising how seeing a variable as a link in a causal chain which leads to something
valued seems to discourage us from considering it, at the same time, as valuable in its own
right. But there is nothing logically incoherent about someone valuing an outcome which also
causally reinforces another outcome which they also value. Of course, a sceptic can say of any
variable, not just an intermediate one, “aha, but why is that valuable, why is student academic
achievement valuable, what does that lead to, what is it good for?”. And indeed this might
provoke a substantive discussion about work-life balance, the problems of stress in young
people, etc. But we have to know where to stop and accept that some things just are valued in
their own right. Otherwise we will fall into an “infinite regress” in which the sceptic keeps
asking “but why is that valuable?”, and the discussion never ends. As Wittgenstein wrote,
(1978, p. 3) “Explanations come to an end somewhere”.
For any of these reasons, if in fact we (also) value an intermediate variable we can mark that
with a “♥” symbol, as is the case with the variable concerning creative techniques in Figure 5.
Figure 5 about here
There is nothing intrinsic to any given variable (neither its duration, nor its purported
sustainability, nor its position in a diagram, nor what it involves – e.g. human behaviour or
technical achievements) which says whether or not it is suitable for being valued by us. A
variable is only valued if someone happens to value it, as the thought experiment in Figures 6
and 7 remind us. Each of these two stories is plausible in its own context. In the first, the
students are told that winning games is ultimately only important because it promotes work
ethic and team spirit; in the second, they are told that actually, team success is what really
counts and that is the real reason for promoting work ethic and team spirit. Both stories are
perfectly coherent.
Value in Theories of Change p. 17
Figure 6 about here
Figure 7 about here
Some philosophers have suggested – see Donnelly (2013) for an overview – that there is in
fact a right and proper place for this kind of (“why is that valuable”) questioning to end. Such
a place might be for example “human rights”; and some approaches to project planning do
implement this idea in practice, as in “Rights-Based Approaches”. Adherents of this theory
would claim that variables are only ultimately worth influencing if they contribute to
improving or protecting human rights. Other authors suggest concepts such as “well-being” as
typical or generic final carriers of value for most theories of change (Mayne, 2015, p. 123).
But limiting our theories of change in this way means limiting their generality. For example,
if we say that only Human Rights are ultimately valuable, there could be no theory of change
of, say, a biodiversity intervention – unless one really wanted to perform the acrobatics of
arguing that all biodiversity interventions are actually valuable because, and only because,
they improve well-being, or human rights. In the present article I am taking the stance that
evaluators need a tool-kit which, having helped them to model the project theory, also allows
them to model the way an arbitrary stakeholder or client happens to value things, without
telling them what they ought to be valuing. (This stance does not imply that, just because
someone says they value some arbitrary thing, they do in fact value it. Claiming that
something is valuable is firstly not a fact-free choice, separate from what others value, and
secondly it only makes sense if the person’s actual behaviour is generally in alignment with
the claim – we touch on this point again below).
So we allow intermediate variables to be valued. This might seem like a relatively trivial step,
but it can have significant ramifications for the way we think about, monitor and manage
Value in Theories of Change p. 18
projects. This step takes us beyond any “Results-Based” approach (Kusek and Rist, 2004) in
the sense that it is not restricted to valuing only the ends of causal chains. So it can also
encompass approaches like Outcome Mapping (Earl, Carden, and Smutylo, 2001), which
specifically values “Progress Markers” – changes in the attitudes and behaviour of immediate
partners (Guijt, 2008, p. 2) – acknowledging that these changes may well, hopefully, go on to
cause other desirable changes, but essentially shifting the focus from such “ends” to the
progress markers themselves. More broadly, this approach encourages us to re-visit the
importance of how we do things (such as acting out of principle) as well as acting in order to
reach a particular goal. “Acting out of principle” is a value which is in a constant tension with
the goals it might help reach further down a causal chain; it is meaningless without them, but
cannot be reduced to them.
This step also encourages us to free ourselves from other restrictions imposed upon project
planning by hierarchical templates such as those involving timing and duration. So if we no
longer automatically assume that the value of a variable comes purely and automatically from
the number of links separating it from the end of a causal chain, we also no longer need to
assume that the most valuable variables are necessarily just exactly those which take longest
to achieve or which will sustain the longest.
“No-child” variables need not be valued
There is no reason why a Theory of Change shouldn’t include “no-child” variables which are
not valued. For example, we might want to note the possibility of some interesting
downstream consequences which we do not directly care about ourselves – beyond our
intervention, beyond even the variables which we value. For example, we might note that our
project to open a new cultural centre might have some implications for traffic flows.
Value in Theories of Change p. 19
Intervention variables can be valued
Having acknowledged the use of the “♥” symbol to mark not only “no-child” variables but
also intermediate variables as valuable, we now realise there is nothing to stop us valuing
intervention variables themselves. For example, perhaps I value helping my neighbour for its
own sake. I’ve been brought up to be a helpful person and I just love that role. I also, of
course, value the consequences, but that isn’t my only motivation.
Figure 8 about here
A related example is what is sometimes called the “humanitarian imperative” (International
Federation of the Red Cross and Red Crescent Societies, 1994, p. 3): I might claim that it is a
good thing, or an essential thing, in and of itself, to lend a helping hand to people in need – for
example after a disaster. I might also get involved in important and complicated arguments
about the possible unintended negative consequences versus the positive consequences of
such help; but this may have nothing to do with my valuing the act of helping in its own
right
12
. Again, the temptation to judge the act of helping as “merely” a means to an end –
presented, as it is, as part of a causal chain, and therefore not one which can be “really”
valued – is a strong one which should be resisted.
12
Considered from the perspective of Complex Adaptive Systems (Holland, 2014, p. 368), in the short term,
values are simply the rules which dictate which behaviours are favoured, but in the long term (“slow dynamic”)
they may themselves be strengthened or weakened according to other criteria, which may include how well they
promote the collective of other values. Indeed, a consideration of the different ways in which adaptive systems
can evolve casts doubt on the claim – inherent in the promotion of “Results-Based” approaches – that agents
which pursue a “results-based” strategy generally succeed over agents which at least to some extent pursue, say a
“norms-based” or a “principles-based” strategy, namely, strategies which value initial, intermediate and process
variables as well as final variables.
Value in Theories of Change p. 20
Extensions
This basic idea of explicitly marking the variables which we value within a Theory of Change
can be extended in various ways.
Highlighting relative priority by using more than one “♥” symbol
It can be useful for stakeholders during project implementation, as well as for ex-ante
evaluators, to know the relative importance of different valued variables, for example to help
decide where resources should go, or should have gone. While there might sometimes be
good reasons for not having or publicising such a prioritisation, here is a suggestion for how
to do it when required: If there is more than one valued variable, we can use varying numbers
of “♥” symbols to show their relative priority. For example, we might put two symbols next to
“Student academic achievement ♥♥” and just one by “Extent to which teachers use creative
techniques ♥”, to show the greater priority of the former. This can be done in a purely
illustrative manner, or in conjunction with a more precise weighting (see final section). This
kind of weighting could be expressed in a different form, for example like this: “♥ = 2.5”, “♥
= 4”, etc.
Highlighting negative value and cost
What about variables like “morbidity” or “stress” or “environmental damage” which have a
negative valence? These break the “more is better” rule given above. We could use the “♥”
symbol for them but this might be misleading. Instead I suggest using a different symbol
which suggests a negative valence such as “☹”
13
or some other suitable symbol; reserving the
13
It might seem more logical to use a pair of symbols which are more obviously opposites of one another, e.g. to
use an upside-down heart for negative valence, or to use a “smiley” for positive valence and this “frowny”
symbol for negative valence. However given the technical requirements for these symbols mentioned earlier
(being widely available as part of the Unicode standard), this particular pair seems to be the most technically
robust.
Value in Theories of Change p. 21
heart symbol for positive valence.
Similarly, although variables which represent costs could also be marked with a “☹” symbol,
we could also consider a special way to mark them, perhaps with one or more “$” symbols.
For example, stakeholders could be reminded of which “no-parent” variables were
particularly expensive by marking them with a greater number of “$” symbols.
Highlighting valued variables which are not ordered
There are many other ways in which a variable might be intrinsically important to us but we
cannot say of it that “more is better”.
For example, in Ainsworth’s (1978) theory of children’s emotional and developmental
attachment, the “attachment style” of a child to a main caregiver is seen as a very important
factor going forward; some attachment styles (“anxious-ambivalent”; “anxious-avoidant”;
“disorganized-disoriented”) are seen as being less beneficial for the child, whereas “secure” is
more beneficial. But I wouldn’t recommend characterising the variable “attachment style” as
valuable without further comment because it is, as such, not even an ordered variable: we
cannot sort the different styles into a single dimension of better versus worse. Their clinical
relevance cannot be reduced to points on a good/bad continuum. We could perhaps use a “♥?”
symbol to show that something here is valuable but not in a “more-is-better” way. We can
contrast this with an alternative, ordered variable which represents simply how secure a
child’s attachment is (and which loses some of the other aspects of the original variable): this
could certainly be marked with the “♥” symbol.
Dealing with multiple stakeholders
Up to now, we have only seen value ♥ and intervention ▶ symbols used globally to show
what “we”, the makers of the Theory, value and control. Now, using the same ideas, I will
Value in Theories of Change p. 22
“zoom out” to take a perspective which includes multiple stakeholders. This has similarities
with an evaluation approach
14
introduced by Hansen and Vedung (2010) which keeps the
project theories held by different stakeholders separate rather than trying to always establish a
consensus theory, which they argue is the approach used in an overwhelming majority of
cases.
Figure 9 about here
When different stakeholders or “agents” interact, they usually control different variables. This
can be done by writing the name of the agent before the ▶ symbol, as in Figure 9. We can use
♥ symbols, preceded by the name of the agent who values this particular variable, in the same
way
15
. Alternatively it might be more convenient to colour the ♥ and ▶ symbols differently
for each (type of) stakeholder, along with an appropriate legend to explain the colours.
This kind of presentation can be really useful for underlining that different agents might do
different things, or put a different emphasis on the same things, because their motivations are
different. It suggests a way of modelling stakeholder motivation which is very different from
the conventional, more behaviourist perspectives familiar from public health research such as
COM-B (Michie, Stralen, and West, 2011). It might help understand the behaviour of
stakeholders and stakeholder groups in terms of their own (cognitive, if not explicitly
formulated) theories of change – the way they view the world and how they can get what they
want.
In Figure 9, all the agents value the main outcome, improved teaching, whereas the increased
14
The approach is named “Theory-based Stakeholder Evaluation” which rather hides its special feature of
dealing with multiple theories
15
There may, of course, be some stakeholders who value things in my Theory but are not able to make any
interventions on any variables within it, or there may be agents who are able to intervene on variables within the
Theory but have no interest in any of its variables, but I make no provision for these kinds of cases. In the
approach presented here, the point of a valuation is that the agent will act to maximise it, according to their
Theory of Change; and the point of an intervention is to maximise what is valued.
Value in Theories of Change p. 23
pay to which the teacher is entitled after completing the training is a positive motivation for
the teacher but a negative one for the Ministry. In addition, part of the trainer’s motivation is
simply the act of providing pro-bono training itself.
In this case, the agents differ (but overlap) on two of the aspects of the Theory of Change –
the variables on which they can intervene and the variables which they value – but share the
third aspect, namely the Theory itself, the variables and causal connections shown here. What
if they differ on the Theory itself?
If their different Theories mostly overlap but one or two stakeholder groups take
account of one or two variables which are ignored by others, it is possible just to mark
that fact on the relevant variables using the “agent: …” notation used already for value
and intervention in Figure 5 (or by using different colours).
It might also be possible to mark different ideas about the causal links by marking the
arrows in the same way, i.e. by using the “agent: …” notation or by using different
colours.
But when there are large differences between the Theories, this approach becomes
very unwieldy and it will be necessary to present a series of separate Theories, using
common elements wherever possible.
It is interesting to note that one of Chen & Rossi’s original papers (Chen and Rossi, 1980) on
Theory-Based Evaluation focuses also on multiple goals and the possibility of re-assessing
originally posited goals.
Modelling valuation theories
The first parts of this article have presented, firstly, two symbols for turning a project Theory
into a Theory of Change by highlighting which variables are to be intervened upon, which are
Value in Theories of Change p. 24
valued, and how the former influence the latter; and secondly, more details on how this idea
could be implemented and extended in various ways.
The final part of the present article takes the idea of simply marking which variables are
valued, and generalises it to modelling everything about the way a stakeholder calculates and
compares value within and between projects: their complete valuation theory. “Valuation
theory” is related to the concept of “normative theory” as discussed by Chen: “guidance on
what goals and outcomes should be pursued or examined” and is even closer to “normative
theory” as discussed by Hansen and Vedung (Hansen and Vedung, 2010, p. 300), with
reference to Chen: “Notions concerning why the various aspects of the situation that are
supposed to be affected by the intervention are preferable or not preferable to the situation
without the intervention or with another intervention.” However, Hansen & Vedung do not
give many further details of this concept.
In the simplest case, an organisation might well care about, say, both environmental benefit
and number of children vaccinated due to a project without ever wanting to combine the two
into one measure. There are often good reasons for keeping variables separate even when they
seem to measure the same thing – for example, a rescue mission might have saved five lives
but sadly led to the death of one of the rescuers. It might be quite abhorrent, and would
probably be of little use, to say that the net outcome was “plus four lives”. In a Theory of
Change with valued variables, if we do not need to ask or answer questions about how much
we value specific changes on each valued variable, or about their combined value, as long as
they are “more is better” variables, we can simply mark them as valued and leave it at that.
Our evaluation report could mention the valuation of both without trying to combine or
synthesise them.
But, what if an evaluation task does involve modelling more sophisticated valuation theories
Value in Theories of Change p. 25
in which value is defined in terms of transformations and combinations of other variables?
We have not yet covered this possibility – although it was already implied when we added the
idea, above, of marking variable priority and marking variables which were associated with
cost.
For example, a client or stakeholder may want to define some overall score or assessment by
combining the scores on many related valued variables such as a cumulative number of events
over time, or the scores of a cohort of students. These can be combined by taking, for
example, a numerical total or average by taking note of the number of students who exceed a
minimum threshold; or by some other method.
We can actually show these combinations directly within the Theory of Change by
introducing defined variables which are also valued variables, as in Figure 10. Suppose our
client tells us that student academic achievement and teachers’ use of creative techniques are
both equally valuable. We agree to rate both variables on a 0-5 scale and construct an overall
rating which is simply the sum of the two. In Figure 10, we model the latter with a defined
variable, marked by a dashed border to show that it is not necessary or even logically possible
to collect additional data for it; the “raw” data from the two variables which define it is all we
need. In addition, a dashed arrow is also used to show a definitional, rather than causal,
relationship (Author, 2017). This network of dashed definitions is the “valuation theory”, built
on top of the causal theory
16
.
16
A key strength of Pearl’s (2000) partially realist approach to causal modelling is that it encapsulates our
knowledge of a domain – creativity in secondary education, perhaps – by breaking it down into important
variables and individual functional relationships between them. These individual pieces are relatively robust and
can be transferred, at least sometimes, into similar but different contexts. This is how we accumulate knowledge.
In a similar way, a key idea put forward in the present article is that the valuation part of someone’s Theory of
Change can be modelled by breaking it down into the way they characteristically value individual variables, and
combinations of them. This knowledge about how a stakeholder values things should be at least somewhat
transferrable into other contexts. This is how we accumulate knowledge about what agents value and how they
are likely to act – knowledge which is important if we want to build our own Theories of Change which include
the behaviour and interests of other stakeholders.
Value in Theories of Change p. 26
Figure 10 about here
Another example: suppose a local non-governmental organisation in a flood-prone area is
working on flood resilience. They want to ensure that every village has both an early warning
siren and an early warning plan. They are not interested in villages only installing a siren or in
villages only adopting a plan. They want to see villages with both. Again, we can model this
situation using dashed lines, as in Figure 11.
Figure 11 about here
In Figure 11, the arrowheads are joined to show some kind of interaction, and the label
“AND” specifies that both parent variables have to be true for the child variable to be true.
But where does the “♥” symbol go? On the defined variable B or the variables S and P which
are part of its definition? In this case at least, S and P are only valuable in combination so
perhaps we should just put the symbol on B.
There are myriad other ways in which our valuation of a project might involve defined
variables which transform and combine other variables. For example, we will need to be able
to deal with variables which are stretched across time – and potentially into the future
17
– as
well as with variables which are associated with some particular time-point, such as an end-
line. For example, we often need to define a variable which collects the cumulative value of
another variable. So, given a Theory of Change which includes a variable representing the
number of visitors to a youth centre over time, we can define a variable which collects the
total number of visits over a given period, accompanied perhaps by another valued variable
which collects information about the percentage of female visitors.
We can also define valued variables on the basis of other variables which are themselves
17
As is the case with the future sustainability of some variable, in particular of a valued outcome
Value in Theories of Change p. 27
defined, as in Figure 12.
Figure 12 about here
It is not always necessary to even show all of the intermediate variables in a Theory of
Change if only the final, “no-child” variable is valued
18
, especially if they are themselves only
defined variables, as in Figure 12. By all means we could live without these intermediate
variables and note the details of the corresponding calculations separately. In the valuation
Theory just as in the causal Theory, the level of detail which should be shown depends on the
use.
Often it is not enough to simply report that there was a very high score on, or very substantial
difference made to, a valued variable. We need to know is that good enough? Did it meet a
certain target or standard? To allow the asking and answering of this kind of question, it will
be necessary to allow the interpretation of specific levels of specific variables – i.e. not just
“more / less valuable”, but, for example, “good enough / not good enough” and more
specifically to address questions like “Was the change made quickly enough?”. Essentially
this establishes variables which are not only valued but have a specific form, in which the
lowest values are interpreted as something like “very poor, very unsatisfactory” and the
highest values are interpreted as something like “very good, very satisfactory”. Scriven (1967)
calls this kind of scale a “goal scale”. We can use an anchor symbol (“⚓”) instead of a heart
in this case. The anchor symbol says: “we’ve cut to the chase: specific scores on this scale are
not just arbitrary, but mean things like accept, reject, outstanding, not good enough, etc.”
Figure 13 about here
The use of evaluative rubrics (King, McKegg, Oakden, and Wehipeihana, 2013) is a very
18
The author would like to thank one of the anonymous reviewers for flagging the need for this clarification
Value in Theories of Change p. 28
promising way to facilitate this process and provide the necessary interpretations of the levels
of the variables in question
19
.
We can use the same ideas to extend a simple valuation theory into the logical skeleton of an
evaluation. We can define new variables to provide evaluative comparisons, for example
comparing benefits with costs. Some classic evaluation questions such as “cost-effectiveness”
can be understood in this way as comparisons between variables which are usually already
present in a Theory of Change. In this way, we can see an evaluation as, amongst other things,
constructing a valuation theory on top of a Theory of Change: a valuation theory in which the
values are constructed by the evaluator in interaction with the Terms of Reference and with
stakeholders. This (e-)valuation theory often differs in some ways from that intrinsic to the
project, perhaps including additional (defined) criteria which were not part of the original
project plan.
Using defined variables in our (e-)valuation theories also allows us to model how our
valuation of some (numerically measurable) variables does not seem to be linear. For
example, in the case of the body weight of a child, we might value medium weights the most
and extreme weights less. So we can provide a new valued variable, called something like
“healthy weight score” which is defined in terms of the physical weight using a non-linear,
inverted-U-shaped, function, giving a low score for extreme weights and a high score for
moderate weights.
Real-life theories of change very often include “goal-level” variables which are not actually
caused by the variables pointing to them but which are a summary of them. Using dotted lines
can help to avoid mixing up causal with definitional links (Author, 2017).
19
These evaluative phrases – ranging from something like “worst imaginable in this kind of case” to “best
imaginable in this kind of case” seem to follow something approaching a universal rule which governs how
humans conduct ratings – what Kahneman (2011) calls “intensity matching”.
Value in Theories of Change p. 29
As a final point, it will also be essential to show how this task of modelling a valuation theory
– like all other evaluation tasks – can work with non-numerical, as well as numerical,
variables. So in Figure 10, if the scores were expressed simply as “outstanding” in both cases,
then even without a numerical score we should be able to summarise these two results at least
as “outstanding” and certainly not as “poor”: a non-numerical reasoning process – see
(Scriven, 2012) – which we could call “soft arithmetic”.
The evaluation task which I have here described as “modelling the valuation theory”
corresponds very well with the list of “evaluation-specific methodologies” given by Davidson
(2015, p. 6)
20
and can be seen as a central one in evaluation.
This section has discussed some essential extensions to the basic definition of a Theory of
Change given above. Further possible extensions are covered in the final section of this
article.
20
Declaring the task or skill of modelling a valuation theory as being what is essential to “evaluation-specific
methodologies” is also an improvement on Davidson’s list just because it attempts to describe the essential
feature rather than listing an (apparently arbitrary, if persuasive) set of examples.
Value in Theories of Change p. 30
Conclusion and challenges
In the Introduction I presented the Flexibility Problem: (If we are to use a more flexible
format for theories of change, how can we show which variables we value, and which we
intervene on?) and the Definition Problem (What is the difference between a Theory showing
the causal influences within and around a project and, more specifically, a Theory of Change
for the project?). I argued that the “♥” and “▶” symbols are a direct answer to both problems.
The body of the present article has suggested a definition of “Theory of Change” as a special
kind of Theory which involves this two concepts, in the course of which it was first necessary
to provide a loose definition of “Theory”. I have paid particular attention to the “♥” symbol
and the task of modelling what we value within a Theory. Showing explicitly, within a
Theory, which variables are valued can help bring the issue of value within Theories of
Change to the forefront.
The tools presented only allow us to do quite rudimentary modelling of value. For one thing,
the underlying definition of Theory presented here needs to be enriched, for example to
include Theories which include emergent, changing or hard-to-predict variables and links,
feedback loops, etc
21
. But in particular, the way we model value needs to be extended to
address at least some of the following issues:
valuing the difference made on valued variables as a consequence of some
intervention. Above, I suggested that evaluations need to be able to assess the
difference which an intervention makes to valued variables. But does this mean that
we are dealing with some kind of subtraction – if an intervention improves some
valued outcome score from, say, 5 to 7, can we say that the intervention has produced
“two units of value”?
21
The present author has made some attempts in this direction at http://slides.theorymaker.info.
Value in Theories of Change p. 31
distinguishing between valuation which is implicit and valuation which is explicit.
Does it make sense to say to a client “In your plan you’ve forgotten to say that you
also really value X, … you do, don’t you? … shall we add it?” Does it perhaps even
make sense to say “this agent actually values X, but won’t admit it”?
capturing the way an evaluator might feel ethically or professionally compelled to
critique a client’s own valuation, perhaps from the perspective of a wider, shared
system of values, effectively saying: “These things that you value, they aren’t really so
valuable”, or “You said you value X, but doesn’t that conflict with your commitment
to human rights”? What happens if the evaluator actually manages to shift the client’s
views?
capturing how clients may value not only specific variables but a whole process –
for example, when a donor wants to know about any detrimental impacts of any part of
a project on the environment.
modelling what happens when a valuation theory is changed, e.g. during a project,
perhaps in accordance with a fixed set of “higher” values or as part of a process of
adaptation of values
22
.
In conclusion, we can note that the approaches introduced here have had some interesting and
important side-effects. Three of the most interesting are as follows.
They suggest that it is the task of evaluators to model how stakeholders value aspects
of a project just as much as it is their task to model the causal chains within a project.
These approaches can help model the fact that stakeholders may value variables which
are not at the end of a causal chain. This throws a new light on the debate between
results-based and principles-based programming.
22
See footnote #13, on Complex Adaptive Systems
Value in Theories of Change p. 32
They open up a way to understand the behaviour of stakeholders and stakeholder
groups in terms of their own theories of change – the way they view the world and
how they can get what they want – rather than from the kind of behaviourist
perspective more familiar to most evaluators.
References
Author. (2017).
Author. (in press).
Ainsworth, M. D. S., Waters, E., and Wall, S. (1978). Patterns of attachment: A psychological
study of the strange situation. Lawrence Erlbaum Associates.
Anderson, A. A. (2005). The community builder’s approach to theory of change. The Aspen
Institute.
Blamey, A., and Mackenzie, M. (2007). Theories of Change and Realistic Evaluation: Peas in
a pod or apples and oranges? Evaluation, 13(4), 439–455.
https://doi.org/10.1177/1356389007082129
Chambers, R., and Pettit, J. (2004). Logframe. A Critique. Retrieved from
http://www.betterevaluation.org/sites/default/files/simonhearn_en_Logframe_A_Critique_199
-1.pdf
Chen, H.-T. (1990). Theory-driven evaluations. Sage.
Chen, H.-T., and Rossi, P. H. (1980). The multi-goal, theory-driven approach to evaluation: A
Value in Theories of Change p. 33
model linking basic and applied social science. Social Forces, 59(1), 106–122.
Coleman, G. (1987). Logical framework approach to the monitoring and evaluation of
agricultural and rural development projects. Project Appraisal, 2(4), 251–259.
https://doi.org/10.1080/02688867.1987.9726638
Davidson E J. (2015). Question Driven Methods or Method Driven Questions: How We Limit
What We Learn by Limiting What We Ask. Journal of Multidisciplinary Evaluation, 11(24),
i–x.
Davies, R. (2004). Scale, complexity and the representation of theories of change. Evaluation,
10(1), 101–121.
DFID. (2011). How to note: Guidance on using the revised Logical Framework. Department
for International Development, 1–33.
Dhillon, L., and Vaca, S. (2018). Refining Theories of Change. Journal of MultiDisciplinary
Evaluation, 14(30), 64–87.
Donnelly, J. (2013). Universal human rights in theory and practice. Cornell University Press.
Earl, S., Carden, F., and Smutylo, T. (2001). Outcome mapping: Building learning and
reflection into development programs. IDRC, Ottawa, ON, CA.
Earle, L. (2002). Lost in the matrix : The logframe and the local picture. INTRACS 5th
Evaluation Conference: Measurement, Management and Accountability?, (December 2002),
1–17.
Guijt, I. (2008). Critical readings on assessing and learning for social change: A review.
Institute of Development Studies at the University of Sussex.
Value in Theories of Change p. 34
Hansen, M. B., and Vedung, E. (2010). Theory-Based Stakeholder Evaluation. American
Journal of Evaluation, 31(3), 295–313. https://doi.org/10.1177/1098214010366174
Holland, J. H. (2014). Complexity: A Very Short Introduction. Oxford: OUP.
https://doi.org/10.1093/actrade/9780199662548.001.0001
International Federation of the Red Cross and Red Crescent Societies. (1994). Code of
Conduct in Disaster Relief. IFRC. Retrieved from http://www.ifrc.org/en/publications-and-
reports/code-of-conduct/
Kahneman, D. (2011). Thinking, fast and slow. New York: Farrar, Straus; Giroux.
King, J., McKegg, K., Oakden, J., and Wehipeihana, N. (2013). Rubrics: A method for
surfacing values and improving the credibility of evaluation. Journal of Multidisciplinary
Evaluation, 9(21), 11–20. Retrieved from
http://journals.sfu.ca/jmde/index.php/jmde_1/article/viewFile/374/373
Kusek, J. Z., and Rist, R. C. (2004). Ten Steps to a Results-Based Monitoring and Evaluation
System: A Handbook for Development Practicioners.
Lewis, D. (1974). Causation. The Journal of Philosophy, 70(17), 556–567. Page references
are to his (1986) reprint.
Lewis, D. (1986). On the plurality of worlds. Oxford, 14, 43.
Mayne, J. (2015). Useful Theory of Change Models. Canadian Journal of Program
Evaluation, 30(2).
Michie, S., Stralen, M. M. van, and West, R. (2011). The behaviour change wheel: A new
method for characterising and designing behaviour change interventions. Implementation
Value in Theories of Change p. 35
Science, 6(1), 42. https://doi.org/10.1186/1748-5908-6-42
Patton, M. Q. (2008). Utilization-focused evaluation. Los Angeles: Sage.
Pearl, J. (2000). Causality: Models, reasoning and inference. Cambridge Univ Press.
Pearl, J., and Mackenzie, D. (2018). The Book of Why: The New Science of Cause and Effect.
Hachette UK.
Scriven, M. (1967). The methodology of evaluation. In Perspectives of curriculum evaluation.
Scriven, M. (1981). Evaluation Thesaurus (3rd ed., p. 183). Inverness, CA: Edgepress.
https://doi.org/10.1016/0020-7489(92)90035-F
Scriven, M. (2012). The logic of valuing. New Directions for Evaluation, (133), 17–28.
https://doi.org/10.1002/ev
Shaw, I., and Crompton, A. (2003). Theory, Like Mist on Spectacles, Obscures Vision.
Evaluation, 9(2), 192–204. https://doi.org/10.1177/1356389003009002005
Vaca, S., and Vidueira, P. (2016). Using Graphical Perception Principles to Improve the
Systems Thinking Tools’ Data Visualization: Revisiting the Systems Dynamics Model.
JMDE Journal of MultiDisciplinary Evaluation, 12(26), 18–24.
Vogel, I. (2012). Review of the use of Theory of Change in international development. UK
Department for International Development (DFID).
Weiss, C. H., and Others. (1995). Nothing as practical as good theory: Exploring theory-based
evaluation for comprehensive community initiatives for children and families. New
Approaches to Evaluating Community Initiatives: Concepts, Methods, and Contexts, 1, 65–
Value in Theories of Change p. 36
92.
Wittgenstein, L. (1978). Philosophical investigations. Oxford: Blackwell.
Value in Theories of Change p. 37
Figure 1. A simplified Theory about the influence of a training course
Value in Theories of Change p. 38
Figure 2. A simplified Theory of Change showing “heart” and “intervention” symbols
Value in Theories of Change p. 39
Figure 3. A simplified Theory of Change in which the labels of the variables are expressed in
terms of the difference(s) made by the intervention
Value in Theories of Change p. 40
Figure 4. A simplified Theory of Change in which the labels of the variables are expressed in
terms of the difference(s) made by the intervention, with more detail given about the
differences
Value in Theories of Change p. 41
Figure 5. Simplified Theory of Change in which an intermediate variable is valued as well as
a “no-child” variable.
Value in Theories of Change p. 42
Figure 6. Simplified Theory of Change in which success of school soccer team is explained as
being important only because it promotes work ethic and team spirit. To that end, the teacher
intervenes to provide training designed primarily to improve changes of winning games.
Value in Theories of Change p. 43
Figure 7. Simplified Theory of Change in which work ethic and team spirit in school soccer
team are explained as being important because they promote team success. To that end, the
teacher intervenes to provide training designed primarily to improve work ethic and team
spirit.
Value in Theories of Change p. 44
Figure 8. Simple Theory of Change in which a “no-parent” variable is valued as well as a “no-
child” variable.
Value in Theories of Change p. 45
Figure 9. Simplified Theory of Change in which three stakeholders differ in what they value
and where they intervene.
Value in Theories of Change p. 46
Figure 10. A simplified Theory of Change showing a valued variable which is defined in
terms of other variables
Value in Theories of Change p. 47
Figure 11. Simplified Theory of Change in which the single valued variable B is defined as S
“AND” P, i.e. it is only true when both S and P are true
Value in Theories of Change p. 48
Figure 12. Fragment of a Theory of Change and valuation theory for a youth project. The
right-hand variable is defined in terms of variables which are themselves defined in terms of
others.
Value in Theories of Change p. 49
Figure 13. As Figure 12, but the right-hand variable is an “anchor” variable: for example,
specific low scores or ranges of low scores mean something like “inadequate”, and high
scores mean something like “adequate”.
Value in Theories of Change p. 50
Additional notes on the definition of “Theory” as presented on p. 8
Things to note in this definition:
“Theory” in this sense refers only to causal theories about what affects what and not,
for example, theories expressing general principles, classification or categorisation.
“Theories” in this sense do not have to express any particularly profound, general, or
complete piece of knowledge. They should simply express, in a Bayesian sense, our
best guess at how things work.
Also, although this is an explicitly causal, not correlational, approach, I have not
defined “causal”, “causality”, etc; these are probably best left as primitive, undefined,
concepts, see (Pearl, 2000, p. 96).
With this definition, a Theory consists of variables, things which can take (or could
have taken) different values or levels or scores. (It is more usual to talk about the
“value” of a variable, e.g., the value of the variable “current year” is “2018”, but here I
will use the word “level” or “state” as I am already using the word “value” in a
different sense). We can also include variables (actually, sets of variables) which
stretch or repeat across time or bind together sets of cases such as student scores
within a school class. So we can talk about a “variable” like “total number of hours of
sunshine today” which is actually a set of many different variables, one for each day –
or like “student score in the maths test”, which can have one score for each of, say, 30
students, and so is really a set of 30 variables. This contrasts with the way “variable” is
defined in statistics as a set of data points, but is consistent with the broader use of
“variable” in other areas of mathematics, and consistent with Pearl’s (2000) usage.
Variables, so conceived, might have numerical levels, or ordered but non-numerical
levels, or just the levels “true” and “false”; or they might be more vaguely-defined sets
of more vaguely-defined possibilities. Perhaps we can even include variables which
Value in Theories of Change p. 51
are a fuzzily-defined bunch of narrative possibilities: for example, we can think of the
sentence “the impressive and insightful workshops led to the teachers’ enthusiastic
promotion of the new ideas in their teaching” as describing a causal relationship
between two very rich, narrative-type variables. For example, instead of being
charismatic and insightful, the workshops could have been confusing, or divisive, or
simply pleasing, and so on, and anything in between. The teachers’ response could
have been reluctant, or inspiring, or any of a host of other things. Neither of these
variables can be reduced to a simple set of numeric or ordered levels. Yet we can
understand the causal story because we possess enough psychology, or at least enough
folk psychology, to see how the state of the first variable leads to the state of the
second variable.
A Theory is usually presented in the form of a kind of network of arrows or “directed
graph” linking the variables, so the set of all the variables pointing to some variable
are those which influence it causally. As Pearl (2000, p. 13) emphasises, the main
advantage of presenting a Theory in the form of a network is that it restricts the direct
causal influences of a given variable only to its parents – we can predict the state of
that variable only by referring to its parents, without worrying about all the other
variables in the Theory. There are two important assumptions: 1) all the variables in a
Theory are connected into one network, i.e. there are no separate fragments; 2) we
only allow graphs without loops, i.e. we restrict the definition of Theory to so-called
Directed Acyclic Graphs (Pearl, 2000, p. 13). This does not mean that we forbid
“feedback loops”, because a such a loop as usually understood actually comes back to
a different variable – identical to the first in every way but at a different point in time,
which makes it a different variable. Dealing with this is beyond the scope of the
present article.
Value in Theories of Change p. 52
In most practical cases, at least some parts of the composite Theory will be derived
from different sources in different contexts with different kinds, and quality, of
evidence. More details about each causal connection, whether vague or precise, should
ideally be provided as a constituent part of the Theory in order to answer questions
like: “if this parent variable is set to this level …, what level will the child variable
take?”. These influences can be noted directly on the child variable or the incoming
arrows, or elsewhere, as appropriate. Such influences can be incomplete, probabilistic,
non-linear, fuzzy, or, as in Figure 1, involve other unspecified and unknown factors.
Scriven (2012) argues persuasively against the idea that “approximations [have] no
place in laws of nature”. All we need for a Theory about the influence of one thing on
another, is reason to believe that manipulating the influence Variable(s) might,
sometimes, make some difference to the consequence Variable.
The direction of the connections (most frequently, positive or negative) is often not
explicitly stated in theories of change, and nor are they in Figure 1; often, as here, it is
safe to interpret them as, broadly speaking, positive: “The more teachers are exposed
to the new teaching method, the more creative teachers are, and in turn the more
creative teachers are, the more student academic achievement will improve”.
