ChapterPDF Available

Does our theory match your theory? Theories of Change and causal maps in Ghana

Authors:
  • Bath Social & Development Research Ltd
  • Causal Map Ltd

Abstract and Figures

What do the intended beneficiaries of international development programmes think about the causal drivers of change in their livelihoods and lives? Do their perceptions match up with the theories of change constructed by organizations trying to support them? This case study looks at an entrepreneurship programme aiming to economically empower rural women smallholders in Ghana. The programme provided a combination of financial services, training and peer support to improve the women’s productivity, and purchase and sale options. It was implemented by two Ghanian savings and credit organizations, Opportunity International Savings and Loans, and Sinapi Aba Savings and Loans, with support from the development organization Opportunity International UK (OIUK). We report on a mid-term qualitative evaluation of the programme that used the Qualitative Impact Protocol (QuIP) to gather stories of change directly from the programme participants. These stories were coded, analysed and visualized using a web application called Causal Map.
Content may be subject to copyright.
This version is close to the submitted version which is now in an inaccessible Google Drive.
Powell, S., Larquemin, A., Copestake, J., Remnant, F., & Avard, R. (2023). Does our theory
match your theory? Theories of change and causal maps in Ghana. In L. Simeone, D. Drabble,
N. Morelli, & A. de Götzen (Eds.), Strategic Thinking, Design and the Theory of Change. A
Framework for Designing Impactful and Transformational Social Interventions. Edward Elgar.
Title: Does our theory match your theory? Theories of change and causal maps in
Ghana
Abstract
What do the intended beneficiaries of international development programmes think about the
drivers of change in their livelihoods and lives? Do their perceptions match up with the theories
of change constructed by organizations trying to support them? This case study looks at an
entrepreneurship programme aiming to economically empower rural women smallholders in
Ghana. The programme was implemented by two Ghanian savings and credit organizations,
with support from Opportunity International UK (OIUK). We report on an evaluation of the
programme that used the Qualitative Impact Protocol (QuIP) to gather stories of change from
the programme participants. These stories were coded, analysed and visualized using a web
application called Causal Map.
The chapter illustrates how the causal maps produced were used to answer evaluation questions
by comparing them with the programme’s original Theory of Change. We discuss how this
information can be used to enrich the implementing agencies’ strategic thinking.
Keywords
Theory of Change,
Causal mapping,
Evaluation,
Qualitative data analysis,
International development
Rural livelihoods
Summary
What do the intended beneficiaries of international development programmes think about the
causal drivers of change in their livelihoods and lives? Do their perceptions match up with the
theories of change constructed by organizations trying to support them? This case study looks
at an entrepreneurship programme aiming to economically empower rural women smallholders
in Ghana. The programme provided a combination of financial services, training and peer
support to improve the women’s productivity, and purchase and sale options. It was
implemented by two Ghanian savings and credit organizations, Opportunity International
Savings and Loans and Sinapi Aba Savings and Loans, with support from the development
organization Opportunity International UK (OIUK). We report on a mid-term qualitative
evaluation of the programme that used the Qualitative Impact Protocol (QuIP) to gather stories
of change directly from the programme participants. These stories were coded, analysed and
visualized using a web application called Causal Map.
The first part of this chapter introduces causal mapping: a group of methods that we think
has unrealized potential to inform strategic thinking of organizations by prompting them to
think harder about the ideas and assumptions underpinning their actions and objectives. We
explain how both theories of change and the type of evidence gathered using qualitative impact
evaluations can be understood as different types of causal map. The second part of the chapter
illustrates how the causal maps produced in the selected evaluation were used to answer
evaluation questions by comparing them with the programme’s original Theory of Change. We
highlight differences revealed by this comparison and also discuss how this information can be
used to enrich the implementing agencies’ strategic thinking. This occurs through a process of
evidence-based adjustment to the wider or ‘middle-level’ theory, informing the design and
operation of the programme and others similar to it.
Authors
Steve Powell is director of Causal Map Ltd and freelance evaluator. He has been working in
social research and evaluation internationally since 1995 and is particularly interested in the
logical foundations of theories of change and causal maps.
James Copestake is a professor of international development and director of the Centre for
Development Studies at the University of Bath, UK. His research interests and experience are
centred on development finance modalities and their evaluation, particularly attribution of
impact.
Fiona Remnant is a communications and research professional with a specific interest in
making rigorous research methods more accessible to practitioners. She is co-author of the
Qualitative Impact Protocol and co-founder of Bath Social & Development Research Ltd and
Causal Map Ltd.
Aurélie Larquemin is an independent consultant and researcher in international development.
She has been working in the field of monitoring and evaluation for the past ten years and is
particularly interested in financial inclusion and microfinance.
Rebekah Avard is a project manager and researcher for Bath Social & Development Research
Ltd. She specialises in the QuIP approach to impact evaluation.
Introduction: QuIP and causal mapping
What is a causal map?
A causal map can be defined as consisting of links or arcs between nodes, such that a link
between C and E means, in some sense, that someone believes that C has or had some
causal influence on E. In this chapter, we refer to the nodes as causal factors. This minimum
definition is common to a wide variety of causal mapping approaches (Laukkanen, 1994, p.
323) (Axelrod, 1976, p. 3). In this minimum and inclusive definition, we understand the causal
links only as expressing beliefs about some kind of causal influence, rather than more specific
claims such as 'C was a necessary cause of E' or even 'C was the cause of E'. Evaluators are
interested in any such claim, although of course they are additionally interested in how to
quantify or at least compare the relative size, importance or significance of different causal
claims; hence, causal maps may also differ according to the kind of additional causal
information encoded in the links, for example, strength and/or polarity of the causal
connections.
What is causal mapping?
Constructing and testing theories (including theories of change and middle-range theories) is
increasingly recognized as central to any evaluation that aims to find out how an intervention
worked, and to generalize from the findings. 'Causal mapping' (Ackermann & Eden, 2004;
Axelrod, 2015; Eden et al., 1992; Laukkanen, 1994; Nadkarni & Shenoy, 2004) has been used
quite widely for several decades across a variety of disciplines and fields and provides very
useful tools for these tasks. In particular, it has been frequently used in the field of strategic
management to not only map but also inform strategic management decision-making (Buckley,
2018; Hodgkinson et al., 2004; Huff, 1990; Laukkanen, 1994). Similar approaches sometimes
appear under different names, but reference to it in the evaluation literature is relatively sparse.
'Causal mapping' is an umbrella term which includes the construction of theories of change,
which can themselves be seen as causal maps. It provides a useful framework for constructing
and testing theories from the data collected for an evaluation as well as for making deductions
and drawing conclusions and also suggests a range of tools for gathering primary evaluation
evidence. It provides ways to deal with unstructured and unexpected evidence, and if required,
to combine it with more structured evidence. Various causal mapping approaches provide
explicit methods for gathering and analysing data, but overall, causal mapping is less ‘another
method’ than it is an approach that sheds new light on what evaluators have always done when
they collect, assemble and analyse causal evidence.
Later in this chapter we will compare the original OIUK ToC (which can be considered to be
a kind of causal map) with evidence provided by a corresponding QuIP study (another kind of
causal map) and see how this helps answer some of the evaluation questions as well as illustrate
some of the general ideas associated with causal mapping.
QuIP as causal mapping
The Qualitative Impact Protocol (QuIP) methodology (Copestake et al., 2019) is used to collect
primary data from intended beneficiaries about their perception of change over a specified
period in selected domains of their lives. The stories are collected using a process of partial
‘blindfolding’ that entails explicitly avoiding any reference to the specific intervention being
evaluated at the point of data collection. In most applications of QuIP, the interviewers do not
know who has commissioned the evaluation, nor the Theory of Change being tested by the
study. The purpose of a QuIP study is usually to gather data on changes experienced over the
period of the programme’s implementation, and crucially, to capture intended beneficiaries’
perceptions of the main causal drivers of these changes. Respondents are asked open-ended
questions about changes in relevant areas of their lives, such as health, work and wellbeing.
Respondents (in individual and group interviews) are asked questions about changes in key
aspects of their lives, the causes of these changes, the causes of the causes and so on: a form of
elicitation we call ‘causal back-chaining’. When analysing the interview data, the analyst
(typically not blindfolded) reviews what the different sources are saying, and bit by bit
(inductively, iteratively) tries to identify the common elements in their narratives using causal
factor labels, such as ‘health’ and ‘amount of exercise’. While respondents are invited to share
their experiences and perceptions of change across predetermined well-being domains relevant
to the programme’s Theory of Change, other important outcomes may emerge from their
narratives which were not envisaged by the commissioner; to this extent, the QuIP can be
considered a form of goal-free evaluation (Scriven, 1991). Different respondents will, of
course, have different experiences and perceptions, and will not always use exactly the same
phrases to express their causal narrative. Therefore, it is an exciting and creative challenge to
code and curate a list of causal factors through what can be viewed as a specialized form of
thematic qualitative data coding and analysis. We use software designed explicitly for this task,
Causal Map
1
. To our knowledge, this is the only software designed explicitly for the qualitative
'coding' of narrative text to create a one-to-one correspondence between sections of text which
contain causal claims.
The causal stories collected using QuIP mostly deal with actual events which happened to the
respondents rather than with general events or phenomena. The explanations given often take
the form of changes in one factor explaining changes in another (‘improved’ seeds, ‘increased’
income, and ‘the early arrival of rains’ leading to ‘increased’ yields, for example), or they
mention specific events such as 'support/guidance from organization X', with the implicit
assumption that, in another scenario, these events might have been absent.
Analysing a causal map
One advantage of causal mapping as a research and evaluation technique is that while the
results of ordinary qualitative research on texts is usually more text (with perhaps some tables
to show the frequency of occurrence of particular themes or combinations of them), the main
products of causal mapping are graphical causal maps. They organize and visualize a
potentially very confusing heap of interrelated causal claims in a way that is relatively intuitive
and easy to understand. It can also be directly queried in order to answer research questions
like the evaluation questions in the study presented later in this chapter. Furthermore, while a
‘global’ causal map resulting from a research programme can contain a large number of causal
factors and links by applying filters and other algorithms, the map can be queried to answer
different questions, for example, to simplify it, to trace specific causal paths from particular
drivers or to particular outcomes, or to identify significantly different sub-maps for groups of
sources with different characteristics. With certain assumptions, it is possible to ask and answer
questions such as, 'Which factor has the widest influence?' or 'Which factor leads to the most
positive outcomes?'.
Figure 1 gives an example of a simple causal map. This one arose from the study presented
later in this chapter. It shows only causal links mentioned between COVID-19 and factors just
one step 'downstream' of it; as in, it presents COVID-19 and every factor which was mentioned
by the respondents as having been directly influenced by it. The numbers on the links are the
number of times each link was mentioned.
1
www.causalmap.app
Figure 1. Causal map showing the causal factor 'COVID-19' as well as factors one step downstream of it; simplified to show
only the most frequent factors.
Making deductions from a QuIP causal map and comparing it with a Theory of Change
A QuIP study like the one presented here aims to be able to give the evaluation commissioner
sufficient information to understand what changes the respondents perceive as having been
important in their lives, to what extent they mentioned the intervention as being a driver of any
of those changes, and in relation to other drivers and outcomes. The empirically derived causal
claims can help to validate or challenge the Theory of Change, determine whether there are
unexpected drivers and outcomes (positive or negative), and consider whether this suggests
ways in which the Theory of Change could be modified to improve this or similar interventions
in the future.
Main findings from the OIUK study
Overview of the programme under evaluation
The themes in this chapter are illustrated with a midline evaluation of Opportunity International
UK’s (OIUK) intervention 'Roots of Change: Increasing the economic empowerment of
women in Ghana and the Democratic Republic of Congo through rural financing'a UK Aid
Match project which has been running in Ghana since October 2018. OIUK is an NGO working
in international development. The original report was written by Bath Social and Development
Research in collaboration with Participatory Development Associates Ltd who conducted the
fieldwork in Ghana. Research staff interviewed 48 women and conducted eight focus groups:
four with men and four with women. The interview topics covered farming and other work,
market access, income, savings, borrowing and spending, relationships within the household
as well as the community, well-being, and self-esteem. The questions were framed in an
exploratory manner to try to keep the conversation as open as possible; for example, the
questions about farming started with this broad discussion starter: 'Please tell me about any
changes in the food and cash crops your household grows over the last two years'. This was
then followed up with optional probing questions such as, 'What is the most important reason
for any change? Why did you make these changes? What happened as a result of that change?'.
The evaluation questions, and how the QuIP was used to answer them
Central evaluation questions relevant to this chapter were as follows:
1. Effectiveness/Impact: To what extent did positive (and negative) changes take place in
important areas, and to what extent can these be attributed to the programme?
2. Relevance of the programme plan: Is the programme’s Theory of Change well adapted
to the realities on the ground from the perspective of key stakeholders?
The QuIP was used to answer these evaluation questions by assembling the stories of change
told by respondents into a collective causal map (see Figure 2), using the web application
Causal Map (causalmap.app). The two questions above can be operationalized as follows:
- Effectiveness/Impact: To what extent was the programme explicitly or implicitly
mentioned as a driver in causal chains culminating in positive change in valued areas?
(The aforementioned blindfolding of interviewers and interviewees is particularly
relevant to questions of impact, as it helps mitigate the risk of confirmation bias, thereby
increasing confidence in credibility of the causal claims linking the programme
intervention to changes in valued areas of participants’ lives).
- Relevance of programme plan: How do the causal maps generated from the analysis of
respondent stories correspond to the original programme Theory of Change?
The OIUK Theory of Change as a causal map
The OIUK Intervention Theory of Change is presented in Figure 2.
Figure 2. OIUK Programme Theory of Change
This ToC graphic is elaborated much further by OIUK in text form, but given the whole text is
too long for this chapter, a shortened version follows: 'Our Theory of Change asserts that
providing a combination of financial services, training and peer support will set off multiple,
interlinking changes that will empower rural women in many areas of their lives. A better
understanding of and access to financial servicescombined with increased agricultural
knowledgewill enable women to improve productivity, and purchase and sale options, which
will have a positive impact on immediate revenues as well as long-term income sustainability.
[…] When family and community members observe the roles women are fulfilling, it will
challenge their views regarding women’s capabilities and skills. As women gain in self-
confidence, increasingly positive feelings about their capabilities will inspire bigger dreams
not only for themselves but also for their children’s future roles in society.'
Overall, there is an implication that a set of complementary inputs collectively lead to outputs,
which collectively lead to the outcomes via a set of reinforcing causal processes. As is often
the case (R. J. Davies, 2018), the diagram makes this less complex by simplifying how the
different elements are causally connected. For example, the items near the right-hand side
concern savings. Their positioning presumably implies particularly strong causal links,
probably from bottom to top, between these items in particular. In addition, the fact that the
items within the outcomes, outputs and inputs rows are not clearly lined up with one another
suggests that there may be implied causal links cutting across the diagram, so that, for example,
'Access to inclusive financial services' at the bottom right, might be ultimately linked, via
various outputs, to the outcome 'Increased Financial Inclusion' at the top left. This vagueness
in the diagram about causal connections can be seen as a weaknessmaking the programme
ToC less evaluable, for example (R. Davies, 2012), but it can also be viewed as a strength. For
example, the human mind can arguably get the overall gist of what is being asserted without
being distracted by potentially dozens of connecting arrows; the representation can also
accommodate potential synergies between different activities and outputs. 'Assumptions' are
listed at the far right of the diagramfactors which are largely beyond the control of the
programme. Again, in common with many real-life theories of change, the top level, 'Impact',
is arguably more of a synthetic summary of the outcomes than a separate state or event which
is caused by them; hence, it is perhaps not necessary or possible to assess or measure it
separately from those outcomes (Goertz, 2020; Powell, 2017).
Ideally this ToC would be based on, or at least compatible with, insights from relevant
programme theory. In practice this ToC is largely a theory of action (Patton, 1989), i.e. it
focuses on causal factors which OIUK can influence, and we can assume that the existence of
the causal connections implied in the ToC is supported by programme staff’s experience with
similar interventions.
Although underpinned by wider theory about interventions of this nature in this context,
the OIUK Theory of Change is not expressed in the form of connections between theoretical
variables but rather in terms of more or less concrete events or changes which are expected to
happen. That is why the factors within it are expressed as concrete true/false events or
propositions like 'improved capacity to serve rural women' or ‘capacity to serve rural women
is improved’ and not as timeless variables of the form capacity to serve rural women' such as
those we are likely to see in other forms of causal maps such as systems diagrams. Factors of
the first form will be true or false at a some point in time which is implicitly or explicitly
specified, e.g. the end of the programme, whereas factors of the second form are variables in
the sense that they make take any of a range of (implied or specified) values at any time point.
The analysis
The analysis process involved identifying and coding causal links between causal factors such
as 'Membership of savings group' and 'Purchase of farm equipment' which were identified
through immersion in the complete set of narrative data. When joined together, these links
constituted a very large global map with over 1700 causal links between causal factors and
outcomes across a range of thematic areas. The overall map is too large for a reader to be able
to make much sense of, so filters and other queries are used to answer specific requests such
as:
show all the causal factors obtained from interviews with women on the theme
‘farming’ and other causal factors immediately connected to them;
show only causal claims cited more than five times;
show only factors mentioning the word ‘farm’, and so on.
In particular, a causal generalization of hierarchical coding (Richards & Richards, 1995) was
used such that, for example, 'Gender sensitization' was coded as a subfactor of 'Support/advice
from OIUK', in the form 'Support/advice from OIUK: gender sensitization'. This facilitated a
second way to simplify maps, parallel to the process of simply excluding less frequent factors,
in which less frequent and more detailed factors can be 'rolled up' into more frequent, higher-
level factors.
First evaluation question: effectiveness/impact
Reflecting on the first evaluation question (effectiveness/impact), the causal maps generated
from the coded interviews in this study did show some evidence of changes in the key areas in
the OIUK Theory of Change, and we were able to identify drivers which explicitly linked to
the OIUK programme.
The overview which follows exemplifies the kinds of summaries resulting from the causal
mapping process. In the causal map figures, the boxes represent causal factors and the arrows
between them represent links of causal influence. The numbers on the links denote the number
of times each causal claim was made in the narratives.
2
Factors are given a darker background
where they contain a particular search term.
Subject-specific searches were made once the whole text had been analysed, for example, on
gender, as shown in Figure 3.
Figure 3. Causal map showing causal factors which include the words 'gender' or 'women' or 'relationships' as well as factors
one step upstream or downstream of them, simplified to show only the most frequent factors and links.
Some women reported positive gender-related changes within the household in the form of
increased joint decision-making and a more equal distribution of the workload. At the
community level, they reported occupying leadership positions more often. However, other
2
In these figures, the numbers represent the number of times the link was mentioned. It is also possible to calculate
other metrics such as the number of respondents mentioning each link. The simplification process means that
some factors may appear to be more than the specified number of links upstream or downstream, because
intervening factors and/or links may be hidden.
respondents reported no change in gender relations or social norms, holding to the belief that
certain tasks and positions are more suitable for men than women.
The richest material covered changes in agriculture and farming activities, the main reported
outcome being increased income from farming activities (Fig. 4). This was driven by changes
in agricultural practices leading to increased yield, including better access to information,
inputs and equipment. This was complemented by an increase in prices and subsequent income.
This improvement in agricultural practices and yield is attributed to support and advice
received from several actors, including agricultural extension workers and OIUK partners.
However, despite the inputs, not all respondents experienced this increase in yield and income,
with setbacks attributed to bad weather conditions, pest damage, and lack of inputs and
equipment in some cases.
Figure 4. Causal map showing causal factors which include the words 'agriculture/agricultural' or 'farm' as well as factors
one step upstream or downstream of them, zoomed out to the top level and simplified to show only the most frequent factors
and links.
In the area of financial management, the respondents reported two main changes: the ability to
save more and the need for less borrowing. The drivers of increased savings were increased
income and membership of saving groups, and this was linked to long-lasting outcomes,
including the ability to make business investments and pay for school fees. Borrowing less was
not, however, always perceived to be a positive outcomethere were sometimes complex and
ambivalent stories from different respondents. For some, it was a consequence of increased
income, and associated with increased self-esteem linked to being more self-reliant; for others,
it arose from the inability to repay loans and associated with reduced resilience to shocks.
Coding and interpreting this kind of factor requires special care.
The Covid-19 crisis (Fig. 1) and resulting measures affected many dimensions of the
programme, in both its implementation and in the areas it aims to impact. The respondents were
asked specifically about the changes resulting from Covid-19 in order to identify its impact and
isolate it from changes prior to the crisis. Respondents reported how the situation affected their
well-being through not being able to meet with others for fear of contracting the virus. It also
affected their financial situation by constraining group farming activities and market sales.
Second evaluation question: relevance
How do these findings help to inform the second evaluation question: Is the programme Theory
of Change well adapted to the realities on the ground from the perspective of key stakeholders?
Figure 5. Causal map showing an automatic summary of all causal links. Hierarchical factors are 'zoomed out' to the most
general level, showing the least detail, and only 15 of the most frequently-mentioned connections are shown.
Figure 5 shows a causal map generated automatically as a summary of all the causal links.
First, hierarchical factors are 'zoomed out' to show only the most general level and the least
detail. Then, of all the causal links remaining in the map at that point, only the 15 most
frequently-mentioned are retained. At this top level, the main difference to the original ToC is
of course the presence of the unforeseeable factor, Covid-19. On the other hand, the other
factors on the right of this figure are broadly comparable with the outcomes in the original
ToC. However, they are expressed in a more concrete fashion, for example, 'invested in
farming business' compared with, for example, 'Increased financial inclusion for rural women'.
Comparing the original ToC with the empirical causal maps topic by topic helps to highlight
both unexpected changes and areas where expected changes have not occurred. In this study
there were domains where change was not as clearly articulated by respondents as had been
expected in the programme ToC. But the most frequent finding was that the causal pathways
as presented by respondents were variations or elaborations of those laid out in the programme
ToC, with positive feedback loops helping to reinforce some of the outcomes in unexpected
ways. In this section we consider how the causal maps contributed to a revision of the
programme ToC following this mid-term review.
One of the programme ToC outcomes, which was harder to identify in the causal maps, was
‘improved employment opportunities’. Evidence of new employment opportunities did not
come out clearly in stories of change, rather changes in employment were strongly linked with
changes in the potential of existing agricultural work, increased farming activities and income
and thus enabling farmers to hire farm workers.
There was no evidence of job creation further along the agricultural value chain, for example,
selling produce at markets. This highlighted the need for a clearer definition of how the
outcome would be reached in the programme ToC, including the analysis of opportunities for
the development of jobs outside agriculture, which would help to build the resilience of women.
There was also no mention of market linkages as a cause for change in farming income
experienced by the beneficiaries. This can be explained by the fact that this component had not
yet been implemented by the programme staff.
The causal maps from the study also reveal a different sequence in the programme ToC’s
expected causal mechanisms regarding gender equality and women’s empowerment (see Fig.
3). The study identified four key drivers of change: joint decision-making, joint labour, better
community relationships, and women occupying leadership positions. Some of these were
articulated as (highest-level) outcomes in the programme ToC, therefore not articulating the
follow-on positive impacts, including improvement in household relationships and reduced
workloads for women. Joint decision-making is not just an outcome but is shown in some cases
to be part of a longer causal pathway (see Fig. 3) which reinforces further positive outcomes
which were not explicitly mentioned in the programme ToC: better community relationships,
collective working opportunities and men being more involved in household tasks. This type
of causal mapping elaborated further on the causal mechanisms taking place, going beyond the
programme ToC, highlighting the additional positive outcomes that the programme did not
expect and therefore did not monitor.
Looking at the agriculture-focused component of the programme, the programme ToC’s
assumptions are borne out in the causal maps: Good Agricultural Practices (GAP) training and
access to improved inputs, market linkages and other productive resources all featured as key
drivers of change. However, there is more detail about the consequences of improved
knowledge of agricultural management and practices (see Fig. 4). This could be logically
expected to result from the GAP training delivered but was not mentioned at the output level
in the programme ToC. The causal map reveals that this type of support and advice, delivered
by the programme as well as other stakeholders, played a driving role from an early stage in
some of the most significant positive outcomes.
Regarding financial changes, the programme ToC assumes that the opening of savings
accounts, access to loans, and financial training will lead to both increased savings and easier
access to loans, leading in turn to improved financial inclusion and resilience. The causal maps
confirm that the key driver of financial change is improved financial management resulting
from the training and advice received, attributable to the OIUK programme.
However, contrary to the programme ToC, some of the intended beneficiaries reported
borrowing less. Some of them linked this to negative drivers, including decreased income from
farming, saving less and savings groups collapsing, but others also identified it as a positive
change resulting from increased savings or income arising from investments.
The original programme ToC linked each level of results, collectively, to one another, and
showed general connections among the components of the intervention but did not make these
links explicit. The causal maps produced from the interviews helped to clarify how the different
pillars of the programme are connected to one another.
How Theory of Change and causal mapping contributed to strategic thinking within the
programme
Both the process of causal mapping illustrated above and the mostly positive outcomes it
revealed generated high interest and positive feedback among staff in OIUK and its partner
agencies. The submission of the report was followed by a presentation on both the QuIP
methodology and the key findings of the study, and participants included the country-based
and headquarters programme teams as well as staff running other programmes.
The programme teams were very responsive to the findings, taking in and reflecting on the fact
that the positive changes expected had not been experienced by all intended beneficiaries but
also noting that there were additional unexpected positive feedback loops which were acting to
reinforce some of the programme ToC assumptions, but in different ways. The presentation
also stimulated staff to reflect further on the consequences of negative changes and reasons for
missing positive outcomes. Looking at the unexpected positive changes, the team reflected on
adaptations such as supporting women in leadership positions and further monitoring of use of
savings.
Assessing empirically how far this one relatively recent evaluation influenced the wider
strategic thinking of the organizations involved is much more difficult. However, an integral
part of an organization’s strategic thinking is to develop, review and refine a body of 'middle-
level theory' (Cartwright, 2020) or useful generalizations about how it can influence change in
a range of different contexts, and an important mechanism for doing this is to learn
systematically from specific cases of its application.
A strength of the QuIP is that a non-specialist can also see the plausibility of drawing these
kinds of conclusions from a QuIP study. However, while a more technical discussion of the
logic of how to make inferences with and from causal maps would go beyond the scope of this
chapter, it is worth mentioning a few potential pitfalls that can arise from interpreting this kind
of evidence.
Potential pitfalls in the interpretation of causal maps based on the QuIP
a. Beliefs about causation are not facts about causation
'Source S believes that C influences E' does not logically imply that C influences E, even when
source S is the world’s best expert on the subject matter (as the respondents in QuIP studies
may or may not be). It is the job of the evaluator to critically discuss to what extent we can
make such links from this intermediate feedback loop, triangulating against other evidence
available (monitoring data and experience of local programme staff in short feedback loops,
and applied research in long feedback loops).
b. Absence of a causal claim does not mean absence of a causal link
If Source S did not mention a link between C and E, does that mean there is no such link? Not
necessarily. Due to the exploratory and semi-structured nature of QuIP interviews, the
conversations arising in a QuIP interview and the causal chains within them, are, to some
extent, random walks. Respondent R may mention a factor like 'flooded fields' somewhere in
a causal chain, but respondent S mentions it nowhere. From this, we cannot conclude that
flooded fields were not a factor in any causal link for respondent S. We don’t even know if
respondent S has fields which might get flooded. If they do, are they specific to this respondent
or are they shared across a whole village? Part of a village? This problem with negative
concepts and negative cases is well-known in qualitative research (Mahoney & Goertz, 2004).
In this type of causal map, our sense of what is a 'case' is quite vague, and it is also,
correspondingly, difficult to think of the causal factors as variables which are defined for every
case (indeed, we usually do not start with a codebook from which we code deductively, rather
factors are created inductively based only on what is contained in the statements). So although
we can report '12 (out of 24) respondents mentioned a link between X and Y and that we believe
this to be a significant story within the cases selected', we should hesitate to report this as some
kind of percentage which would imply half the respondents did not experience this; we do not
necessarily know if the other respondents are in fact aware of such a link but did not happen to
mention it, or even if X and/or Y make sense for them. Follow-up sense-making workshops
(with no blindfolding) are a crucial element to raise and further investigate questions like this.
c. The problem of contexts and the transitivity trap
If (1) Source P [a woman farmer] says: 'I received a loan for farming investment (L), so I had
more cash (C)', and (2) Source W [a woman wholesaler] says: 'I had more cash (C), so I
bought more stock (S)', then can we (3) deduce that L → C → S, and therefore that (4) L → S
(receiving a loan for farming investment leads to people buying more stock)?
The simple answer is no, we cannot, because the first part only makes sense for farmers and
the second part only makes sense for wholesalers. In general, in causal mapping, we can only
use the 'transitivity rule' (from B C and C D, deduce that B D) to the extent that the
contexts of the first and second statement overlap. The evaluator should discuss this overlap
when making this kind of inference. In this case, the relevant contexts of farmers and
wholesalers do not overlap, and therefore, the deduction is not valid. The transitivity trap is a
potential danger for all forms of causal mapping and systems mapping, although it is rarely
mentioned. Analysts have to be wary of where they might be falling into this trap and
sensemaking workshops may be necessary to help verify these kinds of deduction.
d. Aggregation and generalization of findings
Each interview (whether with an individual, household, or group) constitutes one source,
although the narratives presented in each may not be completely independent from each
other. To make credible generalizations across a wider population of intended beneficiaries
care is necessary both to select who to interview (Copestake, 2020) and to weigh up carefully
how the evidence obtained from multiple sources can be aggregated to sustain wider
generalization. Construction of causal maps incorporates simple frequency counts of how
often a particular causal claim (or set of causal claims) is cited by the same source and across
multiple sources. But there is no mathematical formula through which these citation counts
can be converted into more general claims to wider truth. What the weight of evidence can do
is (a) add to the range of causal factors to be aware of and (b) strengthen or undermine
confidence about the potential relevance of prior understanding, including the programme’s
official Theory of Change.
Conclusion
This chapter illustrates the use of causal mapping to support a theory-based evaluation via a
case study, which asked what the intended beneficiaries of an international development
programme thought about the causal drivers of change in their livelihoods and lives. The
evaluation shed light on how their perceptions matched up with both narrative and
diagrammatic versions of the Theory of Change constructed by the organization intervening to
support them. The specific programme we looked at had the goal of increasing the economic
empowerment of women in Ghana. A mid-term qualitative evaluation used the Qualitative
Impact Protocol (QuIP) to gather focused stories of change directly from programme
participants. These stories were analysed and visualized using an approach called causal
mapping and using a web application called Causal Map. This process involved identifying
causal links between causal factors such as 'Membership of savings group' and 'Purchase of
farm equipment', which were identified across the stories in an iterative process. When joined
together, these links constituted a very large map with over 1,700 causal links, which presented
an overall picture of change and causes of change related to each of the thematic areas. This
map was filtered and queried in different ways in order to answer some of the evaluation
questions.
In terms of programme effectiveness and impact, the causal maps were able to identify many
frequently mentioned causal pathways from the programme interventions to outcomes of
interest, which is notable, as the qualitative interviewing was double blind ed: neither
researchers nor interviewees knew the identity of the evaluation commissioner or the specific
intervention being evaluated. In terms of programme relevance, we argue that the original
programme Theory of Change can also be considered as a type of causal map, facilitating a
direct comparison with the causal maps arising from the evaluation research. We highlight
differences revealed by this comparison and discuss how this information was used to enrich
the implementing agencies’ strategic thinking by supporting evidence-based adjustment to the
theory informing the design and operation of the programme and others similar to it.
Overall, the case study confirms the potential for evidence-based causal mapping to validate,
modify and extend theories of change. It also illustrates scope for presenting such evidence
with a ‘broader brush’ (corresponding to the original diagrammatic form of the organization’s
ToC) or in more detail (thereby permitting a more detailed assessment of specific causal
assumptions underpinning the programme). Institutionalized through repeated use, we believe
this form of analysis has the potential to inform strategic thinking through the incremental
adjustment of the wider generalizations or middle-level theories that inform the kind of
interventions that development agencies such as OIUK undertake.
References
Ackermann, F., & Eden, C. (2004). Using Causal mapping: Individual and group; traditional
and new. Systems Modelling: Theory and Practice, 127145.
Axelrod, R. (2015). Structure of decision: The cognitive maps of political elites. Princeton
university press.
Buckley, P. J. (2018). How Theory Can Inform Strategic Management Education and Learning.
Academy of Management Learning & Education, 17(3), 339358.
https://doi.org/10.5465/amle.2017.0076
Cartwright, N. (2020). Using Middle-Level Theory to Improve Programme and Evaluation
Design. October.
Copestake, J. (2020). Case selection for robust generalisation: Lessons from QuIP impact
evaluation studies. Development in Practice, 111.
Davies, R. (2012). Rick On the Road: Criteria for assessing the evaluability of Theories of
Change. http://mandenews.blogspot.ba/2012/04/criteria-for-assessing-evaluablity-
of.html
Davies, R. J. (2018). Representing Theories Of Change: A Technical Challenge With
Evaluation Consequences. February.
Eden, C., Ackermann, F., & Cropper, S. (1992). The analysis of cause maps. Journal of
Management Studies, 29(3), 309324.
Goertz, G. (2020). Social Science Concepts and Measurement: New and Completely Revised
Edition. Princeton University Press.
Hodgkinson, G. P., Maule, A. J., & Bown, N. J. (2004). Causal Cognitive Mapping in the
Organizational Strategy Field: A Comparison of Alternative Elicitation Procedures.
Organizational Research Methods, 7(1), 326.
https://doi.org/10.1177/1094428103259556
Huff, A. S. (1990). Mapping strategic thought. John Wiley & Sons.
Laukkanen, M. (1994). Comparative Cause Mapping of Organizational Cognitions.
Organization Science, 5(3), 322343. https://doi.org/10.1287/orsc.5.3.322
Mahoney, J., & Goertz, G. (2004). The possibility principle: Choosing negative cases in
comparative research. American Political Science Review, 98(4), 653669.
https://doi.org/10.1017/S0003055404041401
Nadkarni, S., & Shenoy, P. P. (2004). Nadkarni and Shenoy 2004 -A causal mapping
approach.pdf. 38(2).
Patton, M. Q. (1989). A context and boundaries for a theory-driven approach to validity.
Evaluation and Program Planning, 12(4), 375377.
Pearl, J., & Mackenzie, D. (2018). The Book of Why: The New Science of Cause and Effect.
Hachette UK.
Powell, S. (2017). Visualizing Theories of ChangeHow not to confuse causes and
definitions. Eval - Online Journal of the Bosnia & Herzegovina Evaluation Society,
1(1). https://www.elval.org/en/vizualizaciji-teorije-promjene-kako-razlikovati-uzroke-
i-definicije/
Richards, T., & Richards, L. (1995). Using hierarchical categories in qualitative data analysis.
Computer-Aided Qualitative Data Analysis, 8095.
Scriven, M. (1991). Prose and cons about goal-free evaluation. Evaluation Practice, 12(1), 55
62.
... We see the job of the causal mapper as being primarily to collect and accurately visualise evidence from different sources, often leaving it to others (or to themselves wearing a different hat) to draw conclusions about what doing so reveals about the real world. This second interpretative step goes beyond causal mapping per se (Copestake, 2021;Copestake et al., 2019a;Powell et al., 2023). ...
... Causal maps help us to assemble evidence for the causal processes at work in specified domains, including the influence of activities being evaluated. They can also help expose differences between the evidence given by different sources and differences between the analysed data and theories of change derived from other sources, including those officially espoused by the commissioner of the evaluation (Powell et al., 2023). The identification of differences in understanding can then feed into further enquiry, analysis and action concerning why people have different views, what the implications of this are and how these might be addressed. ...
Article
Full-text available
Evaluators are interested in capturing how things causally influence one another. They are also interested in capturing how stakeholders think things causally influence one another. Causal mapping – the collection, coding and visualisation of interconnected causal claims – has been used widely for several decades across many disciplines for this purpose. It makes the provenance or source of such claims explicit and provides tools for gathering and dealing with this kind of data and for managing its Janus-like double-life: on the one hand, providing information about what people believe causes what, and on the other hand, preparing this information for possible evaluative judgements about what causes what. Specific reference to causal mapping in the evaluation literature is sparse, which we aim to redress here. In particular, the authors address the Janus dilemma by suggesting that causal maps can be understood neither as models of beliefs about causal pathways nor as models of causal pathways per se but as repositories of evidence for those pathways.
... We see the job of the causal mapper as being primarily to collect and accurately visualise evidence from different sources, often leaving it to others (or to themselves wearing a different hat) to draw conclusions about what doing so reveals about the real world. This second interpretative step goes beyond causal mapping per se (Copestake, 2020a;Copestake, Davies, et al., 2019;Powell et al., 2023). ...
... Causal maps help us to assemble evidence for the causal processes at work in specified domains, including the influence of activities being evaluated. They can also help expose differences between the evidence given by different sources, and differences between the analysed data and theories of change derived from other sources, including those officially espoused by the commissioner of the evaluation (Powell et al., 2023). The identification of differences in understanding can then feed into further enquiry, analysis and action concerning why people have different views, what the implications of this are, and how these might be addressed. ...
Preprint
Full-text available
p> Evaluators are interested in capturing how things causally influence one another. They are also interested in capturing how stakeholders think things causally influence one another. Causal mapping, the collection, coding and visualisation of interconnected causal claims, has been used widely for several decades across many disciplines for this purpose. It makes the provenance or source of such claims explicit and provides tools for gathering and dealing with this kind of data, and for managing its Janus-like double-life: on the one hand providing information about what people believe causes what and on the other hand preparing this information for possible evaluative judgements about what actually causes what. Specific reference to causal mapping in the evaluation literature is sparse, which we aim to redress here. In particular we address the Janus dilemma by suggesting that causal maps can be understood neither as models of beliefs about causal pathways nor as models of causal pathways per se but as repositories of evidence for those pathways.</p
... Causal maps help us to assemble evidence for the causal processes at work in specified domains, including the influence of activities being evaluated. They can also help expose differences between the evidence given by different sources and differences between the analysed data and theories of change derived from other sources, including those officially espoused by the commissioner of the evaluation (Powell et al., 2023). The identification of differences in understanding can then feed into further enquiry, analysis and action concerning why people have different views, what the implications of this are and how these might be addressed. ...
Preprint
Full-text available
p> Evaluators are interested in capturing how things causally influence one another. They are also interested in capturing how stakeholders think things causally influence one another. Causal mapping, the collection, coding and visualisation of interconnected causal claims, has been used widely for several decades across many disciplines for this purpose. It makes the provenance or source of such claims explicit and provides tools for gathering and dealing with this kind of data, and for managing its Janus-like double-life: on the one hand providing information about what people believe causes what and on the other hand preparing this information for possible evaluative judgements about what actually causes what. Specific reference to causal mapping in the evaluation literature is sparse, which we aim to redress here. In particular we address the Janus dilemma by suggesting that causal maps can be understood neither as models of beliefs about causal pathways nor as models of causal pathways per se but as repositories of evidence for those pathways.</p
Article
Full-text available
This paper looks at the technical issues associated with the representation of Theories of Change and the implications of design choices for the evaluability of those theories. The focus is on the description of connections between events rather than the events themselves, because this is seen as a widespread design weakness. Using examples and evidence from Internet sources six structural problems are described along with their consequences for evaluation. The paper then outlines a range of different ways of addressing these problems that could be used by programme designers, implementers and evaluators. The paper concludes with some caution speculating on why the design problems are so endemic but also pointing a way forward. Four strands of work are identified that CEDIL and DFID could invest in to develop solutions identified in the paper.
Article
Full-text available
Arrows in Theories of Change, Logical Frameworks and Logic Models are supposed to show causal influences. But often they are (mis-)used to show definitional connections as well, e.g. when one variable summarises others. This article explores the problem and suggests a solution.
Article
Full-text available
Increasingly, thoughtful managers recognize the role of knowledge and learning in corporate action and performance. Concurrently, a new field, management and organization cognition (MOC), has emerged producing useful insights and findings. Thus far, empirical studies have largely focused on single cases or actors, using often archival data and sometimes ambiguous methods. To advance the field will require pragmatic tools for eliciting data on thinking in real organizations and for conducting rigorous and more comparative studies of management and organization cognitions. This paper describes a method for comparatively studying real-life managerial thinking, defined here as the respective manager's beliefs about key phenomena and their efficacy links in their strategic and operative situation. The applicability of such a definition will depend on the requirements of research at hand. The payoff is that, thus defined, key elements in managerial and organizational cognitions can be usefully captured by cognitive mapping, an established approach in MOC research. The approach contains, first, a method for eliciting comparison-enabling interview data of several subjects. Then, using researcher-based, interpretive standardization of the individual natural discourses, databases of standard concepts and causal links, constituting the cause map elements, are distilled. This facilitates a text-oriented description of the thinking patterns of single actors like managers or organizational groups, which can be used in traditional-type mapping studies, which typically assume unitary or quasi-unitary actors. However, the method is intended for comparative analyses, e.g., for pinpointing the cognitive differences or similarities across organizational actors or for constructing and comparing groups, assumed cognitively homogenous. Also, it is applicable for longitudinal studies or aggregated, e.g., industry-level, descriptions of MOC. A PC application is available for the technique, although many of the processing tasks are amenable to general-purpose relational database software. The paper presents a study case comparing the cognitive structures of managers in two interrelated industries in terms of their concept bases and causal beliefs. The objective was to understand the substance of management thinking, as well as the formative logic behind how managers come to think in the shared ways. It is shown that patterns of industry-typical core causal thinking, manifestations of a dominant logic or recipe, can be located, operationalized and comparatively analyzed with this method. Substantively, the contents of management thinking are typically products of complex long-term mechanisms. These consist, first, of organizational problem-solving, recurrently facing a specific, adequately stable constellation of strategic tasks and environment elements, similar within industries and systematically different across them, and, second, of various social processes, which directly transfer and influence management thinking. The paper concludes with discussing the cause mapping method and suggests some options for further studies.
Article
What wider lessons can be drawn from a single impact evaluation study? This article examines how case study and source selection contribute to useful generalisation. Practical suggestions for making these decisions are drawn from a set of qualitative impact studies. Generalising about impact is a deliberative process of building, testing and refining useful theories about how change happens. To serve this goal, purposive selection can support more credible generalisation than random selection by systematically and transparently drawing upon prior knowledge of variation in actions, contexts, and outcomes to test theory against diverse, deviant and anomalous cases.
Article
This paper examines the ways in which theory can improve strategy education and learning. It does so by examining one key question – how and why, and across which dimensions do firms differ? Pedagogical approaches can be formulated to test the proposition that firms differ, to challenge students to explore the reasons for differences between firms (and the impact of this on strategy) and to discover systematic differences and similarities in firms cross a wide variety of contexts
Book
This book outlines a new approach to the analysis of decision making based on "cognitive maps." A cognitive map is a graphic representation intended to capture the structure of a decision maker's stated beliefs about a particular problem. Following introductory chapters that develop the theory and techniques of cognitive mapping, a set of five empirical studies applies these new techniques to five policy areas. Originally published in 1976. The Princeton Legacy Library uses the latest print-on-demand technology to again make available previously out-of-print books from the distinguished backlist of Princeton University Press. These paperback editions preserve the original texts of these important books while presenting them in durable paperback editions. The goal of the Princeton Legacy Library is to vastly increase access to the rich scholarly heritage found in the thousands of books published by Princeton University Press since its founding in 1905.
Article
TRACES are what evaluators left behind—discoveries, records, tracks—which made marks on the profession of program evaluation. Published here are excerpts from our past (e.g., articles, passages from books, speeches) that show where we have been or affected where we are g(r)o(w)ing. Suggestions for inclusion should be sent to the Editor, along with rationale for their import. Photocopies of the original printed versions are preferred with full bibliographic information. Copying rights will be secured by the Editor.
Article
The present study evaluates two alternative causal cognitive mapping procedures that exemplify key differences among a number of direct elicitation techniques currently in use in the organizational strategy field: pairwise evaluation of causal relationships and a freeh and approach. The pairwise technique yielded relatively elaborate maps, but participants found the task more difficult, less engaging, and less representative than the freeh and approach. Implications for the choice of procedures in interventionist and research contexts are considered.