Content uploaded by Barbara Christine Van Mierlo
Author content
All content in this area was uploaded by Barbara Christine Van Mierlo on Jan 16, 2015
Content may be subject to copyright.
Evaluation
2015, Vol. 21(1) 99 –115
© The Author(s) 2015
Reprints and permissions:
sagepub.co.uk/journalsPermissions.nav
DOI: 10.1177/1356389014564719
evi.sagepub.com
The need for reflexive
evaluation approaches in
development cooperation
Marlèn Arkesteijn, Barbara van Mierlo
and Cees Leeuwis
Wageningen University, The Netherlands
Abstract
Within development cooperation, development issues are increasingly recognized as complex problems
requiring new paths towards solving them. In addition to the commonly used two dimensions of
complex problems (uncertainty and disagreement), we introduce a third dimension: systemic stability;
that is, stability provided by rules, relations and complementary technology. This article reflects on
how development evaluation methodologies and especially those introducing a complexity perspective
address these three dimensions. Inferring that this third dimension deserves more attention, we explore
the characteristics of reflexive evaluation approaches that challenge systemic stability and support
processes of learning and institutional change. We conclude that reflexive evaluation approaches may
well complement current system approaches in development evaluation practice.
Keywords
complexity, development, evaluation, reflexive methodologies, system learning
Introduction
Within development cooperation, there is growing recognition that development problems such
as poverty, health risks and depletion of natural resources are complex problems that require
new paths towards problem solving and systemic evaluation approaches (Hummelbrunner,
2011; Jones, 2011; Nobuko, 2010; Ramalingam, 2013). Over the last few years, various national
and international development organizations have paid explicit attention to the complexity of
development. UNICEF and EvalPartners organized webinars on the theme (2012), GIZ (the
German development organization) held a conference on complexity and systemic approaches
in evaluation in 2011, while others highlighted complexity in a range of articles, working notes
and books, like the Overseas Development Institute (Jones, 2011), the Japanese Foundation for
Corresponding author:
Marlèn Arkesteijn, Bergsingel 121a, 3037 GC Rotterdam, The Netherlands.
Email: info@capturingdevelopment.com
564719EVI0010.1177/1356389014564719EvaluationArkesteijn
research-article2014
Article
100 Evaluation 21(1)
Advanced Studies on International Development (Nobuko, 2010), the British PANOS (2009)
and the Dutch organization PSO (Van Ongevalle et al., 2014).
Most researchers writing about complexity characterize complex problems as having two
dimensions; namely, uncertainty and disagreement. In this article, we suggest adding a third
dimension of complex problems: systemic stability. This dimension refers to the historically
grown mechanisms that provide stability for the current system and favour existing, undesirable
but normalized social practices, and helps to explain why many interventions fail to improve the
situation. In this article, we explore the extent to which evaluation approaches used in develop-
ment cooperation address the three dimensions of complexity and conclude that challenging the
mechanisms that stabilize the current problematic situation deserves far more attention.
Complex problems revisited
Many problems that prevail generation after generation (for instance, the depletion of natural
resources and health risks) are characterized as complex, wicked, unstructured or persistent
problems (Hisschemöller and Hoppe, 1996; Rittel and Webber, 1973). In vast bodies of litera-
ture, such problems are categorized as having two dimensions. The first dimension refers to
the lack of certainty about the nature of the problem and its solutions because of limited (sci-
entific) knowledge about the causes and hence the solution to the problematic situation. The
other dimension refers to the lack of consensus on relevant values regarding solutions and
interventions: ‘A problem is called unstructured when there is neither consensus nor certainty,
yet there is still a widespread sense of discomfort with the status quo’ (Hisschemöller and
Hoppe, 1996: 52). Scholars referred to by practitioners and evaluators within development
cooperation like Patton (2011), Snowden and Boone (2007), Stacey (1992) and Zimmerman
and Glouberman (2004) use similar dimensions of uncertainty and dissensus to distinguish
between simple, complicated, complex and chaotic problems or interventions (see Figure 1).
Whether they call problems unstructured, wicked, persistent or complex, all the above-
mentioned scholars seem to agree on the implications for handling such problems: complex
problems are neither defined nor solved by one single actor or organization. They require
the involvement of various actors with different perspectives that need to become involved
in a social learning process (Jones, 2011; Patton, 2011). It is uncertain how or whether an
Figure 1. Zone of complexity (adapted, with permission, from Patton, 2011: 90).
Arkesteijn et al.: The need for reflexive evaluation approaches in development cooperation 101
intervention leads to a specific result, because of multiple interactions. It is therefore impos-
sible to know beforehand what interventions will work and what the effects of interventions
will be. Even if actors have agreed on a solution, and in hindsight they understand how it
worked, a second time the same solution may fail. Acknowledging the inherent unpredict-
ability of any change path taken is assumed to require an integrative, adaptive management
style of probing and learning, and a recurrent reflection on emerging patterns (Jones, 2011;
Patton, 2011; Snowden and Boone, 2007).
This distinction between simple, complicated and complex situations is criticized by
Mowles (2014). He states that the mere act of choosing whether a situation should be treated
as simple or complex is an adaptation to rationalist thinking (the dominant understanding in
evaluation) and, as such, inconsistent with a complexity understanding. Rather than expand-
ing the discussion on the merit of the distinction, in this article we want to investigate further
the character of complex situations. The dimensions of uncertainty and disagreement empha-
size the undefined character of a problem in its current form. They help to explain why people
cannot easily define solutions or come to an agreement on a desired future state. They also
provide some general guidelines on how to deal with such problems. However, the historic
context of the problem is not yet taken into account.
Many problems continue to exist, even in cases considered as highly urgent and where
generations of interventions have already taken place, if there is no understanding of the
way problems have grown historically and are institutionally embedded in society. Kouévi
et al. (2013), for instance, describe how six decades of fishery management interventions
in Benin have failed to improve fishermen’s livelihoods. Co-evolutionary theories on inno-
vation processes that emphasize the integration of technological and social change help to
explain the perpetuation of an undesirable situation by showing how problems are embed-
ded in society (Grin and Van Staveren, 2007). Local and supra-local institutional barriers
hinder, for example, smallholder farmers in West-Africa from taking advantage of availa-
ble technological options (Hounkonnou et al., 2012; Totin et al., 2012). Choices made in
the past guide current actions even if they are far from desirable, since the ‘institutionally
embedded practices pre-suppose and re-enforce each other’ (Grin and Van Staveren, 2007:
137). Around social practices, systems have emerged that entail the whole configuration of
technology, knowledge, infrastructure, symbolic values and role division (Rip and Kemp,
1998; Tukker et al., 2008). Existing systems are stabilized through various self-reinforcing
or lock-in mechanisms, such as scale economies and sunk investments in machinery and
infrastructure (Arthur, 1989). Also, shared beliefs, discourses and power relations stabilize
existing systems. These lock-in mechanisms (see Box 1) thus create path dependence
which means that current and future states, actions or decisions depend on the path of pre-
vious states, actions or decisions (Page, 2006). This thinking has increasingly been used to
analyse socio-economic developments as well as political processes since the ground-
breaking work of Arthur (1994; see also David, 1994; Page, 2006; Pierson, 2000).
Box 1. Example of a lock-in.
A good example of a lock-in is the carbon lock-in as described by Unruh (2000), a vicious cycle of govern-
ments allowing new capacity to generate electricity with fossil fuels, expansion of the grid, increased avail-
ability of cheap electricity that encourages consumption, the development of new applications and end-use
technologies, and governments again approving of more capacity to meet the expanding demand. National
governments do not recognize this lock-in, manifested in their ignorance of laws that inhibit carbon-saving
technologies and the problems that they have in removing subsidies for fossil fuel industries (Unruh, 2000).
102 Evaluation 21(1)
Figure 2. Three dimensions of complex problems.
Building on scholars who introduced the concepts of lock-in and path dependence in inno-
vation studies, Geels (2004) discerns three mechanisms that provide stability to a socio-
technological system. First, the rules in the system, including cognitive heuristics, normative
rules and formal regulations, provide stability because they guide actions and perceptions.
Established practices of fishing, farming, development cooperation and evaluation are closely
associated with these rules. They are deep structures on which knowledgeable actors draw in
their actions and therefore provide the context of action (Giddens, 1984). In their actions,
they also adapt or redefine these structures. Given that rules are aligned within a system, it is
hardly possible to change one rule without altering others. Second, the mutual dependence
between actors (persons or organizations) contributes to the stability of a system. Once net-
works have formed around a policy issue, market or programme, the actions of the actors
involved, like the suppliers, traders and buyers from a value chain, become increasingly
intertwined. In such networks, actors can become locked into their relationships, thus block-
ing new ideas from outside and discouraging other potentially fruitful collaborations (Klein
Woolthuis et al., 2005). Finally, long lifetimes of the material components of a system add to
its stability, as well as the investments sunk in infrastructure and the complementarities of
material components of technologies. These mechanisms explain why technological, scien-
tific, political and market developments tend to follow certain trajectories and consist of
incremental innovation merely.
So in many situations, it is the structuredness of a problem – or in other words its embed-
dedness in stabilized systems – in addition to its undefined character that adds to the difficulty
of solving it. Therefore, we propose to add a third analytical dimension to the well-known two
dimensions that emphasizes the structured rather than the unstructured character of complex
problems. In our definition, complex problems display high levels of uncertainty and disa-
greement on the goals and underlying values, as well as a high level of stability in a system
with many interlinked rules, interdependent actors and material components that favour exist-
ing, undesirable but normalized practices (see Figure 2).
Arkesteijn et al.: The need for reflexive evaluation approaches in development cooperation 103
Poverty as a complex problem
In a growing body of literature on development cooperation, complexity has become a central
concept (Jones, 2011; Nobuko, 2010; Ramalingam, 2013; Ulrich, 2010). Major issues, such as
poverty in developing countries, are addressed from a complexity perspective. Interestingly
enough, we see that many practitioners and scholars (like Jones, 2011) refer to the complexity
of solving poverty and the related management approaches rather than to the complexity of the
poverty problem and its causes.
Poverty in developing countries has been studied by a range of researchers with different
points of view about its causes. Among them are scholars who mention causes of poverty that
cannot, or can hardly, be influenced by human beings, such as geographical characteristics for
instance, being land-locked (Huntington, 1968), the availability of natural resources (Diamond,
1997) or a combination of both (Collier, 2007). Others refer to the absence often formulated
as ‘lack of’ of strong and credible institutions that allow people access to capital, asset accu-
mulation, investing, and/or claiming their rights like police, judiciary system and the registra-
tion of property (De Soto, 2000), contractual rights and an enforced rule of law (Landes, 1998;
Moyo, 2010).
Several influential scholars have analysed poverty as resulting from various existing, inter-
linked rules and institutions that sustain poverty and hence show the systemic stability in
which poverty is embedded, without necessarily defining it as such. Cleaver (2005), for
instance, refers to institutions that exclude poor people from having access to land, capital,
health and education services, and social networks, that keep them poor. Acemoglu et al.
(2006, 2012) introduce the term extractive institutions, referring to institutions through which
the elite of a country extracts resources and/or labour, or monopolizes the most lucrative busi-
nesses. The authors explain why societies will not naturally gravitate toward inclusive institu-
tions, like enforcement of property rights, constraints on the elites of a country and equal
opportunities as follows: ‘powerful groups will often opt for institutions that do not provide
any rights to the majority of the population so that they can extract resources or labour from
them, or monopolize the most lucrative businesses’ (Acemoglu et al., 2006: 21). Wade (2005)
presents a worldwide system perspective and provides an explanation of poverty at country
level in terms of a lock-in. Poor countries’ economies are seriously constrained by limited
export revenues which squeeze out domestic use of resources for reducing poverty and pro-
viding education, health and civil services. Governmental authorities are likely to become
more corrupt in response, military and police finance themselves through predation and crimi-
nality, the sinews of state power erode, resulting in a fragile state and conflict, national and
international companies invest even less in production and growth, which again reduces the
resources for services; a vicious cycle. Moyo (2010) and Easterly (2006a) follow a similar
chain of thought, and add aid like bi- and multi-lateral budget support as a major factor sus-
taining poverty and keeping corrupt governments in place, whereas, according to others, aid1
in general is not substantial enough (Banerjee and Duflo, 2011) to play that role. Essers (2011:
2) uses the concepts of lock-in and path dependence and links these concepts to a growing
body of literature on conditions and mechanisms that seem to keep ‘developing countries in a
self-sustaining state of underdevelopment, so called poverty traps’. According to him, two
models are prevalent in the current literature. The critical threshold model assumes the exist-
ence of a barrier that can only be overcome with a ‘big push’ (i.e. large amounts of investments
and interventions at once that can thus help countries to escape from this poverty trap). The
104 Evaluation 21(1)
Millennium Goals (United Nations, 2013)2 provide an example of this type of thinking. The
other model focuses on interactions and arrangements of economic agents that cause poverty
to persist – even in the case of a big push – in what is called the dysfunctional institution model
(Acemoglu et al., 2005; Hoff, 2003). Again, another explanation is provided by authors like
Khor (2008) who shows how unfavourable national and international financial and trade poli-
cies and their implementation, which result in overproduction in developed countries com-
bined with farmers in developing countries withdrawing, keep countries in poverty traps.
Notwithstanding their diverse explanations of poverty, the studies addressing existing insti-
tutional factors provide good reasons to regard poverty as a complex problem with various
systemic mechanisms sustaining it.
Complexity in development evaluation
Numerous small and large donor funded programmes aim at redressing poverty and stimu-
lating development. Both public and private donor organizations in the North have a keen
interest in assessing the performance of the programmes they support, not least because
they feel the need to secure continued support from their respective constituencies (e.g.
tax-payers and benefactors). Hence, evaluation is a very prominent activity in the interna-
tional development field. Having defined poverty as a complex problem, it becomes rele-
vant to ask how the evaluation approaches used in development settings address complexity,
and especially whether they address systemic stability. In this section, we briefly discuss
the most prominent approaches and trends in the evaluation of development programmes
with the view of establishing how they deal with systemic stability as an important dimen-
sion of complexity.
For quite some time, development evaluation has been dominated by instrumental
approaches that assume that development and change can be planned in advance, based on
rational planning and problem solving procedures as well as predictive expert knowledge
about causes and effects in human behaviour and societal dynamics (see Long and Van der
Ploeg, 1989). This kind of thinking is exemplified by the ‘logical framework approach’ or
logframe (see BOND, 2003; NORAD, 1999; Örtengren, 2004; Shields, 1993). The logical
framework is a planning tool that often is imposed on development interventions, and requires
practitioners to describe projects and programmes in terms of:
the ultimate development objectives to which a project/programme is expected to
contribute;
the specific purposes that are to be realized by the project/programme;
the specific outputs and results that are expected in a certain time frame;
the activities through which outputs are achieved;
the measurable or objectively verifiable indicators that will be used to assess progress;
the data and information that will be collected to verify progress;
the inputs needed; and
the core assumptions about the wider environment underlying the project (Ringsing and
Leeuwis, 2008).
This way of thinking and operating has been criticized strongly for failing to recognize
complexity. While this approach suggests that critical reflection on assumptions about the
Arkesteijn et al.: The need for reflexive evaluation approaches in development cooperation 105
relations between an intervention and changes (‘effects’) is needed during implementation and
between project cycles, the overall idea is that uncertainty can and should be reduced. One
major critique hence is that uncertainty is not regarded as an inherent feature of system change
that should be acknowledged.
Moreover, the logical framework renders disagreement and conflicts invisible as the focus
is on what is ‘agreed’ during project formulation, even in the approaches embracing complex-
ity (as in Rogers, 2008). Numerous authors have described development projects as ‘arenas of
struggle’ in which objectives, goals and purposes are contested and in which interest collide
(see e.g. Crehan and Von Oppen, 1988; Long and Van der Ploeg, 1989).
The phenomenon of systemic stability and lock-in, re-inforced through inter-dependencies,
power configurations, prevailing rules, institutional set-ups and (bio-)material configurations,
is not acknowledged either in this mode of thinking. Essentially, society is seen as maleable
and amenable to rational design, whereby development projects are central triggers and
‘causes’ in fostering change and development. The latter is also reflected in the recently
renewed interest in randomized controlled trials as a method for studying the efficacy of
development interventions (White, 2011).
The mode of thinking underlying the instrumental approach described above has also been
labelled ‘hard systems thinking’ (Checkland, 1993). Evaluation approaches that emerged in
response to this, start from the recognition that people have different values and perceptions
of reality and hence of the interrelations in a system. The generation of knowledge is not seen
as a role of experts only, but as a collective process including different stakeholders. Within
development, many evaluation approaches that emerged from the 1990s onwards are inspired
by this constructivist perspective such as soft systems methodology (Checkland, 1993), dia-
lectical inquiry (Dick, 1997), Participatory Monitoring and Evaluation (Estrella et al., 2000;
Guijt, 1999) and Most Significant Change (Dart and Davies, 2003) . Typically, these approaches
recognize the existence and importance of different perspectives and values in shaping devel-
opment outcomes, and often evaluation approaches are geared to stimulating interaction and
learning as a means of dealing with such ambiguities and disagreements. The exchange of
values and perspectives (for instance, in the form of stories) is expected to facilitate learning.
Along these lines, constructivist approaches like the Fourth Generation evaluation (Guba and
Lincoln, 1989), Responsive Evaluation (Stake, 1983), and Learning Histories (Kleiner and
Roth, 1997) explore the grounds for disagreement in order to shape the future agenda of pro-
jects and programmes.
A learning-oriented approach that explicitly addresses issues of uncertainty and emergence
is Developmental Evaluation (Patton, 2011). The main idea of this approach is to guide col-
laborative action in innovative initiatives that face high uncertainty in a process of experimen-
tation, co-creation and social learning. Fundamental changes of the intervention theory will
necessarily occur in order to accommodate changing conditions and to respond to emergent
phenomena. More generally, we see that widely used learning-oriented evaluation approaches
are geared towards assessing whether ex-ante formulated theories of change or impact path-
ways (i.e. explications of the assumed causal chain from intervention activities to outputs,
outcomes and impacts) need to be adapted in light of emergent dynamics and progressive
insights (see e.g. Mayne and Stern, 2013). These approaches also recognize complexity in the
sense that impacts are seen to emerge from multiple influences, in which programmes and
projects may play a more or less significant role that can be assessed through evaluation in
terms of greater or lesser degrees of causality (Mayne and Stern, 2013). This interest in
106 Evaluation 21(1)
multi-causality is also reflected in Most Significance Change (Dart and Davies, 2003), an
approach geared towards identifying stories about intended and unintended changes that have
come about, without making any a-priori causal assumptions about the relation between the
interventions and outcomes.
In all, these more constructivist or ‘soft-systems’ approaches are more sensitive to issues
like disagreement, uncertainty and emergence. In line with this they position evaluation as a
key activity to deal with these dimensions of complexity. As is perhaps most vividly expressed
in the term Most Significant Change (as against ‘Most Significant Stability’), however, these
approaches tend to be optimistic about the prospect of orchestrating or triggering change, and
hardly pay attention to the systemic lock-ins that are re-produced through dominant power
configurations and existing rules and infrastructures. Moreover, the frequent emphasis on ex-
ante formulated impact pathways and programme theories of change, still reflects a significant
degree of confidence in the possibility to intervene and learn so as to achieve socially desira-
ble impacts by means of projects and programmes. In fact, many of the above mentioned
approaches are designed to evaluate clearly demarcated interventions and programmes, and
much less geared towards assessing change in systems with less clearly defined boundaries in
time and space (Kouévi et al., 2011, 2013).
The third category of approaches emerged as a critique on both hard and soft systems
approaches. Here the proposal is that evaluation methodologies should challenge existing
power relationships or conflicts that are built into the structure of society, and should address
broader social and economic relationships that reproduce the status quo (Midgley, 2007;
Ulrich, 2010). From this third group, critical system heuristics (CSH) (Midgley, 1992;
Reynolds, 2007; Ulrich, 2010; Ulrich and Reynolds, 2010) and critical system thinking
(Boyd et al., 2007) entered development evaluation. Here it is argued that politics are
involved in the definition of projects and programmes. When defining the boundaries of the
project, specific issues and actors are either included or excluded. Thus, choices about sys-
tem boundaries build on values and need to be reflected upon. What CSH notably does is to
explicate the judgements about boundaries and consider alternative boundaries by identify-
ing 12 key boundary decisions with the aid of a heuristic in order to understand the conse-
quences of interventions in a specific situation and to prevent social inequities increasing.
Thus, this evaluation approach places power configurations as a stabilizing mechanism at
the forefront.
In conclusion, we have seen that approaches to development evaluation can be linked to
different kinds of systems thinking. The short overview shows that they place different dimen-
sions of complexity at the core of evaluation practice. The instrumental or hard systems
approaches largely ignore the existence of disagreement, but to some extent acknowledge the
existence of uncertainty. However, they aim to reduce it and eliminate it eventually. Soft sys-
tems or learning-oriented approaches do regard uncertainty, emergence and coincidence as
fundamental characteristics of social change, and also take into account disagreement as a key
dimension of complexity. Here, evaluation is a significant way to enhance learning in an effort
to deal with uncertainty and disagreement (see Leeuwis, 2000). Both types of system thinking,
however, underrate the complexity dimension of systemic stability, and tend to ignore the
interdependencies, power configurations, institutional set-ups and bio-material configurations
through which the status quo is reproduced. An apparent exception is CSH. This approach
aims to stimulate critical reflection on system boundary decisions and in this way challenges
the stabilizing mechanisms of power relations and explores alternative action areas. It does
Arkesteijn et al.: The need for reflexive evaluation approaches in development cooperation 107
not, however, address the stabilizing mechanisms additional to power relations. Moreover, for
promoting fundamental change, we think it is important to go beyond critical reflection on
power relations. In order to address systemic stability more systematically, we discuss in the
next section what kind of evaluation methodologies would enable identifying stabilizing
mechanisms while changing the embeddedness of social practices in relations, rules and mate-
rial artifacts.
Reflexive evaluation for development
Results of interventions and evaluations that address uncertainty and disagreement may be
short-lived if the mechanisms that provide stability to the current system remain untouched.
The core challenge is to find ways to bring about diversions from pre-existing lock-ins.
This means that complex problems need a freeing-up of formal and informal rules and rela-
tions that guide actions and practices. Simultaneously, ways of thinking, of problem solv-
ing, of managing resources and people, and of planning, need to be reconsidered because
they are in many ways part of the problem (Beck et al., 1994). The management strategy
often proposed is to challenge the stabilizing mechanisms of practices by stimulating
inquiry, dialogue, interactive learning and learning by doing (Geels and Schot, 2007; Voβ
et al., 2006). To this end, the elaboration of complex problems into practical options for
action must include a reflexive perspective; that is: ‘a critical scrutiny of things that are
usually taken for granted, in such a way that their historically grown self-evidence (path
dependency) is challenged’ (Loeber et al., 2007: 84). Evaluation approaches to evaluate
policy or other interventions should include such a reflexive perspective as well (Van
Mierlo et al, 2010a, 2010b).
In Western Europe, evaluation approaches appear (at times under the heading of ‘reflex-
ive’) with the above mentioned characteristic of reflexivity (Grin and Van Staveren, 2007;
Regeer et al., 2009; Taanman, 2014; Van Mierlo et al., 2010a). Such reflexive approaches have
emerged to support the design and analysis of processes of system change towards a sustain-
able development in, among others, the energy and building sectors, and in agriculture and
welfare. An example of an approach that has proliferated in a diversity of domains is Reflexive
Monitoring in Action,3 see Box 2. While reflexive approaches have not been used in a devel-
opment cooperation context yet, they have attracted interest. Since we think they are promis-
ing and interesting for addressing the complexity of international development issues, we now
turn to exploring how they would be able to address systemic stability.
Box 2. Reflexive Monitoring in Action.
Reflexive Monitoring in Action (RMA) is an interactive methodology to encourage reflection and
learning within groups of diverse actors that seek to contribute to system change in order to deal
with complex problems. It builds on the assumption that recurrent collective reflection on the current
system (barriers as well as opportunities) helps to stimulate collective learning and to design and
adapt targeted systemic interventions. While doing so, these system innovation initiatives develop new
or change local rules, practices and relations within the network of actors involved. This takes place
in the muddiness of everyday struggles of change trajectories (Van Mierlo et al., 2010a, 2010b, 2010c).
(Box Continued)
108 Evaluation 21(1)
Key to this methodology is first, recurrent reflection on the institutional setting in relation to long
term ambitions and concrete actions and their effects and second, the support of system learning and
institutional changes while evaluating these ex-durante. Reflexivity is the outcome; the emergent property
of an intervention programme or bottom-up innovation initiative developing new coordinated practices
while the rules of the game change along in the process of designing new systems.
RMA builds on the premise that while the contribution of a single system innovation initiative to a
long, capricious system innovation processes cannot be assessed, it is possible to characterise the actions
of a project and their outcomes in terms of relevance for system innovation. The ongoing innovation
process is evaluated with the aid of middle-range theories about processes of societal change, including
communication, learning, network building and conflict management as well as sociological and institu-
tional theories about system innovation and social practices specifically. Originally, RMA was developed
for small scale initiatives that aim to contribute to system change. Over the past decade, being applied in
around 20 projects and programmes with groups of diverse actors, the methodology has developed into
a coherent body of basic starting points, principles and intervention strategies. It has been taken up by a
range of European and international research and training programmes.
Monitoring activities are an integral part of the change initiative; the appointed reflexive monitor,
whether a hired person, or someone from the project team, usually starts at the moments of interaction,
such as regular team meetings to observe how the ambition to change is articulated, whether learning is
taking place and ambitious collaborative actions are being designed and carried out. The reflexive moni-
tor’s frame of reference is the particular system innovation ambition i.e. the drive to develop new rules,
relations and material artefacts as articulated by the innovators, if necessary with the aid of the monitor
(see Van Mierlo et al., 2010). Challenges encountered on the pathway of change in the form of resistance
to change that relates to the institutional setting of the innovation initiative, define the activities of the
reflexive monitor. Hence, the reflexive monitor is not only an observer but also a facilitator and a sparring
partner, with sufficient distance to take a critical stance if needed. In a diversity of ways, he or she encour-
ages participants to reflect upon the relationships between the project activities and results and its insti-
tutional setting, and the ambition to change in both short-term actions and long-term goals and future
perspectives. In this way, RMA addresses the mechanisms that provide stability to the unsustainable
system.
To start with, the stabilizing mechanisms should not be seen just as ‘contextual factors’ to be
taken into account to explain the outcomes of interventions as in the realistic evaluation
approach of Pawson and Tilley (1997). Interventionists, change agents and other initiators of
change themselves are part and parcel of an institutional setting that needs to change, but tend
to relapse into old roles and dominant practices when intervening in complex situations, thus
contributing to the systemic stability. A reflexive perspective in evaluation hence means evalu-
ating programmes or initiatives that seek to contribute to system change in order to deal with
complex problems while supporting the change process among initiators as well as those who
resist change. Basically, it should encourage groups of diverse actors to reflect on the rules and
relations underlying current practices in order to induce institutional changes (Van Mierlo et al.,
2010a, 2010b, 2013). Hence, reflexive evaluation activities need to be firmly embedded in
initiatives that seek change in order to overcome path dependency.
In order to stimulate an interactive learning process, which is key to any fundamental
change process, reflexive evaluation methdologies should go beyond the often used theories
about organizational learning in order to pass the boundaries of organizations. They need theo-
ries to be able to analyse whether and to what extent collective learning towards system inno-
vation is taking place among the initiators, their partners, supporters and constituencies, and
Box 2. (Continued)
Arkesteijn et al.: The need for reflexive evaluation approaches in development cooperation 109
other network contacts in terms of new insights as well as new actions and practices. As an
example, we present the concept of system learning. The main entry point for the conceptual-
ization of system learning is the aim to prevent initiators from relapsing into old patterns of
thinking and acting, while they walk on and try out new paths, rather than to increase system
understanding before defining or adapting actions. Hence, system learning is used to evaluate
whether the current, relatively stable set of social structures (which are by and large guided by
‘unsustainable’ orientations) are challenged as well as changing. Whether system learning is
taking place is assessed along the following dimensions (Van Mierlo et al. 2010a, 2013):
a) Recognition of relationships
This happens if initiators explicate, define and discuss the main systemic innovation barriers to
sustainability and their interrelationships, and hence recognize complexity, multi-causality and
unexpected results. This dimension is close to system thinking (Patton, 2011; Senge, 1990;
Williams and Hummelbrunner, 2011), but goes beyond it with a reflection on the wider system.
b) Redefinition of barriers into windows of opportunity
While reflecting on innovation barriers and the actors that reproduce them, participants may
question these ‘given’ conditions and start seeing them as changeable (i.e. redefine them into
windows of opportunity). According to Loeber et al. (2007), such system learning takes place
if actors challenge and redefine the very structures that hinder their aspirations for more sus-
tainable practices; that is, they regard the relationships between the structures in which they
operate and their own practices in a new light.
c) Design of radical network activities
Radical collaborative options challenge the stabilizing mechanisms of the socio-technological
system. Initiators may check whether the intended activities are radical, or reorient the direc-
tions chosen previously. This dimension links closely to the work of Argyris and Schön (1978)
on single and double loop learning who conceive of thinking and acting as intrinsically linked,
while supplementing it with a focus on systemic stability.
To enable the evaluation of changes in terms of their relevance for system innovation in the
longer term, it seems wise to use and adapt middle-range sociological theories on learning, insti-
tutional change, and system innovation that could add to the currently used organizational learn-
ing concepts and ecological system thinking in the current system approaches in evaluation.
Giddens’ (1984) notion of agency, for instance – the ability to choose between maintaining the
situation and bringing about institutional change – is helpful to evaluate how initiatives for
change develop and make use of their agency to develop into systems of their own while con-
tributing to changes in the incumbent systems. The work of Geels presented in section 2 could
help to distinguish which mechanisms of systemic stability are challenged. In general, with such
middle-range theories markers can be developed to characterize ‘progress’ in terms of learning
and institutional change relevant for system innovation. In addition, they may help to develop
concrete ideas about how to promote such change in and through evaluation activities.
While engaging the actors and their contacts closely in the evaluation, reflexive approaches need
to go beyond participatory approaches of involving ‘stakeholders’ and seeking mutual understanding.
110 Evaluation 21(1)
Instead, the central focus of reflection is on the interaction between the initiative and the institutional
setting. Whereas in Developmental Evaluation such reflection and learning are stimulated with
inquiry frameworks that are chosen on the basis of the needs of the users, reflexive evaluation would
support and assess innovation initiatives from the perspective of system innovation. For the evalua-
tion, it would mean tracking relevant learning and changes in rules and relations over a long period of
time, providing feedback and stimulating reflection on and adaptation of the actions in the name of
the initiative in light of the ongoing change processes within an innovation initiative, at its boundaries
and in the wider context. Hence, it would essentially be an adaptive evaluation approach, responding
to institutional challenges along the way with relevant and diverse activities and tools.
Conclusion
Policy makers and practitioners in the development sector increasingly address problems from
a complexity perspective. In this article, we argued that while complex problems are generally
defined in terms of disagreement and uncertainty, they should also be seen as problems that
are structural because of diverse systemic stabilizing mechanisms: aligning rules, growing
interdependencies and interweaving material system components.
Many development problems essentially are complex problems in all three dimensions,
given the causes of their perpetuating existence. Regarding poverty, development scholars
mention, among others, extractive institutions, corrupt governments and international rela-
tionships increasing the developing countries dependency; obvious signs of a lock-in. As a
result, intervention programmes trying to enhance local food production through research
and extension, for instance, or value chain and multi-stakeholder approaches may fail. The
third complexity dimension calls for other ways to intervene and evaluate. It requires
donors, interventionists and supposed beneficiaries to move off the beaten track and start
challenging the mechanisms that sustain poverty while designing new directions of
solutions.
In support of such ventures, a reflexive evaluation approach provides promising ideas,
principles and concrete tools that may well complement current system approaches in devel-
opment evaluation. Reflexive evaluation approaches build on the assumption that dominant
problematic social practices are historically grown and institutionally embedded and need to
be challenged as such. Albeit in different ways, they essentially aim for stimulating forms of
fundamental social learning along long-term trajectories of system change. They are ways to
help an intervention programme or innovation initiative to develop new practices while the
rules of the game change in the process of designing new systems. Key to such methodologies
is, first, recurrent reflection on the institutional setting in relation to long-term ambitions and
concrete actions and their effects, and, second, the support of system learning and institutional
changes while evaluating these ex-durante. In conclusion, reflexive evaluation approaches,
like Reflexive Monitoring in Action, may provide valuable means by which evaluators are
able to help actors reflect upon and challenge the dynamics providing systemic stability while
designing and enacting new practices, and for that reason seem worth being used in
development.
Funding
This research received no specific grant from any funding agency in the public, commercial, or not-for
profit sectors.
Arkesteijn et al.: The need for reflexive evaluation approaches in development cooperation 111
Notes
1. Banerjee and Duflo (2011) doubt this since aid hardly ever constitutes more than 5–12 percent of
government budgets.
2. This model underlies much of the rhetoric surrounding the Millennium Development Goals, with
Jeffrey Sachs (2005) as strong proponent, and William Easterly (2006b) as an equally strong oppo-
nent. With only limited time to go to 2015, it is becoming evident that in some places this big push
(see Easterly, 2006b) has worked, whereas in other places it has not (United Nations, 2013).
3. The Reflexive Monitoring in Action guide can be downloaded from http://www.wageningenur.nl/
nl/show/Reflexive-Monitoring-in-Action.htm
References
Acemoglu D, Johnson S and Robinson JA (2005) Institutions as a fundamental cause of long-run
growth. In: Aghion P and Durlauf SN (eds) Handbook of Economic Growth Vol. 1A. Amsterdam:
North-Holland, 385–472.
Acemoglu D, Johnson S and Robinson JA (2006) Understanding prosperity and poverty: geography,
institutions and the reversal of fortune. In: Banerjee AV, Bénabou R and Mookherjee D (eds)
Understanding Poverty. New York: Oxford University Press USA, 19–36.
Acemoglu D and Robinson JA (2012) Why Nations Fail. The Origins of Power, Prosperity and Poverty.
New York: Crown Publishing.
Argyris C and Schön D (1978) Organizational Learning: A Theory of Action Perspective. Reading,
MA: Addison-Wesley.
Arthur B (1989) Competing technologies, increasing returns, and lock-in by historical events. The
Economic Journal 99: 116–31.
Arthur WB (1994) Increasing Returns and Path Dependence in the Economy. Ann Arbor, MI: University
of Michigan Press.
Banerjee AV and Duflo E (2011) Poor Economics. Barefoot Hedge-Fund Managers, DIY Doctors and
the Surprising Truth about Life on Less Than $1 a Day. London: Penguin Books.
Beck U, Giddens A and Lash S (1994) Reflexive Modernization: Politics, Tradition and Aesthetics in
the Modern Social Order. Cambridge: Polity Press.
BOND (2003) Logical Framework Analysis. Guidance Notes No. 4. London: BOND Networking for
International Development.
Boyd A, Geerling T, Gregory WJ, Kagan C, Midgley G, Murray P and Walsh MP (2007) Systemic
evaluation: A participative, multi-method approach. Journal of the Operational Research Society
58: 1306–20.
Checkland P (1993) Systems Thinking, Systems Practice: Includes a 30-Year Retrospective. New York:
John Wiley and Sons.
Cleaver F (2005) The inequality of social capital and the reproduction of chronic poverty. World
Development 33(6): 893–906.
Collier P (2007) The Bottom Billion: Why the Poorest Countries Are Failing and What Can Be Done
about It. Oxford: Oxford University Press.
Crehan K and Von Oppen A (1988) Understandings of development: an arena of struggle. The story of
a development project in Zambia. Sociologia Ruralis 28: 113–45.
David PA (1994) Why are institutions the ‘carriers of history’? Path dependence and the evolution
of conventions, organisations and institutions. Structural Change and Economic Dynamics 5(2):
205–20.
Dart J and Davies R (2003) A dialogical, story-based evaluation tool: the most significant change tech-
nique. American Journal of Evaluation 24(2): 137–55.
De Soto H (2000) The Mystery of Capital: Why Capitalism Triumphs in the West and Fails Everywhere
Else. New York: Basic Books.
112 Evaluation 21(1)
Diamond J (1997) Guns, Germs and Steel: The Fates of Human Societies. New York: W. Norton & Co.
Dick B (1997) Option one-and-a-half. URL (consulted 2 July 2013): http://www.aral.com.au/resources/
options.html.
Easterly W (2006a) The White Man’s Burden: Why the West’s Efforts to Aid the Rest Have Done So
Much Ill and So Little Good. London: Penguin Books.
Easterly W (2006b) The big push déjà-vu: a review of Jeffrey Sachs’s ‘End of poverty: Economic pos-
sibilities for our time’. Journal of Economic Literature 44(1): 96–105.
Essers D (2011) A review of Brian Arthur’s increasing returns and path dependency in the economy.
Applications in economic development theory. Paper for the course Advanced Macroeconomics.
Antwerp, Belgium: University of Antwerp.
Estrella M, Blauert J, Campilan D, Gaventa J, Gonsalves J, Guijt I, Johnson DA and Ricafort R
(eds) (2000) Learning From Change: Issues and Experiences in Participatory Monitoring and
Evaluation. London: Intermediate Technology Publications.
Geels FW (2004) From sectoral systems of innovation to socio-technical systems: insights about dynam-
ics and change from sociology and institutional theory. Research Policy 33(6/7): 897–920.
Geels FW and Schot J (2007) Typology of sociotechnical transition pathways. Research Policy 36:
399–417.
Giddens J (1984) The Constitution of Society: Outline of the Theory of Structuration. Cambridge: Polity
Press.
GIZ (2011) Conference on Systemic Approaches in Evaluation. URL (consulted December 2010):
http://www.evaluation-conference.de/en/index.html.
Grin J and Van Staveren A (2007) Werken aan systeeminnovaties. Lessen uit de praktijk van
InnovatieNetwerk [Working on System Innovations. Lessons from the Practice of Innovation
Network]. Assen, the Netherlands: Van Gorcum.
Guba EG and Lincoln YS (1989) Fourth Generation Evaluation. Newbury Park, CA: SAGE.
Guijt I (1999) Participatory Monitoring and Evaluation for Natural Resource Management and
Research. Best Practice Guidelines, Socio-Economic Methodologies. Chatham and London:
Natural Resource Institute / DFID.
Hisschemöller M and Hoppe R (1996) Coping with intractable controversies: the case for problem
structuring in policy design and analysis. Knowledge and Policy: The International Journal of
Knowledge Transfer and Utilization 8(4): 40–60.
Hoff K (2003) Paths of institutional development: a view from economic history. World Bank Research
Observer 18(2): 205–26.
Hounkonnou D, Kossou D, Kuyper TW, Leeuwis C, Nederlof S, Röling N, Sakyi-Dawson O, Traoré M,
Van Huis A (2012) An innovation systems approach to institutional change: smallholder develop-
ment in West Africa. Agricultural Systems 108: 74–83.
Hummelbrunner R (2011) Systems thinking and evaluation. Evaluation 17(4): 395–04.
Huntington S (1968) Political Order in Changing Societies. New Haven, CT: Yale University Press.
Jones H (2011) Taking responsibility for complexity. How implementation can achieve results in the
face of complex problems. ODI Working Paper 330. London: ODI.
Khor M (2008) The Impact of Trade Liberalization on Agriculture in Developing Countries: The Case
of Ghana. Penang: Third World Network.
Klein Woolthuis RJA, Gilsing V and Lankhuizen M (2005) A system failure framework for innovation
policy design. Technovation 25(6): 609–19.
Kleiner A and Roth G (1997) Learning histories: a new tool for turning organizational experience into
action. URL (consulted 11 July 2007): http://ccs.mit.edu/lh/21CWP002.html and http://ccs.mit.
edu/lh/.
Kouévi AT, Van Mierlo B and Leeuwis C (2011) Repetitive discrepancy between espoused and in-use
action theories for fishery intervention in Grand-Popo, Benin. International Journal of Learning
and Change 5(2): 114–38.
Arkesteijn et al.: The need for reflexive evaluation approaches in development cooperation 113
Kouévi AT, Van Mierlo B, Leeuwis C and Vodouhè S (2013) The design of a contextualized responsive
evaluation framework for fishery management in Benin. Evaluation and Program Planning 36(1):
15–28.
Landes DS (1998) The Wealth and Poverty of Nations. Why Some Are So Rich and Some So Poor. New
York/London: W.W. Norton & Co.
Leeuwis C (2000) Re-conceptualizing participation for sustainable rural development: towards a nego-
tiation approach. Development and Change 31: 931–59.
Loeber A, Van Mierlo B, Grin J and Leeuwis C (2007) The practical value of theory: conceptual-
ising learning in the pursuit of a sustainable development. In: Wals A (ed.) Social Learning
Towards a More Sustainable World. Wageningen, the Netherlands: Wageningen Academic
Publishers, 83–98.
Long N and Van der Ploeg JD (1989) Demythologizing planned intervention: an actor perspective.
Sociologia Ruralis 29 (3–4): 226–49.
Mayne J and Stern E (2013) Impact evaluation of natural resource management research programs: a
broader view. ACIAR Impact Assessment Series Report No. 84. Canberra: Australian Centre for
International Agricultural Research.
Midgley G (1992) Pluralism and the legitimation of systems science. Systems Practice 5: 147–72.
Midgley G (2007) Systems thinking for evaluation. In: Williams B and Imam I (eds) Systems Concepts
in Evaluation: An Expert Anthology. Point Reyes, CA: EdgePress, 11–34.
Mowles C (2014) Complex, but not quite complex enough: the turn to the complexity sciences in evalu-
ation scholarship. Evaluation 20(2): 160–75.
Moyo D (2010) Dead Aid. Why Aid Is Not Working and How There Is Another Way for Africa. London:
Penguin Books.
Nobuko F (ed.) (2010) Beyond Log Frame: Using Systems Concepts in Evaluation. Issues and Prospects
of Evaluations for international Development – Series IV. Tokyo: FASID.
NORAD (1999) The Logical Framework Approach (LFA). Handbook for Objectives-Oriented Planning,
4th edn. Norwegian Agency for Development Cooperation.
Örtengren K (2004) The Logical Framework Approach. A Summary of the Theory Behind the LFA
Method. Stockholm: SIDA.
Page SE (2006) Path dependence. Quarterly Journal of Political Science 1: 87–115.
PANOS (2009) How can complexity theory contribute to more effective development and aid evalu-
ation? Workshop Report. London: Panos. URL (consulted 19 May 2010): http://www.panos.
org.uk.
Patton MQ (2011) Developmental Evaluation. Applying Complexity Concepts to Enhance Innovation
and Use. New York/London: The Guilford Press.
Pawson R and Tilley N (1997) Realistic Evaluation. London: SAGE.
Pierson P (2000) Increasing returns, path dependence and the study of politics. American Political
Science Review 94(2): 252–67.
Ramalingam B (2013) Aid on the Edge of Chaos. Rethinking International Cooperation in a Complex
World. Oxford: Oxford University Press.
Regeer BJ, Hoes AC, Van Amstel M, Caron-Flinterman JF and Bunders JFG (2009) Six guiding princi-
ples for evaluating mode-2 strategies for sustainable development. American Journal of Evaluation
30(4): 515–37.
Reynolds M (2007) Evaluation based on critical systems heuristics. In: Williams B and Imam I
(eds) Systems Concepts in Evaluation: An Expert Anthology. Point Reyes, CA: EdgePress,
101–22.
Ringsing B and Leeuwis C (2008) Learning about advocacy: a case-study of challenges, every-
day practices. Evaluation: The International Journal of Theory, Research and Practice 14(4):
407–30.
114 Evaluation 21(1)
Rip A and Kemp R (1998) Technological change. In: Rayner S and Malcolm EL (eds) Human
Choice and Climate Change. An International Assessment, Vol. 2. Washington, DC: Batell
Press, 327–99.
Rittel H and Webber M (1973) Dilemmas in a general theory of planning. Policy Sciences 4: 155–69.
Rogers PJ (2008) Using programme theory to evaluate complicated and complex aspects of interven-
tions. Evaluation 14(1): 29–48.
Sachs J (2005) The End of Poverty: Economic Possibilities for Our Time. New York: Penguin Press.
Senge P (1990) The Fifth Discipline: The Art and Practice of the Learning Organization. New York:
Doubleday Currency.
Shields D (1993) What is the logical framework? Rural Extension Bulletin I 4: 15–20.
Snowden D and Boone ME (2007) A leader’s framework for decision making. Harvard Business
Review November: 69–76.
Stacey RD (1992) Managing the Unknowable: Strategic Boundaries between Order and Chaos in
Organizations. San Francisco, CA: Jossey Bass.
Stake RE (1983) Program evaluation, particularly responsive evaluation. In: Madaus G, Scriven M and
Stufflebeam D, Evaluation Models: Viewpoints on Educational and Human Services Evaluation.
Norwell, MA: Kluwer Nijhoff Publishing, 287–310.
Totin E, Van Mierlo B, Saïdou A, Mongbo R, Agbossou E, Stroosnijder L and Leeuwis C (2012)
Barriers and opportunities for innovation in rice production in the inland valleys. NJAS-Wageningen
Journal of Life Sciences 60–63: 57–66.
Tukker A, Charter M, Vezzoli C, Sto E and Munch Andersen M (2008) System Innovation for
Sustainability. Perspectives on Radical Changes to Sustainable Consumption and Production.
Sheffield: Greenleaf Publishing.
Ulrich HP (2010) Enhancing the effectiveness of international development: a systems approach.
Development in Practice 20(2): 251–64.
Ulrich W and Reynolds M (2010) Critical systems heuristics. In: Reynolds M and Holwell S (eds)
Systems Approaches to Managing Change: A Practical Guide. London: Springer, The Open
University.
United Nations (2013) The Millennium Development Goals Report 2013. New York: United Nations.
Van Mierlo B, Arkesteijn M and Leeuwis C (2010a) Enhancing the reflexivity of system innovation
projects with system analyses. American Journal of Evaluation 31(2): 143–61.
Van Mierlo BC, Regeer B, Van Amstel M, et al. (2010b) Reflexive Monitoring in Action. A Guide for
Monitoring System Innovation Projects. Boxtel, the Netherlands: Boxpress.
Van Mierlo BC, Janssen APHM, Leenstra FR and Van Weeghel HJE (2013) Encouraging system learn-
ing in two poultry subsectors. Agricultural Systems 115: 29–40.
Van Ongevalle J, Huysen H and Van Petegem P (2014) Dealing with complexity through actor-focussed
planning, monitoring and evaluation. Evaluation 20: 447–66.
Voß JP, Bauknecht D and Kemp R (2006) Reflexive Governance for Sustainable Development.
Cheltenham: Edward Elgar.
Wade RH (2005) Failing states and cumulative causation in the world system. International Political
Science Review 26(1): 17–36.
White H (2011) An introduction to the use of randomized control trials to evaluate development inter-
ventions. Working Paper 9. International Initiative for Impact Evaluation. New Delhi.
Williams B and Hummelbrunner R (2011) Systems Concepts in Action. A Practitioner’s Toolkit.
Stanford, CA: Stanford Business Books.
Zimmerman B and Glouberman S (2004) Complicated and complex systems: what would success-
ful reform of Medicare look like? In: Forest P-G, McIntosh T and Marchildon G (eds) Health
Care Services and the Process of Change. Toronto, Canada: University of Toronto Press,
21–53.
Arkesteijn et al.: The need for reflexive evaluation approaches in development cooperation 115
Marlèn Arkesteijn, MSc (Wageningen University), is a monitoring and evaluation practitioner and
researcher working in (innovation) programs in international cooperation, agriculture and nature
conservation.
Barbara van Mierlo, PhD, works as a sociologist at Wageningen University. Her main interest is in sys-
tem innovation trajectories of innovation networks towards sustainability.
Cees Leeuwis, PhD, is Professor of Knowledge, Technology and Innovation, Wageningen University.
He focusses on approaches to developing effective socio-technical innovations in networks, and col-
laboration between different disciplines.
