Content uploaded by Arwin Van Buuren
Author content
All content in this area was uploaded by Arwin Van Buuren on May 17, 2016
Content may be subject to copyright.
1
Beyond the pilot paradox
How the success conditions of pilots also hinder their up-scaling in climate governance
Paper for the workshop “Beyond experiments: Understanding how climate governance innovations
become embedded” (WG2) 25th – 27th April, 2016, Brighton
Draft – please do not cite
Arwin van Buuren*, Heleen Vreugdenhil~, Jitske Van Popering Verkerk*, Gerald Jan Ellen~, Corniel van
Leeuwen*, Bas Breman^.
*Erasmus University Rotterdam; ~Deltares; ^Alterra, Wageningen University
Dr. Arwin van Buuren, corresponding author
Associate Professor of Public Administration
Erasmus University Rotterdam
Room T17-13, P.O. Box 1738
3000 DR Rotterdam, the Netherlands.
vanbuuren@fsw.eur.nl
1. Introduction
In many countries, pilots are frequently used to explore and experiment novel ideas and innovative
approaches of climate change adaptation (Kivimaa et al. 2015). Pilots are perceived and often
acknowledged as an easy and safe manner to explore new pathways for fresh water supply, flood
protection or water retention, to find out whether new solutions or working methods can help to deal
with the consequences of climate change, before they are embedded in formal policies or implementation
programs.
Pilots are usually organized at the boundaries of the formal policy organizations, or even outside the
dominant policy regime. Within transition studies pilots are often defined as policy niches: ‘fringe spaces
for currently dominant technologies or alternatives to currently methods of governance’ (Kivimaa et al.
2015). They function as a safe area to develop novel ways of thinking and allow for exploring creative
methods and mobilizing new sources of knowledge. This distance is necessary to give the pilot some room
for innovation and exploration (Van Buuren & Loorbach, 2009). Furthermore, a pilot is often provided
with additional resources for exploration in terms of time, money and expertise to explore alternative
strategies and to develop new knowledge and insights. This freedom can consist of a reduced pressure to
achieve certain policy targets while rather focusing on developing new insights, and a more flexible
attitude towards permits and regulation.
Another important characteristic of pilot projects is the relative small scale (both geographically and
temporally) at which pilots are implemented. A small scale reduces risks due to limited impact and limited
loss of resources. Combined with an attitude that ‘failure’ is allowed, a small scale often makes it easier
for stakeholders to participate (Vreugdenhil et al. 2010). At the same time none of these characteristics
2
is written in stone, leading to a large variety in the design of pilot projects and related level of risks and
willingness to participate.
At the same time this distance, freedom, additional resources and small scale can hinder the broader
uptake of the pilots’ results and outcomes by the existing policy regime (Vreugdenhil, 2010). The distance
is sometimes designed at the start of the pilot. But the distance could also arise during the pilot, when
actors within the pilot (both organizations and professionals) rapidly learn and develop new insights, while
the different formal policy organizations and their practices are less connected and adjust themselves
much slower or even oppose to the pilots’ outcomes. The policy freedom and a small scale may lead to
non-representative or insufficient results. For example a water system may respond differently if the
measure would be larger or certain effects occur only long after the piloting period. Consequently, results
can be considered as insufficient proof for full implementation of the piloted approach. Moreover, the
argument of limited representativeness can even be used deliberatively to prevent further
implementation. The ultimate result of pilots in terms of generating regime change is due to these reasons
often quite low. A deeper cause is the limited attention for the question how to anchor pilots in the
existing policy regime. In addition, innovative pilots may very well contradict the ‘good virtues’ of the
public policy domain (Duijn, 2009), such as reliability, stability, prudence, etc, potentially harming the
functioning of formal policy organizations.
This brings us to the “paradox of pilots”: the conditions that are necessary to give a pilot room to
experiment and to learn (and thus come to innovative successful results), also seem to constitute the main
barriers for the broader uptake and translation of its results in real policy change. The results do not fit
within the policy regime and due to the distance between pilot and policy regime, the reception of the
results is not prepared at all.
In this paper we explore this potential paradox. We bring together a couple of climate adaptation pilots
we evaluated during the last couple of years. Within these cases we analyse whether and how the paradox
is apparent and thus whether the pilot conditions and design are problematic for the embedding of its
results. We conclude our paper with presenting design and initiation strategies developed in the cases to
deal with this paradox.
This paper is based upon a secondary comparative case study. We rely upon extensive case descriptions
(based upon interviews, focus groups and document study) produced by ourselves and reinterpret this
material from the perspective of this paper.
The reminder of the paper is structured as follows. In the next two sections we explore the conditions for
successful experimentation in pilots and the necessary conditions for policy change and learning in
existing policy regimes. In section four we confront these two sets of conditions to problematize the pilot
paradox. Then we present the methodology of our paper. In section six we present our cases and in section
7 we analyse how the paradox is apparent in these cases. In the final section we answer the question to
what extent we can actually speak of a paradox and with what strategies there can be effectively dealt
with this paradox.
2. Conditions for successful pilots (“innovative outcomes”)
There are many answers possible to the question: what makes a pilot successful? Success is a very
subjective term. What does it mean and to whom? A first assumption is often whether the innovation
3
‘works’, and counts as ‘evidence’ (Cabinet Office 2004, Nutley 2002). In this paper we make a distinction
between two aspects of success: internal success and external success. This is related to the often made
distinction between output and outcome or impact. In section 2 we focus upon internal success while in
section 3 external success is put central.
2.1 What counts for a successful pilot?
The internal success of a pilot has to do with the extent to which the pilot successfully realize its main
ambition. Often this ambition is related to testing innovative approaches or concepts and developing new
insights and knowledge. Subsequently, success can also be considered as usable knowledge being
developed at the basis of which new decisions can be made. Internal success of a pilot can also be more
in terms of collaboration, the establishment of new/ positive cooperation between stakeholders. External
success is the extent to which the knowledge or networks developed in the pilot project are being used
for new initiatives, both on project and policy level.
2.2 Conditions for success
So, depending on the objectives, conditions for a pilot’s success can be identified. For successful pilots a
couple of conditions can be found in the literature.
a. Being a “safe haven”
A first condition for a successful pilot is its status as a safe heaven, at a distance from the heat of the
kitchen. This distance is important because it is necessary for a pilot to have some room for
experimentation and manoeuvre which is not allowed for in the regime itself. Political support is often a
crucial element to safeguard this condition and to prevent for interventions from the regime within the
exploring process of the pilot. Within the literature this condition is often labelled as a protective space
(Smith & Raven, 2012).
b. Enough resources for exploration
A second crucial condition for a pilot has to do with the availability of enough resources to fulfil the
experiment. These resources have to do with organizational resources, but also with sufficient expertise
and means to generate knowledge, with time for undertaking experiments, monitoring and analysis, and
also with attention and manpower from the participating organizations or units.
c. Competent participants
Pilots can be effective places for mutual learning and frame reflection. That presupposes that the actors
involved in the pilot are open for new knowledge and willing to learn. Innovative processes often attract
the more entrepreneurial persons, persons who are more able to span boundaries between different
perspectives and ways of knowing. Such a coalition of the willing is an important success condition for
pilot projects.
d. Principled engagement and social learning
Pilots are often collaborative efforts, especially when it comes about complex governance issues. For
successful collaborative innovation it is essential that actors are able to come to frame reflection and
4
mutual learning, via a process of principled engagement (Sorensen & Torfing, 2012). Conditional for such
a learning process is that actors are able to overcome their predefined boundary judgments and look
beyond their organizational beliefs. Pilots need boundary spanners: people who are able to bridge
different problem perceptions and who are willing to change their own normative and cognitive frames.
e. Limited sale
Often, a small scale is the decisive factor for a pilot to be implemented. Small scale often implies limited
risks, both for the physical system and financially, makes the system boundaries of the pilot clear and
allows for precise outcomes. Small scale refers to both space and time. Note that ‘small’ is a relative
notion: depending on the intentions ‘small’ can refer for example to a municipal level if the intention is a
national roll-out, or to a single floodplain if the idea is to be implemented at a river branch. Something
similar holds for the limitation in time: limiting time gives an indication of when results can be expected
that should be sufficient for understanding the functioning of the innovation. A risk however, is that the
scale at which is piloted and monitored is insufficient to develop insights: some effects may only appear
many years later, on for example on a river stretch level and not at a floodplain level.
It is important to note that several conditions mentioned above can be strengthened during the course of
a pilot process: a pilot that gains some status is often able to attract additional resources for further
exploration. And within a pilot in which there is initial willingness for mutual learning, this can be
reinforced by the social capital developed during the course of the explorations.
3. Conditions for learning from pilots and upscaling (“policy change”)
3.1 What counts for successful uptake?
As explained above, the wider impact of a pilot can be seen as its “external success”. We can define this
success as the matter to which the output of a pilot also results into more enduring change within the
policy regime (by changing policies, visions, rules and standards et cetera) and whether its results are also
applied in other contexts and upscaled to a larger area, time span or set of problems. This also holds for
‘negative’ results, meaning that if the pilots demonstrates that a certain solution does not work at all or
under certain conditions, policies that already anticipated for the solutions are being altered, and that the
solution is not being implemented elsewhere. Another form of success is if on the basis of the pilot the
new questions that arise are being addressed in a follow-up pilot.
3.2 Conditions for successful uptake
The successful diffusion of pilot results depends upon a variety of factors. We summarize them into four
categories.
a. Normative congruence: sufficient proximity between pilot and home bases
For the successful uptake of pilot results, it is important that pilot participants keep in mind what the
dominant values are in their home bases. Innovative ideas only succeed when they sufficiently correspond
with existing insights and knowledge of actors (Van Buuren & Eshuis, 2014). There is something like an
optimal cognitive distance between new ideas and standard routines. Too much distance in terms of
5
differences between the shared beliefs developed within the pilot and the fundamental values and beliefs
of the regime hinders the uptake of innovations. One step further is that the ideas not only fit the
organisation, but also that (some) participants act as ambassadors: they stand for the innovation, are
convinced that it brings large improvement compared to current practice and actively promote it.
b. Organizational appropriateness of pilot outcomes
In line with the first condition, it is important that there is a sufficient degree of compatibility of the
outcomes of a pilot with the standard operation procedures within the current policy regime. The regime
has to be able to handle and apply the solution and to translate it into its existing routines and rules of
the game.
c. External representativeness of pilot composition
For the wider uptake of pilot results it is important that the future users are attached to the experiment.
Sufficient interactions between niche and regime are necessary to safeguard that the regime will value
the pilot as a legitimate source of knowledge (Ingram et al. 2015). Such a strategy can be done by
safeguarding enough interaction between pilot participants and the outer world, to engage
representatives of the various home bases in the same learning process as the pilot process is in (Smink
et al. 2015).
d. Pilot embedding to assure a renewed logic of appropriateness
Studies that focus upon the question: how to come beyond experimentation, emphasize the importance
to look at how to embed pilots or experiments into its wider environment. An important component of
such a strategy, has to do with assuring that the outcomes of a pilot can be easily embedded into the daily
routines of the receiving organizations, their logic of doing things. That requires that the question is
answered what is necessary to mainstream the outcomes of a pilot and to replicate it in normal processes
of planning and decision-making (Kivimaa et al., forthc.). Another important design component has to do
with organizing attention on the question how to make the pilot an instrument to achieve policy change?
Consequently, is the design of the pilot such that it is in continuous interaction with the policy,
stakeholders and areas where it should land?
e. Sufficient system understanding and trust in results
Important for uptake of the outcomes of the pilot is the perception that its results are reliable,
representative and useful. Hence, this places quality criteria to the pilot project. Did the design fit the
questions that had to be addressed, was the scale (both in time and space) appropriate to answer the
questions, and were the ‘right’ questions addressed? All of these elements can be used to both follow up
the pilot or to find a way out. A pilot can for example easily be set aside by classifying results as ‘non-
representative’. For further uptake of the pilot, the pilot results have to be trusted and considered to
contribute sufficiently to formal policy objectives.
4. The pilot paradox
Within the world of international aid and development it is a common expression: “pilots never fail, they
(also) never scale”. Although pilots often succeed in testing innovative solutions and producing new
6
knowledge and often contribute to what participants learn, they are far less successful in generating
enduring policy change. A successful pilot (which delivers useful knowledge and results in more
knowledgeable participants) is anything but a recipe for successful diffusion and upscaling.
To our mind part of the explanation for this has to do with the contradictory conditions for “internal” and
“external” success. In other words: the conditions that contribute to a successful pilot are also the main
barriers for a successful uptake. In table x we summarize the two sides of the coin.
Element
Conditions for successful pilots
Conditions for uptake
Position of the
pilot
At a distance from home bases
(freedom to explore novel ideas)
Keeping connected: conscious strategy to
create normative congruence.
Resource
distribution
Additional resources for the pilot to
enable creativity and exploration.
Solutions fit within the existing system of
resource-distribution and contribute to
organizational aims of efficiency and risk
reduction.
Participants
Coalition of (willing) boundary
spanners.
Representativeness of involved actors from
all relevant disciplines and stakes of the
future implementation arena
Process design
Learning environment, tailor-made
collaborative process design.
Results ready for mainstreaming and
broader embedding. Focus on where the
results have to land.
Project design
Limited scale to reduce risks and
(financial) impacts, high quality
(shared) monitoring and analysis
Sufficient system understanding; outcomes
considered representative and of high
quality
Table 1. The pilot paradox
For our analysis we have conceptualized the various elements of the paradox, see table 2.
Element
Conditions for successful pilots
Conditions for uptake
Position of
the pilot
- Degrees of independence (with regard
to content, way of working, rules) of the
principal organizations
- More flexible applications of law and
regulations (or less strict reinforcement)
- Reporting to the own organisation on
different levels (political, strategic, tactical
and operational)
- Conscious strategy to convince home bases
about fit with existing values of pilot results.
Resource
distribution
- Additional budget; expertise; partners;
time
- Political attention / pressure to ‘score’
with the pilot
- Solutions help the various organizations to
realize their core values
- Solutions are applicable within the existing
resource distribution.
Participants
- Open: 1) easy communication and 2)
openness about interests etc.
- Act as boundary spanners
- Willing (able) to experiment / take risks
- Potential criticasters from participating
organizations are included in process
- Future users are informed and involved
- Participants act as ambassadors.
Process
design
- New roles for (new) partners
- New forms of collaboration
- Possibility of application
- Identify potential future application areas,
identify questions, ‘translate’ results
7
- Degrees of freedom for organizing the
own process
- Connecting with potential future users,
identify their questions and needs
Project
Design
- Limited scale (time and space)
- Well defined questions and limitations
- High quality monitoring
- Sufficient trust in outcomes
- Sufficient insight in how the system which is
the object of the pilot works.
- Representativeness of results
Table 2. Conceptualization of the conditions for success
5. Methodology
For this paper we deliberately selected eleven pilots in the domain of climate adaptation we already
analysed. We thus conducted a secondary analysis of existing cases. In a couple of cases we were involved
as a researcher or expert and for other cases we were asked to conduct formal evaluation studies We
deliberately selected our cases in relation to our main aim, to explore the paradoxical relation between
success conditions for pilots and for the broader uptake of their results. We used two main criteria for our
case selection:
- The selected pilots have to represent the main issues regarding climate change adaptation (flood risk
management, fresh water availability, soil subsidence, ground water management, water nuisance);
- The available data have to allow for an analysis of internal and external success and have to provide
information about whether the various conditions for success were met.
The case studies are based upon empirical material collected by a variety of means: interviews, document
analysis, focus groups and other methods (see appendix 1 for an overview per case). The case analysis
and the comparative analysis was done within a group meeting of all authors. By motivating our scores to
each other and collaboratively deciding upon the scores we realized coherence within the analysis.
6. Case studies
In this section we will describe the 11 case studies. The main characteristics of the analysed pilot projects
are summarized in table 1.
Pilot
Start and end
Type of climate
adaptation
Innovation
Level
Involve-
ment of
actors
Altenheim
1988-1996
Beuningen
2004-2008
Contract-
vormen
2015-2020
8
Koopmans-
polder
2011-ongoing
Landbouw
op peil
2011-2014
Loosdrechtse
plassen
2011-2013
Marken
2012-2015
SMART
water-
management
2012-ongoing
Waalblok
2007-2010
Water
farming
Walcheren
2010-ongoing
Zwolle
Start 2010
Now:
realization
Table 2. Summary of the studied pilot projects (meaning icons: see appendix 2)
In the next paragraphs, we describe these pilot project and pay attention to the aim of the pilot project,
the way the pilot project is organized and its results.
Altenheim: Integrated Rhine Program
The German State of Baden- Wuertemberg decided in 1988 to develop the so-called Integrated Rhine
Program. The objective of this program was to reduce water levels of the Rhine and increase nature values
of the floodplains. For this purpose thirteen floodplain areas have been indicated as inundation areas.
Essentially, floodplains that have been disconnected from the river should get their original function back
and so be able to serve as floodplains. Polder Altenheim was the first in 1989 and functioned as a pilot
project. The pilot was organized nearby its home base. Indeed, the pilot was already included in the
formal policy. Provided the political importance, there have been sufficient resources within the pilot.
Results were considered positive by the policy makers and in 1996 the whole plan has been approved.
However, after the pilot phase the implementation of the program did not progress well. In some of the
indicated areas opposition arose from citizens living near the other planned areas, supported by their
municipalities. They did not approve the results. In their view the plans were harmful for nature (e.g.
drowning wildlife), reduced accessibility and more hindrance due to mosquitoes. The opposition delayed
the implementation enormously (in 2013 only 4 out of 13 areas have been implemented) and in some
9
areas the concept itself was challenged. Policy makers attempt to address the concerns and reduced their
ambitions (i.e. flooding the areas less frequently and less intensively).
Beuningen: River maintenance based on Cyclic Floodplain Rejuvenation
In the floodplains of Beuningen, Rijkswaterstaat (the implementation agency of the Dutch Ministry of
Infrastructure and Environment) has identified increased water levels due to vegetation development.
Hence, the discharge capacity of the river did not meet the norms anymore. Instead of simply removing
all the vegetation, a nature organisation (Ark), together with Radboud University has developed a new
concept called ‘Cyclic Floodplain Rejuvenation’. Free flowing rivers regularly reset vegetation and lower
sandbanks as a result of erosion and so make space for the water to flow. This particularly occurs in places
where the river would naturally flow. A confined river cannot do this. The idea of the concept is to imitate
the natural behaviour and so to create space on one hand and increased natural value resulting from
larger variety on the other hand. Moreover, the resources that become available can be used for
economic activities (biomass, sediment). In 2004 the idea was launched to design the floodplain according
to these principles. The idea was far off from the home base of RWS and resources were limited,
particularly for implementation. Hence, it took up to 2008 before the pilot was implemented. The pilot
was considered positive by the ambassadors, but at the same time could be easily set aside due to a lack
of monitoring. Plans existed to extend the pilot. The ministry of environment would finance this in order
to create a climate buffer. However, RWS meanwhile had reached the flood defence objectives and
decided to follow the strategy of controlling and maintaining the situation as it is, meaning that there is
no juridical space for the CFR approach in which vegetation shows more variety.
Innovative contracts in maintaining water ways
Rijkswaterstaat, as the manager of the waterways in the Netherlands, nowadays outsources the
maintenance of the rivers and canals. In one area (the IJssel and the Twente canals), the contract has been
extended with the clause that during the contract period (5 years), the contractor together with RWS and
a knowledge institute (Deltares) should identify, test and develop innovations that ought to make river
maintenance more efficient and more sustainable. Moreover, these three parties work together in a so-
called learning team, where traditional client-contractor relations do not exist. Rather, the parties should
work as project partners creating added value for each of the parties. Indeed, Business Cases (BCs) should
be developed. So far, several interesting potential innovations have been identified (e.g. flexible groynes,
mats to prevent scouring around the groyne, using herded sheep instead of machines for mowing) and
pilot projects are being designed. However, clear BCs have not been developed yet. The pilot is far off
from the home base of all three organisations. It requires different approaches from all of them how to
deal with the other parties. Traditional roles become vaguer. The pilot receives much positive attention
(it won a European prize for Innovative Public Procurement).
At the moment the pilot has produced a draft text about new, national performance based contracts. This
text will be included in the toolbox for Rijkswaterstaat. This success in terms of uptake is mainly the result
of a personal link between the pilot and the national contract unit: the driving force behind the pilot is
also a team member of this unit.
10
Waalblok
The pilot Waalblok comprises the realization of a retention cellar beneath a greenhouse to deal with water
nuisance because of extreme rainfall. The pilot project was part of a Knowledge and Innovation Program
Water Framework Haaglanden (KIP) and was carried out as a collaborative activity of the regional Water
Board, the municipality and the greenhouse owners in the area. The core of the idea was to realize a
multifunctional concept for water storage and water cleaning in a cellar. Such a cellar combines public
tasks (water retention) and private tasks (water storage and water cleaning).
A small coalition of enthousiastic people worked together to refine the idea of the cellar. With help of
additional resources of the KIP, they were able to do additional analyses and to reflect periodically upon
their own process. The pilot was at a relatively large distance of the various home bases. The pilot
participants maintained a direct link to their political principals and not to the rest of their organization.
The pilot participants had a difficult job in convincing the people within their home bases to enable the
realization of the cellar, especially in case of the Water Board. After a difficult planning process the cellar
was ultimately realized. However, the solution is seen as rather costly and also too complicated due to
the parallel private and public use of the cellar. The idea of large-scale water retention is exchanged for
the idea to look very precisely which tailor-made solutions are necessary and possible on the very local
scale, based upon a detailed calculation of the actual consequences of extreme rainfall.
The internal success of the pilot was thus relatively high. It was implemented, but mainly thanks to the
additional resources that were available within the regional Knowledge and Innovation Program The
external success was almost absent: the solution was not replicated and also did not result in policy
change.
Loosdrecht: flexible water levels
In the recreational area of Loosdrecht, in the Western part of the Netherlands the Water Board decided
to apply flexible surface water level management. This should allow the water level to rise and fall with
the precipitation and evaporation within certain preset boundaries. This is opposed to the ‘regular’ water
management in the Netherlands, were water levels are set on a single level and do not fluctuate. The
reason to do so was to reduce the need to let in water from the surrounding waterways with a high
nutritional values in dry summer, which was harmful for the ecological goals set for the Loosdrecht area
because of the Water Framework Directive (WFD). However the new water regime caused several worries
among citizens about the possible effects (flooding, hindrance for water recreation)
Because the pilot was a subsidized project – within the Dutch innovation programme of the WFD, the pilot
was organized as an interorganizational project with many external research institutes involved in the
team. There was ample funding for both the actual implementation of the pilot as well as the monitoring
and insights into the effects of the pilot. The position of the pilot was well embedded within the
organization of the Waterboard – but only in the ecology oriented part of the organization. The process
was designed around implementation of the pilot, the monitoring and the dissemination of results. The
stakeholders (municipality, marina’s, house owners) were mainly involved by means of participatory
monitoring. The results of the pilot were framed as a success by the Waterboard. The Water board
mainstreamed the concept of flexible surface water level management, which was also thanks to the
active dissemination of the results of the pilot.
11
Koopmanspolder: a paradise for water birds
The Netherlands needs new types of water management in order to be prepared for the impact of climate
change. To create different functions which support nature, fish and leisure facilities the ‘inland shore’
concept, a new approach to water storage was developed. A pilot was instated in the the Koopmanspolder
in the North West of the Netherlands. The first tests with the flexible water level in the Koopmanspolder
are bearing fruit, the polder is partially submerged and nature is flourishing. The pilot was organised close
to the home base of the primary coordinators of the project (Waterboard and Rijkswaterstaat).
Furthermore the project consortium consisted of the province, the municipality, Deltares,
Staatsbosbeheer and a number of local NGO’s. Funding of the pilot came from different sources but the
pilot was primarily funded by the national fund for rural areas (ILG). The pilot has a very long history,
which started in the ’90 by making the area part of the ecological structure of the Netherlands, but
through time the focus changed from nature development to more multifunctional purposes. Which
resulted in an award winning landscape design for the polder in 2011 to combine nature development
with flood risk management, recreation and freshwater supply. The pilot was considered a success within
the coordinating organisation, but also outside the organization. This also resulted in the start of a second
pilot of the ‘inland shore’ concept by public and private parties, with a focus on agriculture, aquaculture
and fishery in other parts of the province.
Marken: a new approach to flood safety
The water safety situation on the island of Marken in the Netherlands is not adequate to resist high water
levels in the future. The dykes around the island were rejected in former test rounds. Therefore, the
National water Agency developed a dike reinforcement plan in 2012. This plan could not count on the
support of the inhabitants of the island because of the impact on culture history and the adjacent houses.
Next to this, the plan was too expensive. At het same time, the Netherlands was developing a new water
safety strategy which pays more attention to the equipment of the land and evacuation strategies in case
of floods. For this reason the involved governments developed a pilot to investigate whether the new
water safety strategy would be a solution for the island of Marken (technical innovation). The pilot is a
cooperation between the National water Agency, the municipality, the water board and local
stakeholders. The involved stakeholders work together in a project group which functions as a safe haven
for the project workers. The pilot is governed by a steering committee and funded by the national
government. The time horizon for the pilot is 3 years. The case Marken was successful in terms of
developing an innovative water safety strategies for the island (smart development of dykes) based on
new water safety principles. The results of the pilot will be used for other delicate water safety situations
in other regions in the Netherlands.
SMART Watermanagement: a new water management approach in the Netherlands
Smart water management is water management that uses a system approach to respond in an effective
and sustainable way to flooding and drought. By doing so it operates across (institutional) borders,
especially between Rijkswaterstaat (responsible for the national waterways) and Water boards
(responsible for the regional water management) as an effective and sustainable response to situations
of flooding and drought. The concept was piloted in the south-east of the Netherlands using the term
dynamic water management. Were it brought further together the two water boards and Rijkswaterstaat.
12
These partners had already been working together on watermanagement since 1994. The pilot did not
have a firm connection to the homebase however, as it was mainly perceived as innovation project. The
pilot was organized around three subjects: technical aspects, governance and learning experiences.
Before the pilot was started it was considered what the pilot could bring to the partners, for this mainly
cost reduction on potential damages and investments were considered. The pilot also brought to light
challenges concerning the division of responsibilities as the pilot would cross institutional boundaries. The
project did not have many resources to use, and mainly the innovation funds of Rijkswaterstaat were used
to develop the pilot. However the results from the pilot were perceived as positive and were considered
worth while pursueing further on a national level. This is currently being developed within a national
research program on the subject of smart watermanagement in an attempt to mainstream the concept.
Landbouw op Peil
Climate change poses Dutch water managers for new challenges. This also accounts for the Water boards
in the region Rijn-Oost. The major part of the area that these water boards manage consists of agricultural
lands. Therefore one of the biggest challenges for these water boards is to develop a future-proof system
of water management which leaves sufficient room for vital agricultural production systems. This requires
innovative measures optimizing a sustainable water system in relation to the agricultural function.
With this challenge in mind the project Landbouw op peil (“Agriculture at Water Level”) was started in
2011. The aim of the project was multi-faceted:
- Gather knowledge and experience on how to maintain a vital agriculture within a changing system of
water management (due to climate change);
- Improve the relation between the agricultural sector and water managers;
- Contribute to a ‘climate proof’ agriculture in the Eastern part of the Netherlands;
- Use local knowledge from agricultural firms to improve future water management.
The pilot is perceived as very successful by almost all of the participating stakeholders. One of the main
factors for this internal success is the feeling amongst stakeholders that the pilot has really contributed
to increased levels of trust and understanding and has helped to narrow the gap between stakeholders.
Next to that, most of the measures that were designed in the pilot for dealing with the water challenges
were accepted by the farmers and were continued after the pilot had finished as well. The pilot received
quite a lot of (media-)exposure and was framed as (very) successful externally as well . At the end of the
pilot-stage, the pilot won a prize (for innovative project) which was accompanied by a sum of money. This
money was ‘reinvested’ in a follow-up of the pilot enabling the possibility to draw-up more
‘waterbedrijfsplannen’ for other farmers that were interested. These tailor-made plans at farm level were
considered to be one of the crucial success factors. This has contributed to a (moderate) external success
of the pilot as well.
Water farming Walcheren: a pilot to improve the fresh water availability of local farmers
Walcheren is a region in the Netherlands where agriculture is an important economic activity. More and
more the consequences of climate change are felt (shortage of fresh water supply). This negatively
influences the yield of farmers. For this reason farmers started to cooperate to improve the availability of
fresh water in dry periods by conserving the water in the soil and watercourses. They started a pilot to
test technical possibilities to infiltrate the water in the soil in winter and increasing the water levels in
13
water courses. The pilot is a private initiative (farmer) and is facilitated by knowledge institutes. The water
board is involved for providing permits but is most of the time at a distance. Therefore, the pilot is rather
far away from the home bases of the water authorities. The pilot has little resources available (only private
investments and subvention of public parties). The process in the pilot can be characterized as ‘learning
by doing’. No end date of the pilot is stated. Based on the effectiveness technical measures are added in
the pilot. The pilot is successful in effectiveness of chosen measures, but less successful in generating
enduring policy change. The pilot is local customized and based on the physical characteristics of the
territory.
Zwolle: climate proofing the city
Zwolle is a city in the IJssel-Vecht Delta, which is a small and vulnerable delta area in the Netherlands. In
this delta and especially in Zwolle, many spatial investments are planned. The regional and local
governments decided to anticipate on these developments by starting a pilot project with the aim to
climate proof the delta area. Based upon this regional aim, the city of Zwolle has the aim to climate proof
the city by bridging the boundaries between climate adaptation and spatial developments (like housing,
area (re)development and (re)construction of roads).
This pilot is organized in a layered way. At the regional level intergovernmental cooperation is organized
which is more at distance from the home bases. At the local level, a small group of civil servants within
the municipality is responsible for the project. To stimulate the pilot, the province made a substantial
budget available. Last four years, this budget was used to realize local climate adaptation projects. The
pilot-status, the financial resources, and the local demarcation, makes it easier to come to rapid and
successful decision-making. However, this makes it hard for other actors to understand and to participate
in the pilot. The process design changes during the pilot. In the first years, the creative process at the
regional level dominates and the municipality of Zwolle has to follow this process. In the later years, there
is much more room for local process design. In this local process design, the municipality of Zwolle invites
for each climate proofing project a variety of actors: governments, companies, NGO’s, interest groups and
citizens.
The pilot Zwolle resulted in various realized projects to climate proof the city. In this way, the pilot was
successful. A small staff within the municipality is closely involved and takes care of the realization of this
next series of projects within the pilot, but broader mainstreaming the results within local policies hardly
takes place.
7. Analysis of the paradox
In table 3 we summarize the way the pilot projects are organized, by using the elements of the pilot
paradox (see table 1). For each case we indicate whether this element is present: very low (- -), low (-),
moderate (0), high (+), and very high (+ +) .
14
Conditions for internal success
Conditions for external success
Distance of home base
Additional resources
Coalition of boundary
spanners
Tailor made process design
Low risk Scale
Normative congruence
Organizational
appropriateness
External representativeness
Pilot embedding
System understanding
Altenheim
--
++
+
0
+
++
+
-
+
-/+
Beuningen
+
+
0
+
+
--
--
--
--
0
Contract-
vormen
+
+
+
++
+
0
0
+
+
+
Koopmans-
polder
+
++
++
+
+
+
+
++
+
0
Landbouw op peil
+
++
+
++
+
/0
-
0
-
+
Loosdrecht
+
++
-
++
+
+
+
--
+
+
Marken
+
++
++
0
++
+
0
-
-
+
SMART water
management
-
++
++
+
+
++
--
0
-
-
Waalblok
+
++
+
++
-
-
-
-
Water farming
Walcheren
++
+
+
++
++
--
-
+
-
0
Zwolle
-
++
++
+
++
-
0
+
- -
0
Table 3. Case comparison
Results of the pilot projects
Internal success:
results of the pilot project
External success:
mainstreaming of results
Altenheim
High: the initial idea was successfully
developed and implemented.
High in terms of policy adoption, low in
terms of implementation
Beuningen
Moderate: little monitoring, pilot
already ‘died’ before implementation
Low: other requirements put in place
15
Contractvormen
Moderate: pilot results in promising
innovative ideas, but they are not
implemented yet.
Promising: ideas are considered to be
included in the standard routines
(toolbox) of the organization.
Koopmanspolder
High: attempts are being made to
replicate the pilot
High: also other locations are considered,
using the same consortium
Landbouw op peil
Very high. All stakeholders satisfied.
Increased trust and improved
relations.
Moderate: there was little ‘infrastructure’
to deal with this interest. No structural
follow-up.
Loosdrecht:
Moderate: the pilot was considered a
success but has a distinct usergroup
Moderate: the measure is being
considered by other public authorities
Marken
High: Successful development of
customized water safety strategy
Low: Newly developed water safety
strategy is only slightly embedded in daily
routines
SMART water
managment
High: results do have serious impact
Moderate: the concept has found its way
to the policy table, but implementation is
lacking.
Waalblok
High, the pilot was successfully
implemented
Very low, the solution doesn’t fit into the
normative convictions of the policy
environment
Water farming
Walcheren
Very high: stakeholders typify their
pilot as successful
Low: solutions do not automatically fit in
the policy system
Zwolle
High: successful realization of various
climate adaptation projects
Low: no attention to mainstreaming of
this process innovation
Table 4. Success of the pilots
Explanation of the scores
Altenheim
The pilot was initiated at the core of policy making. The pilot started up very quickly and with large political
support. Hence, resources were available and learning taking place. It was also very clear where else the
innovation should land, according to the developers that included policy makers. However, when defining
the policies, opposition became apparent. This came mainly from citizens and local politicians. This type
of dynamics had not been taken into account in the pilot design: actor involvement was insufficient: not
so much for the current pilot but for the intended future implementation areas.
Beuningen
The pilot was initiated by a nature organisation in cooperation with the university. Resources for
developing the concept and a site were available, not directly for implementation. One of the main issues
to be addressed were therefore who had to pay for it, which came down to the discussion what type of
problem it was: nature or flood defence, both having different resources. Essentially, the innovation
16
addressed resources that were administratively divided in a different fashion. A second issue, determinant
for the external success was that the decision-making process of the river authority (RWS) is based on
static systems, whereas this concept requires dynamics. The concept could not be adopted by the river
authority because system understanding was different.
Contractvormen
For each of the three participants, the required new way of collaboration is far off from the home base.
New business models still have to be developed. On the other hand, resources have been made available
to give it a serious try: financial sources, by making test locations available and managerial support. Each
of the parties is well represented acting as boundary spanners. As said, it is plausible that the resulting
ideas will be included in the toolbox of contract forms for Rijkswaterstaat, thanks to the linking pin role of
one of the pilot members.
Koopmans-polder
This pilot took a very long time to find its final shape – starting in the 1990’s and developing into the
pilotproces that was initiated in 2011 and continues to this day. The available resources that were brought
together because the coalition of the willing was formed brought it to success. And it was actually also
this coalition that created formed the basis for the successful uptake (other locations are being
considered). However one can ask it will not remain a string of pilots instead of mainstreaming the
concept.
Landbouw op Peil
One of the main factors for the (internal) succes of the Landbouw op Peil project was the room for tailor-
made solutions at individual farm level. In a process of intensive guidance specific watermanagement
plans where discussed and developed for each of the participating firms thus targeting the specific
(water)challenges at firm level with tailor-made measures and solutions. This approach resulted in a high
level of enthusiasm, commitment and confidence amongst the participants. In order to be able to come-
up with these tailor-made solutions, the water managers also took quite some room for manoeuvre and
sometimes distanced themselves from the existing practice and protocols within their organizations. In
the field /within the pilot this resulted in a very productive and fruitful dynamic, however as this dynamic
was also quite far-off from the common practice it hindered the up-scaling / the follow-up of the pilot.
Loosdrecht: flexible water levels
The Loosdrecht pilot fits with almost all the conditions for a succesfull pilot – the only thing lacking are
the boundary spanners, as the pilot was more designed top down and implemented by the waterboard –
surpassing the protests of the stakeholders in the area -. Furthermore the pilot clearly established a
normative congruence with its home organisation. And although the stakeholders had protested the pilot
also made it possible to involve these as participants.
Marken
The case Marken scores high on the conditions for successful pilots. It is organized at safe distance from
normal routines. Furthermore a coalition of boundary spanners is present. The pilot has sufficient
17
additional resources. The case scores relatively low on the conditions for successful mainstreaming. The
developed water safety strategy is highly local customized and only slightly embedded in the daily routines
of the involved organizations.
SMART water management
Just as with the koopmanspolder the coalition of the willing created the basis for this pilot, however
placing it further from the home bases. The conditions for uptake were then also met, because the pilot
did not fit with the existing system and the results are not ready for mainstreaming. But the strange aspect
was that the pilot was more or less forgotten, but uptake did occur by means of bringing the concept –
without the results of the pilots – to a national level.
Waalblok
The case Waalblok is a typical illustration of the pilot paradox. It is organized at a safe distance of the
regime. It consists of a coalition of willing participants and has many additional resources for exploration,
which makes it possible to realize the pilot. At the same time it encountered many difficulties in
embedding itself in the responsible home bases and thus there is no upscaling of the pilot.
Water Farming
The case water farming Walcheren is also a typical illustration of the pilot paradox. It is organized at far
distance from water authorities and has a tailor made process design. Its weakness however is its distance
to the dominant values in the home bases of water authorities. The pilot embedding and organizational
appropriateness scores low. At the same time there is a lack of normative congruence between the
stakeholders in the pilot and the policy regime.
Zwolle
In the pilot project Zwolle, the innovative process design was successful because of some organizational
conditions. Especially the additional resources and the involvement of willing actors were helpful to come
to rapid decision-making and realization. However, this rate within the pilot makes it hard for other actors
to keep the connection. Thus although the internal success, the external success is low.
Generic observations
It is interesting to see that in the majority of the cases we can indeed witness that a high score on the
conditions for internal success goes along with low scores on conditions for wider uptake (eg. Marken,
Water Farming, Zwolle, Beuningen, Waalblok, and also for Smart Watermanagement, Landbouw op peil).
There are at least two mechanisms that can explain this.
First of all, there is the mechanism of goal-displacement. Actors within a pilot do a really good job in
making the pilot successful, because of the attention given to the pilot. But they neglect to keep the wider
environment attached. They focus upon realizing the objectives of the pilot in producing an innovative
outcome and thus also focus entirely upon realizing and implementing that outcome. With help of
additional resources, they succeed in doing so. But they forget (or do not have the capacity) to take into
18
account what is needed to enable lesson-drawing and broader dissemination of the ideas and insights of
the pilot in policies and practices.
The second, related and complementary mechanism can be called regime alienation. Due to the
development within the pilot (as a result of intensive learning processes at a distance of the home bases),
the distance with the home bases becomes only larger instead of smaller. In that situation the outcomes
of the pilot are so different from what is acceptable for the regime, that wider uptake is nearly impossible.
At the same time, a couple of pilots do show far less elements of the paradox or even show good results
both internally and externally (eg. Koopmanspolder, Altenheim, Flexible Waterlevels), although uptake of
the results in these cases is also mainly conceptually and with regard to policies rather than substantially
and in terms of implementation.
We can see at least two explanations for simultaneous internal and external success. First of all, in some
cases the pilot results come at the right moment. There is – in other words – a window of opportunity for
the outcomes of the pilot to become mainstreamed in policies and practices of the various home bases
(cf. the case Flexible Waterlevels ). Oftentimes a long incubation phase before the pilot is started, resulting
in fertile ground for its results. In that case the pilot is the result of either growing regime frustration,
because of the ineffectiveness of current solutions. The regime is actually ready for uptaking the results
of the pilot, because they are not able to solve the problems in the traditional way, and they have come
to alternative solutions.
Secondly, in a couple of cases the pilot paradox is effectively prevented for by either self-correcting
mechanisms within the pilot process or by making the broader uptake of the lessons of the pilot explicitly
part of the aims of the pilot. Self-correcting mechanisms are visible in those pilots which are not very well
embedded in their wider environment or which representativeness is low. These pilots do have fairly
appropriate results that are also congruent with the dominant beliefs in the wider policy environment
(Althenheim, Loosdrecht, Smart Water Management). In the case Koopmanspolder the pilot is organized
in such a way that participants continuously inform their environment about what is going on and the
preliminary results (cf. Koopmanspolder). This case is also the only case in which all conditions for internal
and external success were met. In this case we can witness a carefully crafted strategy to organize an
intelligent process for exploration in a safe environment, and an intense process of dissemination to
safeguard the wider uptake of the results.
8. Conclusions: the paradox revisited
Based upon this analysis we can draw a couple of conclusions.
First of all, we can conclude that there is indeed a paradoxical relationship between the conditions for
internal and external success. By focussing upon meeting the conditions for the first, easily at the same
time a barrier is organized for external success. Especially the distance between pilots and the regime and
the dedicated process to find an innovative solution in a non-standard way, forms a barrier to come to
19
appropriate outcomes and to organize sufficient embedding necessary for wider uptake in the existing
regime.
At the same time, the pilot paradox is far from a general causal mechanism. It is possible to prevent for
the pilot, especially by carefully designing the course of events in the pilot and by formulating its main
aims in terms of both internal and external success. Making broader uptake part of the main targets and
organizing attention to this target from scratch is not only a very logical but also rather effective way to
overcome the paradox.
This brings us also to the (managerial) implications of the paradox. Designing experiments in any field can
be improved when actors recognize the pilot paradox they can be confronted with. Within the design of
management of pilots, there have to be attention for issues of external representativeness and boundary
spanning between niche and regime (by ways of dialogue with the wider environment). This can for
example be organized by integrating future users in an early phase of the pilot. Or by starting to
communicate some of the (intermediate) successes and obstacles to a wider public from early on in the
pilot.
Finally, it would be very interesting to complete this first exploration of the pilot paradox with a more
quantitative study of a large sample of pilots and to analyse these in a more quantitative way in order to
get more rigorous findings about the peculiarities of the pilot paradox and the circumstances under which
the paradox is present or can be prevented for.
20
References
Breman, B. C., de Groot, M., Ottow, B., & Rip, W. (2014). Monitoren doe je samen. De meerwaarde van participatieve
monitoring. H2O online, 2014(20-06), 1-9.
Buuren, M.W. van and D. Loorbach (2009) Policy innovation in isolation? Conditions for policy renewal by transition
arenas and pilot projects, Public Management Review, 11(3): 375-392.
Buuren, M.W. van & Eshuis, J. (2009). Van concept naar contract: Waterberging, herstructurering en innovatie in
proeftuin Waalblok. (Extern rapport). Rotterdam: Erasmus Universiteit Rotterdam.
Buuren, M.W. van & Bressers, N.E.W. (2011). Waterproeven in polderland. Een terugblik op Waterkader Haaglanden.
(Extern rapport). Rotterdam: Erasmus Universiteit Rotterdam.
Buuren, M.W. van, G.J. Ellen, C. van Leeuwen & J. van Popering-Verkerk (2015) Die het water deert die het water
keer: overstromingsrisicobeheer als maatschappelijke gebiedsopgave, opbrengsten en lessen uit de pilots
meerlaagsveiligheid [in Dutch], Rotterdam: Erasmus University Rotterdam.
Cash, D., Clark, W. C., Alcock, F., Dickson, N. M., Eckley, N., & Jäger, J. (2002). Salience, credibility, legitimacy and
boundaries: Linking research, assessment and decision making.
Chan, P.M., Havekes, H. en Hofstra, M. (2013). Governance aspecten van dynamisch waterbeheer. Analyse van de
pilot Noord-Brabant (in Dutch). Water Governance Centre.
Duijn, M. (2009) Embedded reflection on public policy innovation: a relativist/pragmatist inquiry into the practice of
innovation and knowledge transfer in the WaterINNovation Program, Eburon
Ellen, G.J. & Ottow, B. T. 2012. Maatschappelijke aspecten van flexibel peilbeheer. Deltares. 1202707-003-BGS-0015.
Karstens, S. (2009). Bridging boundaries: Making scale choices in multi-actor policy analysis on water
management (Vol. 4). IOS Press.
Kuindersma, W. en B.C.Breman (2014). Leren van landbouw op peil. Evaluatie van een experiment met zelfsturing in
het waterbeheer. Alterra-rapport 2512. ISSN 1566-7197. http://edepot.wur.nl/298586
Nutley, S., Davies, H., & Walter, I. (2002). Evidence based policy and practice: cross sector lessons from the UK. ESRC
UK Centre for evidence based policy and practice: working paper, 9.
Sanderson, I. (2002). Evaluation, policy learning and evidence‐based policy making. Public Administration, 80(1), 1-
22.
Sørensen, E., & Torfing, J. (2012). Introduction: Collaborative innovation in the public sector. Innovation Journal,
17(1), 1-14.
Vreugdenhil, H.S.I. (2010) Pilot projects in water management: practicing change and changing practice, PhD Thesis,
Delft University of Technology.
Vreugdenhil, H., J. Slinger, W. Thissen, and P. Ker Rault. 2010. Pilot projects in water management . Ecology and
Society 15(3): 13. [online] URL: http://www.ecologyandsociety.org/vol15/iss3/art13/
Ingram, J., Maye, D., Kirwan, J., Curry, N., & Kubinakova, K. (2015). Interactions between Niche and Regime: An
Analysis of Learning and Innovation Networks for Sustainable Agriculture across Europe. The Journal of
Agricultural Education and Extension, 21(1), 55-71.
Smink, M., Negro, S. O., Niesten, E., & Hekkert, M. P. (2015). How mismatching institutional logics hinder niche –
regime interaction and how boundary spanners intervene. Technological Forecasting and Social Change, 100,
225-237.
Smith, A., & Raven, R. (2012). What is protective space? Reconsidering niches in transitions to sustainability.
Research Policy, 41(6), 1025-1036.
Zeeman, S. (Sascha). (2015, September 25). Van grens naar verbinding. Public Administration. Retrieved from
http://hdl.handle.net/2105/32109
21
Appendix A. Data collection secondary case analysis
Case
Data collection
References
Altenheim
Interviews (9), document analysis
Gewasserdirektion Sudlicher Oberrhein/ Hochrhein
(1999, 2000, 2001), Siepe 2004
Beuningen
Interviews (11), document
analysis, observation workshops
Freude am Fluss 2007, Peters et al. 2006, Vreugdenhil
et al. 2010
Contractvormen
Participatory observation
www.ssrs.info
Koopmanspolder
Document analysis and
participatory observation
https://publicwiki.deltares.nl/display/CAW/Koopmans
polder
Landbouw op peil
Document Analysis +
(semi)structured (Group)
interviews (35 respondents)
Kuindersma & Breman (2014).
Loosdrecht
Document analysis, observations
and interviews
Ellen et al, 2012 ; Breman et al 2014
Marken
Document analysis, observation,
interviews
Evaluation study by Van Buuren et al. 2015
SMART water
management
Document analysis and
interviews
Zeeman et al., 2015
Chan et al., 2013
Waalblok
Extensive document analysis and
12 interviews
Evaluation study by Van Buuren et al. 2009
Water farming
Walcheren
Document analysis, 5 interviews
Van Buuren et al. (forthcoming)
Zwolle
Document analysis, 5 interviews,
and 1 focus group
Evaluation study by Van Buuren et al. 2015
22
Appendix 2. Meaning of the icons
Type of climate
adaptation
Flood safety
Fresh water supply
Water level
management
Nature
development
Urban
development
Innovation
Technical
Process design
Scale
Local
Regional
National
Actors involved
Few
Medium
Many
