ChapterPDF Available

The Data Shake: An Opportunity for Experiment-Driven Policy Making

Authors:

Abstract and Figures

The wider availability of data and the growing technological advancements in data collection, management, and analysis introduce unprecedented opportunities, as well as complexity in policy making. This condition questions the very basis of the policy making process towards new interpretative models. Growing data availability, in fact, increasingly affects the way we analyse urban problems and make decisions for cities: data are a promising resource for more effective decisions, as well as for better interacting with the context where decisions are implemented. By dealing with the operative implications in the use of a growing amount of available data in policy making processes, this contribution starts discussing the chance offered by data in the design, implementation, and evaluation of a planning policy, with a critical review of the evidence-based policy making approaches; then it introduces the relevance of data in the policy design experiments and the conditions for its uses.
Content may be subject to copyright.
Chapter 1
The Data Shake: An Opportunity
for Experiment-Driven Policy Making
Grazia Concilio and Paola Pucci
Abstract The wider availability of data and the growing technological advance-
ments in data collection, management, and analysis introduce unprecedented oppor-
tunities, as well as complexity in policy making. This condition questions the very
basis of the policy making process towards new interpretative models. Growing data
availability, in fact, increasingly affects the way we analyse urban problems and make
decisions for cities: data are a promising resource for more effective decisions, as
well as for better interacting with the context where decisions are implemented. By
dealing with the operative implications in the use of a growing amount of available
data in policy making processes, this contribution starts discussing the chance offered
by data in the design, implementation, and evaluation of a planning policy, with a
critical review of the evidence-based policy making approaches; then it introduces
the relevance of data in the policy design experiments and the conditions for its uses.
Keywords Policy experiments ·Learning cycles ·Evidence-based policy making ·
Policy cycle
1.1 Introduction
The wider availability of data and the growing technological advancements in data
collection, management, and analysis introduce unprecedented opportunities, as well
as complexity in policy making. This condition questions the very basis of the policy
making process towards new interpretative models.
Growing data availability, in fact, increasingly affects the way we analyse urban
problems and make decisions for cities: data are a promising resource for more
G. Concilio (B)·P. Pucci
Department of Architecture and Urban Studies, Politecnico di Milano, Milan, Italy
e-mail: grazia.concilio@polimi.it
P. Pucci
e-mail: paola.pucci@polimi.it
© The Author(s) 2021
G. Concilio et al. (eds.), The Data Shake,
PoliMI SpringerBriefs,
https://doi.org/10.1007/978-3- 030-63693- 7_1
3
4 G. Concilio and P. Pucci
effective decisions, as well as for better interacting with the context where decisions
are implemented.
Such multiplicity of data and its different sources poses several challenges to
policy making. First, the availability of a large amount of data improves the accuracy
and completeness of the measurements to capture phenomena that were previously
difficult to investigate but, at the same time, increases the level of complexity in the
approaches finalized to process, integrate, and analyse this data (Einav and Levin
2013).
Second, processing data is not neutral and irrelevant for its usability in decision
making processes. The selection and interpretation of a large amount of unstructured
information, deriving from data, requires a human based approach finalized to find
what emerging correlations between data are significant or not. In doing so, tools
to examine data are crucial, considering that non-human agents develop potentially
partial ways of understanding the world around them (Mattern 2017) and that some
tools, such as algorithms, can act as technical counters to liberty (Greenfield 2017,
p. 257).
Third, the huge amount of real-time, automated, volunteered data pushes towards
an epistemological change in the methodological approaches of empirical sciences,
transforming how we observe and interpret urban phenomena, moving from a
hypothetical-deductive method, driven by an incremental process of falsification
of previous hypotheses”to“an inductive analysis at a scale never before possible
(Rabari and Storper 2015, p. 33). In addition to using data to test previous hypotheses,
new phenomena and correlations between them may emerge as the result of the
massive processing of data (Kitchin 2014), with repercussions in decision making
activities in a short-medium term planning perspective.
Finally, while data is a non-neutral tool for addressing planning issues, the actors
that produce, manage and own data, both public and private—with the latter typi-
cally being corporations active in fields outside traditional regulations—configure an
unprecedented geography of power, a more complex arena in which urban problems
are defined, discussed and finally addressed by new constellations of actors.
These different implications and conditions related to the larger availability of
data, from data production, management, and analytics, to its potential in decision
making processes for both private and public actors, find synthesis in the expression
of “data shake”.
By dealing with the operative implications in the use of a growing amount of
available data in policy making processes, this article discusses the chance offered
by data in the design, implementation, and evaluation of a planning policy, starting
from a critical review of the evidence-based policy making approaches (Sect. 1.2), for
introducing the relevance of data in the policy design experiments (Sect. 1.3) and the
conditions for its uses. Acknowledging the impossibility of simply relying on data
for framing urban issues and possible solutions to them, and considering the potential
disruptions brought by data into the urban planning practices, this paper focuses on
policy processes where data is used, rather than simply focusing on technological
solutions fostered by data.
1 The Data Shake: An Opportunity … 5
1.2 Evidence-Based Policy Making: New Chances Coming
from the Data Shake
1.2.1 About Evidence-Based Policy Making
Evidence-based policy making (EBPM) represents an effort started some decades ago
to innovate and reform the policy processes for the sake of more reliable decisions; the
concept considers evidence as being a key reference for prioritizing adopted decision
criteria (Lomas and Brown 2009; Nutley et al. 2007; Pawson 2006; Sanderson 2006).
The key idea is to avoid, or at least reduce, policy failures rooted in the ideological
dimension of the policy process, by adopting a rationality having a solid scientific
basis. The fact that evidence should come from scientific experts and guide the
policy makers’ activities appeared and still appears a panacea to several scientists in
the policy making and analysis domain: this makes evidence based policy making a
sort of expectation against which policy makers, and political actors in general, can
be judged (Parkhurst 2017,p.4).
The evidence-based policy movement, as Howlett (2009) defines it, is just one
effort among several others to be undertaken by governments to enhance the effi-
ciency and effectiveness of public policy making. In these efforts, it is expected that,
through a process of theoretically informed empirical analysis, governments can
better learn from experience, avoid errors, and reduce policy-related contestations.
Finding a clear definition of the concept is not easy. In the policy literature,
the meaning is considered sort of “self-explaining” (Marston and Watts 2003) and
is associated with empirical research findings. Many scholars refer the evidence-
based policy concept as evolving from the inspiring experience in medicine: here,
research findings are key references for clinical decisions, and evidence is devel-
oped according to the so-called “golden standard” of evidence gathering that is the
“randomized controlled trial” a comparative approach to assess treatments against
placebos (Trinder 2000). Following the large importance assumed in medicine and
healthcare, there was then an increase in research and policy activists pushing for
evidence-based approach in other domains of policy making more related to social
sciences and evidence produced by the social science research, covering a wider
range of governmental decision making processes (Parsons 2001).
Moreover, the spreading of the evidence-based concept in policy making corre-
sponds to the infiltration of instrumentalism in public administration practices
following the managerial reforms of the last decades: the key value assigned to effec-
tiveness and efficiency by managerialism represented a driving force for evidence-
based policies (Trinder 2000, p. 19), so emphasizing procedures, sometimes at the
expense of substance.
The key discussion is on what makes evidence such: the evidence-based approach
in policy making is strictly correlated to the procedure, empirical procedure, that
makes evidence reliable. The spreading of the concept made social sciences look
at their procedural and methodological approaches to collect evidence although
6 G. Concilio and P. Pucci
Table 1.1 Key concerns raised about the emphasis on evidence in policy making
Key critics to EBPM References
(quoted by Howlett)
Evidence is only one factor involved in policy
making and is not necessarily able to overcome
other
Davies (2004); Radin and Boase (2000);
Young et al. (2002)
Data collection and analytical techniques
employed in its gathering and analysis by
specially trained policy technicians may not be
necessarily superior to the experiential judgments
of politicians and other key policy decision
makers
Jackson (2007); Majone (1989)
The kinds of “high-quality” and universally
acknowledged evidence essential to
“evidence-based policy making” often has no
analogue in many policy sectors, where
generating evidence using the “gold standard” of
random clinical trial methodologies may not be
possible
Innvaer et al. (2002); Pawson et al. (2005)
Government efforts in this area may have adverse
consequences both for themselves in terms of
requiring greater expenditures on analytical
activities at the expense of operational ones
Hammersley (2005); Laforest and Orsini
(2005)
Source Howlett (2009, p. 155)
the research categories of social science are missing deeply structured empirical
approaches.
Evidence matters for public policy making” as Parkhurst (2017, p. 3) underlines
by presenting and discussing three examples,1despite the concept collecting several
critics and concerns all together incriminating the supporters of the evidence-based
concept of being scarcely aware of the socio-political complexity of policy making
processes. Howlett (2009) has summarized such critics and concerns in four main
categories (Table 1.1).
Public policy issues have a prevailing contested, socio political nature that ampli-
fies the complexity of evidence creation processes: decision processes in public policy
making is not a standard, not a rational decision exercise; it is more a “struggle over
ideas and values” (Russell et al. 2008, p. 40, quoted by Parkhurst 2017, p. 5), it is
related to visions of the future and principles, so hardly manageable through rational
approaches and science.2In this respect, Parsons (2001) considers that, when values
are involved more than facts and evidence, policy processes are required which are
1Among them: the risk of SIDS (sudden infant death syndrome) for front slipper children (2005),
the research done by the oil company Exxon on the effects of fossil fuels on the environment (1970–
1980); the security risk posed by Iraqi regime in 2003, according to the US President George W.
Bush (p. 3).
2A widely discussed example in literature is related to policy making on abortion in different
countries: debates on abortion were more related to women rights against the rights of unborn as
1 The Data Shake: An Opportunity … 7
more democratic and which can facilitate … deliberation and public learning
(p. 104).
1.2.2 Evidence-Based Policy Making and the Data Shake:
The Chance for Learning
The increasing production of huge amounts of data, its growing availability to
different political subjects, and the wide exploration of the data potentials in decision
making for both private and public actors, are proceeding in parallel with the fast
advancements in technologies for data production, management and analytics. This
is what we call the data shake, and it is not only related to the larger and larger avail-
ability of data but also to the faster and faster availability of data-related technologies.
As a consequence, data-driven approaches are being applied to several diverse policy
sectors: from health to transport policy, from immigration to environmental policy,
from industrial to agricultural policy. This is shaking many domains and, as never
before, also the social science domain: the larger availability of data, in fact, and easy
to use data-related technologies, make data usable also by non-experts so widening
the complexity of social phenomena.
Nevertheless, although the data shake appears to have promising and positive
consequences in policy and policy making, existing literature underlines the role of
some consolidated critical factors affecting the chance for data to achieve such a
promising perspective. As highlighted by Androutsopoulou and Charalabidis (2018,
p. 576), one of the key factors is “the demand for broader and more constructive
knowledge sharing between public organisations and other societal stakeholders
(private sector organisations, social enterprises, civil society organisations, citi-
zens).” Policy issues “require negotiation and discourse among multiple stakeholders
with heterogeneous views, tools that allow easy data sharing and rapid knowledge
flows among organisations and individuals have the potential to manage knowledge
facilitating collaboration and convergence”. The response to such a demand implies
relevant expertise in organizations to adopt the “right” data, among the wide range
of available data sets, to analyse the data and to produce the effective evidence to
guarantee knowledge production and sharing.
Another key factor is related to the use of data when dealing with social prob-
lems: as again highlighted by Androutsopoulou and Charalabidis (2018), there is an
issue of proper use of data to develop a reliable description of the problem and the
formulation of effective policy measures. Also in this case, the selection of the proper
data set or sets, the application of a data integration strategy, the design of analytical
tools or models able to be effective in representation without losing the richness of
information embedded in data, and the consequent formulation of effective policy
measure consistent with the problem description are not simple rational decisions
well as to what a good society should look like; none of this debate can be definitely be closed with
science or scientific evidence.
8 G. Concilio and P. Pucci
and imply also the consideration of approaches to public debates to negotiate both
the vision and interpretation of the social problem and the solution to adopt.
The simple existence of more and different data and the related availability of
technical tools do not grant the solution of the issues identified by the opponents
to the evidence-based policy making concept. This last point explains why Cairney
(2017, pp. 7,8,9) concludes that attention is needed to the politics of evidence-based
policy making: scientific technology and methods to gather information “have only
increased our ability to reduce but not eradicate uncertainty about the details of a
problem. They do not remove ambiguity, which describes the ways in which people
understand problems in the first place, then seek information to help them understand
them further and seek to solve them. Nor do they reduce the need to meet important
principles in politics, such as to sell or justify policies to the public (to respond
to democratic elections) and address the fact that there are many venues of policy
making at multiple levels (partly to uphold a principled commitment, in many political
systems, to devolve or share power)”.
Better evidence, possibly available thanks to the data shake, may eventually prove
that a decision is needed on a specific issue, or prove the existence of the issue itself;
still it cannot yet clarify whether the issue is the first in priority to be considered
or show what the needed decision is: the uncertainty and unpredictability of socio-
political processes remain unsolved although better manageable.
Still, something relevant is available out there. Although the socio-political
complexity of policy making stays unchanged, the data shake is offering an unprece-
dented chance: the continuous production of data throughout the policy making
process (design, implementation, and evaluation) creates the chance to learn through
(not only for neither from) the policy making process. This opportunity is concrete
as never before. The wide diversity of data sources, their fast and targeted produc-
tion, the available technologies that produce easy to use analytics and visualizations
create the chance for a shift from learning for/from policy making into learning by
policy making so allowing the improvement of the substance and procedure at the
same time as a continuous process.
The learning opportunity is directly embedded in the policy making process as
the chance to shape social behaviours, responses, and achieve timely (perhaps even
real-time) effects (Dunleavy 2016) is out there. Learning by (doing in) policy making
is possible and benefits from a new role of evidence: no longer (or not only) a way to
legitimate policy decisions, no longer (or not only) an expert guide to more effective
and necessary policy making rather a means for learning, for transforming policy
making into a collective learning process. This is possible as the data shake gives
value to the evidence used over time (Parkhurst 2017) so enabling its experimental
dimension.
1 The Data Shake: An Opportunity … 9
1.3 The Smart Revolution of Data-Driven Policy Making:
The Experimental Perspective
1.3.1 About Policy Experiments and Learning Cycles
In social science, a policy experiment is any “[…] policy intervention that offers inno-
vative responses to social needs, implemented on a small scale and in conditions that
enable their impact to be measured, prior to being repeated on a larger scale, if the
results prove convincing” (European Parliament and Council 2013, art.2 (6)). Policy
experiments form a useful policy tool to manage complex long-term policy issues by
creating the conditions for “ex-ante evaluation of policies” (Nair and Howlett 2015):
learning from policy experimentation is a promising way to approach “wicked prob-
lems” which are characterised by knowledge gaps and contested understandings of
future (McFadgen and Huitema 2017); experiments carried out in this perspective,
in fact, generate learning outcomes mainly made of relevant information for policy
and under dynamic conditions (McFadgen 2013).
The concept of policy experimentation is not new. An explanatory reconstruction
of the concept development has been carried out by van der Heijden (2014), who
quoted John Dewey (1991 [1927]) and Donald Campbell (1969, p. 409) as seminal
contributions to it. In particular: Dewey already considered that policies should “be
treated as working hypotheses, not as programs to be rigidly adhered to and executed.
They will be experimental in the sense that they will be entertained subject to constant
and well-equipped observation of the consequences they entail when acted upon,
and subject to ready and flexible revision in the light of observed consequences
(pp. 202–203); while Campbell considered experimental an approach in which new
programs are tried out, as they are conceived in a way that it is possible both to learn
whether they are effective and to imitate, modify, or discard them on the basis of
apparent effectiveness on the multiple imperfect criteria available (p. 409). van der
Heijden considers that Dewey and Campbell had in mind the idea of experimenting
with the content of policy programs (testing, piloting, or demonstrating a particular
policy design), rather than the process of policy design.
Still, as van der Heijden observes, silent remains as to the actual outcomes of such
experimentations, and this consideration makes the scope of his article that develops
two main conclusions:
experimentation in environmental policy is likely to be successful if participation
comes at low financial risk and preferably with financial gain (see Baron and
Diermeier 2007; Croci 2005, quoted by van der Heijden);
in achieving policy outcomes, the content of the policy-design experimentsmatters
more than the process of experimentation.
Intercepting both policy contents and experimentation process, and focussing on
the governance design of policy making, McFadgen and Huitema (2017) identified
three types of experiments: the expert driven “technocratic” model, the participatory
10 G. Concilio and P. Pucci
Fig. 1.1 Learning effects in
policy experimentation
(extracted from Table 1 in
McFagden and Huitema
2017, pp. 3–22)
“boundary” model, and the political “advocacy” model. These models differ in their
governance design and highlight how experiments produce learning; together with
what types of learning they activate.
In the technocratic model, experts work as consultants; they are asked to produce
evidence to support or refute a claim within the context of political disagreement. In
this model, policy makers are out of the experiment, but they supply in advance the
policy problem and the solution to be tested.
In the boundary model, experiments (working on borders among different points of
view) have a double role: producing evidence but also debating norms and developing
a common understanding. In this kind of experiment, the involvement of different
actors is crucial for the experiment to be productive of knowledge and discussion at
different cognitive levels (practical, scientific, political).
In the advocacy model the experiment is aimed at reducing objections to a prede-
fined decision. These experiments are tactical and entirely governed by policy makers
who are obviously interested in involving other actors. This kind of experiment can
also be initiated by non-public actors, even with different scopes.
McFadgen and Huitema (2017) also highlight the different learning taking places
during the three different experimental models. They distinguish mainly three kinds
of learning (Fig. 1.1).
Taking into consideration the goals and the differences in participants of the three
experiment models, McFadgen and Huitema (2017) find that: technocratic experi-
ments mainly generate high levels of cognitive learning, little normative, and some
relational learning, which is mainly due to the disconnection between experiments
and the policy makers; boundary experiments are expected to produce relational and
normative learning while low levels of cognitive learning due to the large importance
assigned to debating and sharing; advocacy experiments cognitive and normative
learning are expected to be activated but little relational learning and this is due to
the intentional selection of participants.
Learning in policy experiments is crucial and is mainly related to the oppor-
tunity embedded in learning to become appropriation of the knowledge devel-
oped throughout the experiment. Consequently, the rationale behind an experi-
mental approach to policy making is to boost public policy makers’ ownership and
commitment, thus possibly increasing the chances that successful experiments are
streamlined into public policy.
The experimental dimension, especially in the boundary and the advocacy models,
is crucial in policy design and policy implementation. It makes the policy evaluation
1 The Data Shake: An Opportunity … 11
scope transversal to the other steps of the policy cycle—described by Verstraete et al.
(2021)—as well as supportive of the other steps. It transforms policy making into an
experimental process as it introduces co-design and co-experience paving the way
for embedding new points of view and new values in the context of the policy. Design
and implementation, in this perspective, become reciprocal and integrated (Concilio
and Celino 2012; Concilio and Rizzo 2012) and:
learning is enhanced and extended to participants by designing “with”, not merely
“for”;
exchange and sharing of experiences are more effective than information transfer
and sharing;
involved actors become the owners of the socio-technical solutions together with
technological actors and decision makers;
changes in behaviours (the main goal of any policy making) are activated
throughout the experiments.
Based on this, different levels of integration are possible and, among them, the
most advanced is the so-called triple-loop learning flow in policy experimentations
(Yuthas et al. 2004; see also Deliverable 3.1 by the Polivisu Project3).
1.3.2 Policy Cycle Model Under Experimental Dimension
As introduced in the previous section, the experimentations and the reflection on the
operative implications in the use of data in urban management and decision making
processes are at the base of a consistent production of critical ex-post evaluations on
the potentials and limits of data-informed policy making produced in the last years
(e.g. Poel et al. 2015; Lim et al. 2018).
The process of policy creation has been left in the background by the focus on the
content, rather than the process of policy design and, in some cases, without a proper
reflection about the selection, processing, and use of data to identify individual or
collective human needs and formulate solutions that “can be not arrived at algorith-
mically” (…); and which cannot be “encoded in public policy, without distortion
(Greenfield 2017, p. 56).
Actually, it is well accepted that a policy process is not a linear and determin-
istic process; it is a set of decisions and activities that are linked to the solution
of a collective problem where the “connection of intentionally consistent decisions
and activities taken from different public actors, and sometimes private ones (are
addressed) to solve in a targeted way a problem” (Knoepfel et al. 2011, p. 29).
In this process, data offer support for strategic activities by aggregating infor-
mation on a time series that support and validate prediction models for long-term
planning; for tactical decisions, conceived as the evidence-informed actions that are
needed to implement strategic decisions and, finally, for operational decisions, giving
3https://www.polivisu.eu/public-deliverables.
12 G. Concilio and P. Pucci
support to day-to-day decision making activities in a short-term planning perspective
(Semanjski et al. 2016).
From a policy perspective, strategic, tactical, and operational decisions use, and
are supported by, data in different ways along the stages of a policy making process.
In the design, in the implementation, and in the evaluation of a policy, data provides
insights in allowing the possibility to discover trends and to eventually analyze their
developing explanation; in fostering public engagement and civic participation; in
dynamic resource management; and, finally, in sustaining the development of “robust
approaches for urban planning, service delivery, policy evaluation and reform and
also for the infrastructure and urban design decisions” (Thakuriah et al. 2017, p. 23).
Among them, an approach in which data may support a policy making process dealing
with a different time frame and multi-actor perspectives can be based on the policy
cycle model, which means conceiving policy as a process composed of different
steps (Marsden and Reardon 2017) to which data contributes differently.
The policy cycle, here not be interpreted as a rigorous, formalistic guide to the
policy process, but as an idealized process, as a “means of thinking about the sectoral
realities of public policy processes”, has the potential to capture the potential of data
shake if used in a descriptive way more than in normative one.
This policy model can be conceptualized as a data-assisted policy experimentation
cycle, consisting of interrelated cyclical stages: the stages are strongly interdepen-
dent, integrated, and overlapping due to the broad availability of data at the core of
policy making’s experimental dimension.
In doing so, the policy cycle model can represent a “bridge”, a sort of “boundary
object” (Star and Griesemer 1989) in which different operational and disciplinary
dimensions (planning, data analytics, data mining) can interact and cross-fertilize
each other since it offers an organized structure, in which data provides a viable base
for acting in each stage.
Based on this, the major weaknesses recognized in the policy cycle model, consid-
ered too simplistic in practice, giving a false impression of linearity and discrediting
its assumption of policy as sequential in nature4(Dorey 2005; Hill 2009; Howlett and
Ramesh 2003; Ryan 1996) may be overpass thanks to the experimental perspective,
able to foster a less linear interpretation of policy cycle, transformed in a continuous
process in which overlap among policy stages.
4Among the critical arguments: the inability of this model to explain what causes policies to advance
from one stage to another, the predetermined manner in with each stage in the cycle occurring in a
precise, far from actual fact (Howlett and Ramesh 2003), because policy needs to be designed and
continuously revised to take into account external conditions and adapt to their eventual change.
Their effects are often indirect, diffuse, and take time to appear; policy making depends on politics,
people, socio-economic factors, and other previous and ongoing policies.
1 The Data Shake: An Opportunity … 13
Fig. 1.2 Decision/reasoning along diverse timeframes
1.3.3 The Time Perspective in the Experimental Dimension
of Policy Making5
Decisions for and about cities are made at different urban scales, refer to different
strategic levels and have different time perspectives, with reciprocal interdependen-
cies that are changing due to data availability. Here we mainly focus on the interplay
between the different steps of decisions in policy making (those introducing the
long-time perspective) and those necessary for the daily management of the city
(connected to the shortest, real-time perspective), an intersection at which data can
play a key role (Fig. 1.2).
Short-term management is embedded in the smart sphere of decisions impacting
cities: here decisions are less analytic and more routine. Routines may depend on
data-driven learning mechanisms (also using data series) supporting smart systems
to recognize situations and apply solutions and decisions that have already been
proven to work. The decision has a temporary value related to the specific conditions
detected in a precise moment by the smart system.
Opposite to real-time decisions, policy making works in a long-time perspective.
Anticipatory is the prevailing mode for reasoning in this case data-driven models are
often adopted as supporting means to deal with the impacts of the policy measures,
representing thus a relevant source for exploring decision options mainly having
a strategic nature (since they consider recurring issues and aim at more systemic
changes).
Between short-term and long-term decisions a variety of situations is possible,
which may be considered as characterized by decisions having a reversible nature:
they are neither strategic in value (like those oriented to a long term perspective
for systemic changes), nor aiming at dealing with temporary, contingency situa-
tions asking decisions which are known as having the same (short) duration of the
phenomenon to be managed. For such decisions, the reasoning is not (fully) anticipa-
tory and its temporariness allows reflection as embedded in action. Within the three
5This paragraph is belonging to a recent publication: Concilio G, Pucci P, Vecchio G, Lanza G
(2019) Big data and policy making: between real time management and the experimental dimension
of policies. In: Misra S. et al. (eds) Computational science and its applications—ICCSA 2019.
Lecture Notes in Computer Science, vol 11620, Springer, Cham, pp 191–202.
14 G. Concilio and P. Pucci
different timeframes, actions are different in nature and show different use and role
of data:
in the short term, the action (the smart action) is mainly reactive; real time data
are used as reference info to interpret situations;
in the medium term, the action is mainly adaptive; data series, including current
data, are used to detect impacts of the action itself and to improve it along time;
in the long term, the action has a planning nature; data series become crucial to
detect problems and to develop scenarios for long lasting changes.
The interdependency between policy design, implementation and evaluation is
strictly related to two factors, especially when considering the role (big) data can
play. Design and implementation can be clearly and sequentially distinguished when
a systematic, impact-oriented analysis is possible at the stage of design as it allows
a clear costs and benefits assessment of different action opportunities (Mintzberg
1973).
Comprehensive analyses have the value to drive long range, strategic actions,
and consequently show a clear dependency and distinction of the implementation
from the design cycle. At the same time, the bigger is the uncertainty (not only
related to the possible lack of data, rather also consequent to the high complexity
of the problem/phenomenon to be handled), the smaller is the chance to carry out a
comprehensive analysis.
Therefore, goals and objectives cannot be defined clearly and the policy making
shifts from planning towards an adaptive mode. Inevitably, this shift reduces the
distance between design and implementation, transforming policy design into a more
experimental activity that uses learning from implementation into food for design
within adaptive dynamics (Fig. 1.3).
Coherently with the discussion on the time frame perspective adopted, it may
be clear that a merge between policy design and implementation is consistent with
Fig. 1.3 Real time management vs policy making
1 The Data Shake: An Opportunity … 15
the situation described in the medium term: within an adaptive mode for decisions,
policy making can clearly become experimental.
1.4 Conclusions: Beyond the Evidence-Based Model
Evidence based policy making is surely the key conceptual reference when trying
to grasp the potentials that the growing availability of data and related technologies
offer to policy making. As it is clear from the previous paragraphs, the concept has
been widely discussed in the literature and can be considered the key antecedent of
experiment-driven policy making.
Experiments may refer to both the policy strategies and measures. They can reduce
the risk of trial-errors approaches while considering the learning in action opportunity
to improve, adapt, adjust the policy while making it in order to increase its capacity
to affect the context in an evolving manner.
Differently from Mintzberg’s considerations (1973), the merge between policy
design and implementation does not represent a sort of inevitable, but not preferred
option when a comprehensive analysis is not possible. In the era of data availability,
this merge can be looked at as an opportunity to create policies while verifying the
policies themselves throughout their interactions with the contexts.
The growing availability of diverse and rich data sets represents an opportunity for
evidence to be transformed into a more valuable resource then what it was intended
to be by the evidence-based policy making supporters: not only, or not necessarily, a
means to support the scientific rationality of the decision making process, rather the
drivers to reflection and learning through action.
References
Androutsopoulou A, Charalabidis Y (2018) A framework for evidence based policy making
combining big data, dynamic modelling and machine intelligence. In: Kankanhalli A, Ojo A,
Soares D (eds) Proceedings of the 11th international conference on theory and practice of
electronic governance, Galway, Ireland, 4–6 April 2018, pp 575–583
Baron DP, Diermeier D (2007) Strategic activism and nonmarket strategy. J Econ Manag Strat
16:599–634
Cairney P (2017) The politics of evidence-based policy making. In: Oxford research encyclopedia
of politics. Available at http://eprints.lse.ac.uk/68604/1/Parkhurst_The%20Politics%20of%20E
vidence.pdf. Accessed on June 2020
Campbell D (1969) Reforms as experiments. Am Psychol 24(4):409–429
Concilio G, Celino A (2012) Learning and innovation in living Lab environments. In: Schiuma
G, Spender JC, Yigitcanlar T (eds) Proceedings of the international forum on knowledge assets
dynamics—knowledge, innovation and sustainability: integrating micro and macro perspective,
Matera, Italy, 13–15 June 2012
16 G. Concilio and P. Pucci
Concilio G, Rizzo F (2012) Enabling situated open and participatory design processes by exploiting
a digital platform for open innovation in smart cities. In: Miettinen S, Valtonen A (eds) Service
design with theory, Lapland University Press, pp 66–72
Croci E (2005) The handbook of environmental voluntary agreements. Springer, Dordrecht
Davies P (2004) Is evidence-based government possible? Jerry Lee Lecture presented at the 4th
annual campbell collaboration colloquium, Washington, DC. 19 February 2004
Dewey J (1991 [1927]) The public and its problems. Swallow Press, Ohio
Dorey P (2005) Policy making in Britain: an introduction. Sage, London, UK
Dunleavy P (2016) ‘Big data’ and policy learning. In: Stoker G and Evans M (eds) Evidence-based
policymaking in social science. Methods that matter. The Policy Press, Bristol, pp 143–168
EC (European Commission) (2013) Guide to social innovation. Brussels, European Commission—
DG Regional and Urban Policy
Einav L, Levin JD (2013) The data revolution and economic analysis. NBER Working Paper Series,
19035
Greenfield A (2017) Radical technologies: the design of everyday life. Verso, Brooklyn
Kitchin R (2014) Big data, new epistemologies and paradigm shifts. Big Data Soc 1(1)
Knoepfel P, Larrue C, Varone F, Hill M (2011) Public policy analysis. The Policy Press, Bristol
Hammersley M (2005) Is the evidence-based practice movement doing more good than harm?
Reflections on Iain Chalmers’ case for research-based policy making and practice. Evid Policy
1(1):85–100
Hill M (2009) The public policy process. Pearson, London
Howlett M (2009) Policy analytical capacity and evidence-based policy-making: lessons from
Canada. Can Public Adm / Administration publique du Canada 52(2) (june/juin 2009):153–175
Howlett M, Ramesh M (2003) Studying public policy: policy cycles and Policy sub-systems. Oxford
University Press, Oxford
Innvaer S, Vist G, Trommald M, Oxman A (2002) Health policy-makers’ perceptions of their use
of evidence: a systematic review. J Health Serv Res Policy 7(4):239–245
Jackson PM (2007) Making sense of policy advice. Public Money Manag 27(4):257–264
Laforest R, Orsini M (2005) Evidence-based engagement in the voluntary sector: lessons from
Canada. Soc Policy Adm 39(5):481–497
Lim C, Kim KJ, Maglio PP (2018) Smart cities with big data: reference models, challenges and
considerations. Cities 82:86–99
Lomas J, Brown A (2009) Research and advice giving: a functional view of evidence informed
policy advice in a Canadian ministry of health. Milbank Q 87(4):903–926
Majone G (1989) Evidence, argument, and persuasion in the policy process. Yale University Press,
New Haven, CO
Marsden G, Reardon L (2017) Questions of governance: rethinking the study of transportation
policy. Transp Res Part A: Policy Pract 101 238–251. https://doi.org/10.1016/j.tra.2017.05.008
Marston G, Watts R (2003) Tampering with evidence: a critical appraisal of evidence-based policy
making. Draw Board: Aust Rev Public Aff 3(3):143–163
Mattern S (2017) Mapping’s intelligent agents. Places J. https://doi.org/10.22269/170926
McFadgen B (2013) Learning from policy experiments in adaptation governance. Paper presented
at the 1st international conference on public policy. Grenoble, France, 26–28 June 2013
McFadgen B, Huitema D (2017) Stimulating learning through policy experimentation: a multi-case
analysis of how design influences policy learning outcomes in experiments for climate adaptation.
Water 9(9):648
Mintzberg H (1973) Strategy-making in three modes. Calif Manag Rev 16(2):44–53
Nair S, Howlett M (2015) Scaling up of policy experiments and pilots: a qualitative comparative
analysis and lessons for the water sector. Water Resour Manag 29:4945–4961
Nutley S M, Walter I, Davies HTO (2007) Using evidence: how research can inform public services.
Policy Press, Bristol
Parkhurst J (2017) The politics of evidence. from evidence-based policy to the good governance of
evidence. Routledge, Abingdon
1 The Data Shake: An Opportunity … 17
Parsons W (2001) Modernising policy-making for the twenty first century: the professional model.
Public Policy Adm 16(3):93–110
Pawson R (2006) Evidence-based policy: a realist perspective. Sage, London
Pawson R, Greenhalgh T, Harvey G, Walshe K (2005) Realist review—a new method of systematic
review designed for complex policy interventions. J Health Serv Res Policy 10 (Supplement
1):21–34
Poel M, Schroeder R, Treperman J, Rubinstein M, Meyer E, Mahieu B, Scholten C, Svetachova
M (2015) Data for policy: a study of big data and other innovative data-driven approaches
for evidence-informed policymaking. Report about the state-of-the-art. Joint venture between
Technopolis group, Oxford internet Institute and the Centre for European Studies
Pucci P, Vecchio G, Concilio G (2018) Big data and urban mobility: a policy making perspective.
Transportation Research Procedia 2352–1465, Elsevier B.V. World Conference on Transport
Research, Mumbai, India, 26–31 May 2019
Rabari C, Storper M(2015) The digital skin of cities: urban theory and research in the age of the
sensored and metered city, ubiquitous computing and big data. Cambridge Journal of Regions.
Econ Soc 8(1):27–42. https://doi.org/10.1093/cjres/rsu02
Radin BA, Boase JP (2000) Federalism, political structure, and public policy in the United States
and Canada. J Comp Policy Anal 2(1):65–90
Russell J, Greenhalgh T, Byrne E, Mcdonnell J (2008) Recognizing rhetoric in health care policy
analysis. J Health Serv Res & Policy 13(1):40–46
Ryan N (1996) Some advantages of an integrated approach to implementation analysis: a study of
the Australian industrial policy. Public Adm 74(4):737–753
Sanderson I (2006) Complexity, ‘practical rationality’ and evidence-based policy making. Policy
Polit 34 (1):115–1322
Semanjski I, Bellens R, Gautama S, Witlox F (2016) Integrating big data into a sustainable mobility
2.0 planning support system. Sustainability, 8, 1142
Star S, Griesemer J (1989) Institutional ecology, ‘translations’ and boundary objects: amateurs and
professionals in Berkeley’s museum of vertebrate zoology, 1907–39. Soc Stud Sci 19(3):387–420
Thakuriah P, Tilahun NY, Zellner M (2017) Big data and urban informatics: innovations and chal-
lenges to urban planning and knowledge discovery. In: P. Thakuriah et al. (eds) Seeing cities
through big data. Springer Geography
Trinder L (2000) Introduction: the context of evidence-based practice. In: Trinder L, Reynolds S
(eds) Evidence-based practice: a critical appraisal. Blackwell Science, Oxford
van der Heijden J (2014) Experimentation in policy design: insights from the building sector. Policy
Sci 47:249–266
Verstraete J, Acar F, Concilio G, Pucci P (2021) Turning data into actionable policy insights. In: G
Concilio, P Pucci, L Raes, G Mareels (eds) The data shake. opportunities and obstacles for urban
policy making. Springer, PolimiSpringerBrief
Young K, Ashby D, Boaz A, Grayson L (2002) Social science and the evidence-based policy
movement. Soc Policy Soc 1(3):215–224
Yuthas K, Dillard J, Rogers R (2004) Beyond agency and structure: triple-loop learning. J Bus
Ethics 51(2):229–243
Grazia Concilio Associate professor in Urban Planning and Design at DAStU, Politecnico di
Milano. She is an engineer; PhD in “Economic evaluation for Sustainability” from the Univer-
sity of Naples Federico II. She carried out research activities at the RWTH in Aachen, Germany
(1995), at IIASA in Laxenburg, Austria (1998) and at the Concordia University of Montreal,
Canada, (2002); she is reviewer for several international journals and former member (in charge
of assessment task of LL new applications) of ENoLL (European Network of open Living Lab).
Team member in several research projects; responsible for a CNR research program (2001)
and coordinator of a project funded by the Puglia Regional Operative Programme (2007-2008)
and aiming at developing an e-governance platform for the management of Natural Parks.
18 G. Concilio and P. Pucci
She has been responsible on the behalf POLIMI of the projects Peripheria (FP7), MyNeigh-
bourhood|MyCity (FP7), Open4Citizens (Horizon 2020 www.open4citizens.eu); she is currently
responsible for the Polimi team for the projects Designscapes (Horizon 2020 www.design
scapes.eu), Polivisu (Horizon 2020 www.polivisu.eu) together with Paola Pucci, and MESOC
(Horizon 2020 www.mesoc-project.eu). She is coordinating the EASYRIGHTS project (Horizon
2020 www.easyrigths.eu). She is the author of several national and international publications.
Paola Pucci Full Professor in Urban planning, she has been Research Director of the PhD course
in Urban Planning Design and Policy (2012–2018) at the Politecnico di Milano. From 2010 to
2011 she taught at the Institut d’Urbanisme in Grenoble Université Pierre Méndes France at Bach-
elor, Master and PhD levels and currently visiting professor at European universities. She has
taken part, also with roles of team coordinator, in national and international research projects
funded on the basis of a competitive call, dealing with the following research topics: Mobility
policy and transport planning, mobile phone data and territorial transformations and including
EU ERA-NET project “EX-TRA – EXperimenting with city streets to TRAnsform urban mobil-
ity”; H2020 - SC6-CO-CREATION-2016-2017 “Policy Development based on Advanced Geospa-
tial Data Analytics and Visualisation”, EU Espon Project, PUCA (Plan, urbanisme, architec-
ture) and PREDIT projects financed by the Ministère de l’Ecologie, du Développement et de
l’Aménagement durable (France). She has supervised and refereed different graduate, postgrad-
uate and PhD theses at Politecnico di Milano, Université Paris Est Val de Marne, Ecole Superieure
d’Architecture de Marseille, Université de Tours. She has been Member of the evaluation panel
for the Netherlands Organisation for Scientific Research (NWO, 2017), and Member of the
NEFD Policy Demonstrators commissioning panel for the ESRC _Economic and Social Research
Council. Shaping Society (Uk), on the topic “New and Emerging Forms of Data - Policy Demon-
strator Projects (2017).
Open Access This chapter is licensed under the terms of the Creative Commons Attribution 4.0
International License (http://creativecommons.org/licenses/by/4.0/), which permits use, sharing,
adaptation, distribution and reproduction in any medium or format, as long as you give appropriate
credit to the original author(s) and the source, provide a link to the Creative Commons license and
indicate if changes were made.
The images or other third party material in this chapter are included in the chapter’s Creative
Commons license, unless indicated otherwise in a credit line to the material. If material is not
included in the chapter’s Creative Commons license and your intended use is not permitted by
statutory regulation or exceeds the permitted use, you will need to obtain permission directly from
the copyright holder.
... The stages and steps do not follow each other in a linear manner, rather they are defined as overlapping and cyclical. Moreover, the stages of policy making tend to become more integrated and overlapping when data is involved (Concilio and Pucci 2021). Before discussing this model in more detail, a common understanding is required of the data-based analysis types that can be employed during the policy making process. ...
... Furthermore, at any stage in the policy making process it is possible to go back a few steps and restart the process guided by the lessons learnt. This is the key core of the impact of data on policy making: it is transformed into a trial/error process where it is essential to register, reuse and share the outcomes and learnings developed throughout the process (Concilio and Pucci 2021). By virtue of these learnings the data maturity of the organization, the public administration, grows, bringing new insights to the field. ...
... It is then absolutely necessary to include clauses on public procurements to access the data and, when necessary, to have included also a good format making it quickly usable. Cities should also consider whether, in addition to getting access to the data themselves, the contracts should require the supplier to make the data available as open data or to other private sector actors on a fair and equitable basis so that innovation and societal benefit can be maximised Concilio G, Pucci P (2021) The data shake. An opportunity for experiment-driven policy making. ...
Book
Full-text available
This open access book represents one of the key milestones of PoliVisu, an H2020 research and innovation project funded by the European Commission under the call “Policy-development in the age of big data: data-driven policy-making, policy-modelling and policy-implementation”. It investigates the operative and organizational implications related to the use of the growing amount of available data on policy making processes, highlighting the experimental dimension of policy making that, thanks to data, proves to be more and more exploitable towards more effective and sustainable decisions. The first section of the book introduces the key questions highlighted by the PoliVisu project, which still represent operational and strategic challenges in the exploitation of data potentials in urban policy making. The second section explores how data and data visualisations can assume different roles in the different stages of a policy cycle and profoundly transform policy making.
... The stages and steps do not follow each other in a linear manner, rather they are defined as overlapping and cyclical. Moreover, the stages of policy making tend to become more integrated and overlapping when data is involved (Concilio and Pucci 2021). Before discussing this model in more detail, a common understanding is required of the data-based analysis types that can be employed during the policy making process. ...
Chapter
Full-text available
It is becoming clearer that data-supported input is essential in the policy making process. But at which point of the process, and in which format, can data aid policy making? And what does an organisation need to turn data into relevant insights? This paper explores the role of data from two perspectives. In the first part, data and data analysis are situated in the policy making process by mapping them onto the data supported policy making model and highlighting the different roles they can assume in each stage and step of the process. The second part discusses a practical framework for policy-oriented data activities, zooming in on the data-specific actions and the actors performing them in each data-supported step of the policy making process. We observe that a close collaboration between the policy maker and data scientist in the framework of an iterative approach permits to transform the policy question into a suited data analysis question and deliver relevant insights with the flexibility desired by decision makers. In conclusion, for data to be turned into actionable policy insights it is vital to set up structures that ensure the presence and the collaboration of policy-oriented and data-oriented competences.
Article
Full-text available
Evidence-based decision making is promoted as offering efficiency and effectiveness; however, its uptake has faced barriers such as underdeveloped supporting culture, limited access to evidence, and evidence that is not fully relevant. Smart city conceptualizations offer economic and environmental sustainability and better quality of life through evidence-based policy decision-making. We wondered whether smart city theory and practice has advanced the knowledge of evidence-based decision-making. We searched major databases for literature containing a mention of smart cities, decision-making, and policy. We identified relevant literature from a range of disciplines and supplemented these by following backwards and forwards citations. Evidence-based decision-making was found mostly in literature regarding the theory and practice of smart city operations, and, to lesser extents, the articles regarding policy decisions and tactical decisions. Better decision-making which supported the achievement of city sustainability objectives was reported in some articles; however, we found significant obstacles to the further achievement of city objectives in the areas of underachievement in collaborative decision-making, privileging of big data evidence, and artificial intelligence agents as decision-makers. We assembled a definition of smart city decision-making and developed an agenda of research which will support city governments, theorists, and practitioners in better achieving sustainability through improved decision-making.
Chapter
Full-text available
It is becoming clearer that data-supported input is essential in the policy making process. But at which point of the process, and in which format, can data aid policy making? And what does an organisation need to turn data into relevant insights? This paper explores the role of data from two perspectives. In the first part, data and data analysis are situated in the policy making process by mapping them onto the data supported policy making model and highlighting the different roles they can assume in each stage and step of the process. The second part discusses a practical framework for policy-oriented data activities, zooming in on the data-specific actions and the actors performing them in each data-supported step of the policy making process. We observe that a close collaboration between the policy maker and data scientist in the framework of an iterative approach permits to transform the policy question into a suited data analysis question and deliver relevant insights with the flexibility desired by decision makers. In conclusion, for data to be turned into actionable policy insights it is vital to set up structures that ensure the presence and the collaboration of policy-oriented and data-oriented competences.
Article
Full-text available
Cities worldwide are attempting to transform themselves into smart cities. Recent cases and studies show that a key factor in this transformation is the use of urban big data from stakeholders and physical objects in cities. However, the knowledge and framework for data use for smart cities remain relatively unknown. This paper reports findings from an analysis of various use cases of big data in cities worldwide and the authors' four projects with government organizations toward developing smart cities. Specifically, this paper classifies the urban data use cases into four reference models and identifies six challenges in transforming data into information for smart cities. Furthermore, building upon the relevant literature, this paper proposes five considerations for addressing the challenges in implementing the reference models in real-world applications. The reference models, challenges, and considerations collectively form a framework for data use for smart cities. This paper will contribute to urban planning and policy development in the modern data-rich economy.
Article
Full-text available
Learning from policy experimentation is a promising way to approach the “wicked problem” of climate adaptation, which is characterised by knowledge gaps and contested understandings of future risk. However, although the role of learning in shaping public policy is well understood, and experiments are expected to facilitate learning, little is known about how experiments produce learning, what types of learning, and how they can be designed to enhance learning effects. Using quantitative research methods, we explore how design choices influence the learning experiences of 173 participants in 18 policy experiments conducted in the Netherlands between 1997 and 2016. The experiments are divided into three “ideal types” that are expected to produce different levels and types of learning. The findings show that policy experiments produce cognitive and relational learning effects, but less normative learning, and experiment design influenced three of six measured dimensions of learning, especially the cognitive learning dimensions. This reveals a trade-off between designing for knowledge development and designing for normative or relational changes; choices that experiment designers should make in the context of their adaptation problem. Our findings also show the role leadership plays in building trust.
Article
The relationship that exists between design, policies and governance is quite complex and presents academic researchers continuously with new opportunities to engage and explore aspects relevant to design management. Over the past years, we have witnessed how the earlier focus on developing policies for design has shifted to an interest in understanding the ways in which design contributes to policy-making and policy implementation. Research into policies for design has produced insights into how policy-making decisions can advance professional impact and opportunities for designers and the creative industries. This research looked into how design researchers and design practitioners themselves can benefit from specific policies that support design activities and create the space for emerging design processes.
Conference Paper
Governments and policy makers are striving to respond to contemporary socio-economic challenges, however, often neglecting the human factor and the multidimensionality of policy implications. In this chapter, a framework for evidence based policy making is proposed, which integrates the usage of open big data coming from a multiplicity of sources with policy simulations. It encompasses the application of dynamic modelling methodologies and data mining techniques to extract knowledge from two types of data. On the one hand, objective data such as governmental and statistical data, are used to capture the interlinked policy domains and their underlying casual mechanisms. On the other hand, behavioural patterns and citizens' opinions are extracted from Web 2.0 sources, social media posts, polls and statistical surveys. To combine this multimodal information, our approach suggests a modelling methodology that bases on big data acquisition and processing for the identification of significant factors and counterintuitive interrelations between them, which can be applied in any policy domain. Then, to allow the practical application of the framework an ICT architecture is designed, with the aim to overcome challenges related with big data management and processing. Finally, validation of the approach for driving policy design and implementation in the future in diverse policy domains, is suggested.
Book
EDOARDOCROCI IEFE - Università Bocconi, Milano, Italy Voluntary approaches in environmental policy represent a “third wave” of regulation in the environmental field. “Command and control” was the first wave. Its core is based on uniform emission standards, the respect of which needs to be enforced through extensive monitoring and severe sanctions. The expected cost of sanction for non-compliance, calculated as its amount multiplied for the probability to be caught, must be superior to the benefits of non-compliance, in order to let the sanction be effective. As the benefits of non-compliance can vary among firms, sanctions need to be very high in order to be effective. In fact sanctions are ordinary correlated to environmental damage and not to the benefits of non-compliance. But very high sanctions can be difficult to enforce as they appear unfair and can lead to dramatic consequences on firms and workers, up to shut-downs of plants. Ambient standards reduce these problems, but oblige the regulator to know a huge amount of information, regarding the specific contribution of each polluter to the polluted body. Information is difficult to obtain because of asymmetric information and costly to produce because it requires large and skilled regulating and enforcing organizations. Nevertheless complex regulation is the base of any environmental policy framework, as it allows the policy maker to fully exercise its power of composition of various interests in a relatively transparent way. Economic instruments were the second wave.
Book
This book is an English version of a successful text* on public policy analysis originally written for policy practitioners in Switzerland and France. It presents a model for the analysis of public policy and includes examples of its application in everyday political-administrative situations. This English version introduces supplementary illustrations and examples from the United Kingdom. Structured and written accessibly for readers who may not have an academic background in the social sciences, Public Policy Analysis applies key ideas from sociology, political science, administrative science and law to develop an analytical framework that can be used to carry out empirical studies on different public policies. British scholars, practitioners and students are introduced all too rarely to ideas from the Francophone world, and this book will contribute to remedying that. It will be particularly relevant for students and practitioners of public administration. © Peter Knoepfel, Corinne Larrue, Frédéric Varone and Michael Hill 2007.
Article
Questions of Governance: Rethinking the Study of Transportation Policy Greg Marsdena and Louise Reardonb a Institute for Transport Studies, University of Leeds, 1-3 Lifton Villas, Lifton Place University of Leeds, Leeds, United Kingdom, LS2 9JT g.r.marsden@its.leeds.ac.uk b Corresponding author Institute for Transport Studies, University of Leeds, 1-3 Lifton Villas, Lifton Place University of Leeds, Leeds, United Kingdom, LS2 9JT l.reardon@leeds.ac.uk Abstract This paper critiques the state of the art approaches to studying transportation policy. It does so through analysing 100 papers sampled from the two leading policy journals in the transportation literature. On applying two different frameworks for understanding policy, the review finds that only 13% of papers consider specific aspects of the policy cycle, that 60% focus on ‘tools’ for policy, and that two-thirds of papers did not engage with real-world policy examples or policy makers and focussed on quantitative analysis alone. We argue that these findings highlight the persistence of the technical-rational model within the transportation literature. This model, and the numerous traditions and disciplines that have fed into it have an important role to play in developing the transportation evidence base. However, we argue there are important questions of governance; such as context, power, resources and legitimacy, that are largely being ignored in the literature as it stands. The substantial lack of engagement with governance issues and debates means that as a field we are artificially, but more importantly, disproportionately generating a science of applied policy making which is unlikely to be utilised because of the distance between it and the realities on the ground. The paper identifies analytical approaches deployed readily in other fields that could be used to address some of the key deficiencies.