Can we measure public sector innovation? A literature review

Abstract

The aim of this article is to give an overview of scholarly state-of-the-art in terms of both
conceptualizing and measuring public sector innovations. In order to do so, the article
consists of following sections: first, we give a brief overview of prevailing attempts to
conceptualize (define) public sector innovation and contrast it with older literature
(Tocqueville, Weber, Schumpeter); second, we briefly summarize private sector innovation
performance measurements and indicators; third, we discuss state of the art in discussions of
measuring public sector performance in general, and look at recent discussions of public
sector productivity, what and how can be measured; fourth, we discuss recent projects and
literature on measuring public sector innovation; this is followed by brief overview of
attempts to measure social innovations and finally we conclude by drawing these various
discussions together by detailing what and how can we measure in public sector innovation
with good scholarly conscious.

Full text

1
Can we measure public sector innovation? A literature review
LIPSE Project Working Paper No 2
Authors: Rainer Kattel, Aleksandrs Cepilovs, Wolfgang Drechsler, Tarmo
Kalvet, Veiko Lember and Piret Tõnurist
Ragnar Nurkse School of Innovation and Governance, Tallinn, Estonia
1-9-2014
To be cited as Rainer Kattel, Aleksandrs Cepilovs, Wolfgang Drechsler,
Tarmo Kalvet, Veiko Lember and Piret Tõnurist. Can we measure public
sector innovation? A literature review. LIPSE Working papers (no. 2).
Rotterdam: Erasmus University Rotterdam.
LIPSE or ''Learning from Innovation in Public Sector Environments'' is a
research consortium of 12 institutions in 11 countries, studying the drivers
and barriers of successful social innovation in the public sector. It is
funded by the 7th Framework Programme of the European Union (N.
320090).
www.lipse.org

2
Introduction
A ghost is making rounds in European capitals – a slow realization that austerity may perhaps
not be the answer to European crisis. (Wolf 2013; Draghi 2014) Arguably more than anything
else, this realization has made public sector innovation
1
seem like a panacea to European ills.
This is understandable in times when public sector’s fiscal capabilities are constrained and
private sector growth is not enough to haul European economies out of the slow-burning
recessionary environment. Public sector innovation promises to deliver more with less.
However, scholarly literature on public sector innovation has been tormented since its
inception in management and organization theory writings few decades ago by recurring
bangs of consciousness: is there such a thing as public sector innovation to begin with?
2
If we
cannot delineate and define public sector innovation then the concept would denote any good
idea or positive change in the public sector organizations as innovations and “will lose
credibility because it has no meaning.” (Lynn 1997, 98) As Pollitt put it, “the sad truth is that
many of today’s management seminars on innovation are filled with a promiscuous litter of
buzz words and woolly concepts whilst being almost entirely bereft of any specific,
empirically grounded propositions.” (Pollitt 2011) Thus, it is not surprising that recent years
have brought increasing interest in conceptualizing and measuring public sector innovation in
ways that are both scholarly sound and useful in policy making and evaluation contexts. That
is not an easy task, as Mintzberg has succinctly summarized problems with measuring public
sector performance in general: “Many activities are in the public sector precisely because of
measurement problem: If everything was crystal clear and every benefit so easily attributable,
those activities would have been in the private sector long ago.” (1996)
The aim of this article is to give an overview of scholarly state-of-the-art in terms of both
conceptualizing and measuring public sector innovations. In order to do so, the article
consists of following sections: first, we give a brief overview of prevailing attempts to
conceptualize (define) public sector innovation and contrast it with older literature
(Tocqueville, Weber, Schumpeter); second, we briefly summarize private sector innovation
performance measurements and indicators; third, we discuss state of the art in discussions of
measuring public sector performance in general, and look at recent discussions of public
sector productivity, what and how can be measured; fourth, we discuss recent projects and
literature on measuring public sector innovation; this is followed by brief overview of
attempts to measure social innovations and finally we conclude by drawing these various
discussions together by detailing what and how can we measure in public sector innovation
with good scholarly conscious.
Section 1. Defining public sector innovation
By and large we can divide scholarly efforts to delineate and conceptualize public sector
innovation into three periods: 1) Schumpeterian period: innovations and public sector are
related to a larger theory of how evolutionary change takes place in societies, mainly
associated with Schumpeter (1912&1939); 2) organizational theory period: innovations in
the public sector are seen similar to innovations in private companies, mostly associated with
early organizational theory and with Wilson (1989); 3) autochthonous theory period is the
most recent trend to disassociate public and private sector innovations.
1
We use public sector innovation to include also social innovation, that is, we do not delineate between these
concepts and discuss social innovation only tentatively in later parts of the paper.
2
Lynn (1997) gives an overview of early literature on the topic.

3
Schumpeterian period is characterized by Schumpeter’s theory of innovation which in fact is
an application in economics and business of his wider theory of how evolutionary change
takes place in societies. Alas, Schumpeter never really developed his wider theory of social
change (see also Andersen 2009). In his 1939 Business Cycles, Schumpeter states, in a
footnote, that he “believes, although he cannot stay to show, that theory [of innovation] here
expounded is but a special case, adapted to the economic sphere, of a much larger theory
which applies to change in all spheres of social life, science and art included.” (1939, 97) His
1912 Theorie der wirtschaftlichen Entwicklung / The Theory of Economic Development
3
assumes apparently a similar theory, without going into greater details either. We can deduce
that what Schumpeter meant by this larger theory of change in social life is that change is
driven by entrepreneurial, creative persons, or “new men” as he called them in 1939, that
look for “new combinations”, that is innovative solutions and thus bring forth evolutionary
changes, entirely new ways of doing things (in business, politics, art, science, etc) that will
spread, in some cases more than others, throughout the given sphere of life.
4
Some of these
changes will change value systems and disrupt incumbent hierarchies.
5
In economic sphere, such individuals drive innovations and, thus, economic growth. The role
of the public sector in entrepreneurial innovation is twofold: first, public sector can take on
the role of the entrepreneur (in fact, Schumpeter argues that in socialism, as there is no
private ownership, the state will be the sole innovator; 1912, 173); second, innovations in
businesses can also be “called forth” by governments (1939, 84).
In sum, what we can take from Schumpeter is that since early theories of innovation, public
sector has had dual character vis-à-vis innovation: it itself can be changed by innovators and
the state can play a crucial role for business innovations as well (either by directly leading or
indirectly supporting entrepreneurial activity). As we will see below, this foreshadows rather
closely currently emerging conceptual dichotomy between innovations in public sector and
innovations through public sector. (European Commission 2013; EU Expert Group on Public
Sector Innovation 2013)
Organizational theory period. Research explicitly dealing with innovation in the public sector
goes back at least to 1960s; however, its inception seems somewhat accidental in nature.
Researchers in organizational theory dealing with innovation and how organizational
structure supports creative work and novel ideas often did not differentiate between public
and private sector organizations (this non-differentiation goes, in fact, back to Taylor’s
Principles of Scientific Management as well as to Weber’s bureaucracy as an ideal type for
both public and private organizations). For instance, Thompson talks explicitly about
business and government organizations and their “capacity to innovate” (1965, 1), and
defines innovation as the “generation, acceptance, and implementation of new ideas,
processes, products or services. Innovation therefore implies the capacity to change or adapt.”
3
We use here the German original first edition as in later editions (that served as the basis for English
translation as well), these discussions were cut by Schumpeter; so, e.g., the second chapter of the original
edition runs to almost 100 pages, the English translation carries only half as many. In this chapter, Schumpeter
discusses his theory of innovation.
4
“Das erste Moment, die Freude am Neugestalten, am Schaffen neuer Formen der wirtschaftlichen Dinge ruht
auf ganz denselben Grundlagen wie das schöpferische Tun des Künstlers, des Denkers oder des Staatsmannes.”
(1912, 142)
5
“Sie werden Neues schaffen und Altes zerstören, kühne Pläne irgendwelcher Art konzipieren und durchführen,
deren Originalität aller Erfassung zu spotten scheint, ihre Mitbürger ihrer Herrschaft unterwerfen, vielleicht die
nationale Politik und Organisation beinflussen, den ‘natürlichen’ Gang der Wirtschaft durch gesetzliche und
ungesetzliche Mittel und jedenfalls anders als durch ‘Tausch’ abändern uws.” (1912, 157)

4
(1965, 2; see also, e.g., Mohr 1969) Much of the subsequent management and organization
theory literature dealing with innovation moves effortlessly from private to public sector and
back, and deals in fact mostly with the paradox of managers calling for innovative ideas that
end up meeting resistance in implementation often from the same managers or organizational
structures (Lynn 1997). This strand of research dealt mostly with diversity of tasks and
incentives in an organization (Becker and Whisler 1967 is a good overview). One of the key
figures in this tradition is James Q. Wilson, who’s definition of (public sector) organizational
innovation remained largely the same from 1960s to 1980s: “real innovations are those that
alter core tasks; most changes add to or alter peripheral tasks” (1989, 225) Wilson, without
referring to Schumpeter, understood these alternations in core tasks to be evolutionary in
nature and in impact: “Government agencies change all the time, but the most common
changes are add-ons; new program is added on to existing tasks without changing the core
tasks or altering the organizational culture”. (Ibid.)
Thus, there is a rather extensive literature that emerged from organizational theory that
incidentally or purposefully deals with public sector innovation and where the latter is
defined more or less similarly through 1960s to 1990s. This literature uses more or less varied
Schumpeterian notion of innovation, but it does not differentiate almost at all between private
and public sectors and thus innovations in any organization can be defined as significant and
enduring changes in core tasks. This way innovation should be different from incremental
changes in organizations (public or private) and in fact are similar to (technological)
breakthroughs familiar from the private sector evolutionary literature (see, e.g., Lynn 1997
who explicitly uses the concept of breakthrough).
6
Autochthonous theory period. In 2000s, literature dealing with public sector innovation tries
to move away both from private sector Schumpeterian approaches emphasizing novelty in
action and from organizational level changes towards innovation genuinely attributable to
public sector and towards discussing innovations in public services and governance. (See,
e.g., Hartley 2005; Moore and Hartley 2008; also Verhoest et al 2006; Pollitt 2011) However,
while there is a distinct attempt to discuss public sector phenomena (i.e. decentralization of
agencies or regions) and move away from the private sector categorization and concepts
(such as product, service and other types of innovations, concepts of life cycles and
trajectories), there is hardly any substantial change in terms of conceptually differentiating
public sector innovations from the private sector ones. The main tenets are still changes that
are new to the organization and that are large and durable enough (e.g., Hartley 2005, 27;
Moore and Hartley 2008, 5). Hartley, for instance, delivers a useful discussion of the
difference between public sector innovations in traditional, new public management and
network-based paradigms of public administration (2005, 28-30). Yet, her conceptual
framework is hardly different from Wilson. Similarly to organizational theory literature, also
the most recent literature on public sector innovation sees in the end innovations in public
sector as something different from incremental improvements and that can also fail and not
lead to better public service. Thus, e-voting would constitute for most public sector
6
Ironically, while this is indeed important for the early Schumpeterian literature, from 1970s and 1980s
onwards, evolutionary economics develops complex theoretical frameworks that show how routine-based
individual skills and company level behaviour leads towards a higher level of complexity and helps to explain
how Schumpeterian creative destruction shapes economies and competitive environments. (See Dosi 1984;
Nelson and Winter 1982) This leads to learning economies and national systems of innovation approaches that
seek to explain innovations not only as breakthroughs but indeed as incremental everyday changes in company
routines, learning, and various levels of interactions (e.g., user-producer). (See Freeman 1982 and 1987;
Lundvall 1992) Thus, the evolutionary economics dealing with private sector innovations moves during 1980s
almost exactly in the opposite direction as the emerging public sector literature.

5
researchers a real innovation and yet some would argue that this innovation did not really
bring any improvement or at least that the jury is still out. However, in most cases the line
between innovation or not, improvement or not, is not only tenuous at best, often it seems
plain arbitrary. Moore and Hartley 2008, for instance, use as examples contracting out and
private public partnerships, in other words, public sector innovation is another term for NPM-
style reform practices.
7
Thus, what is and what is not an innovation, seems rather arbitrary or
subjective and this is further complicated by the fact that most attempts at measuring public
sector innovations use surveys (as we will show below) – in essence further cementing
subjectivity into the discussion.
In Table 1 we samples this most recent period in defining public sector innovations.
Table 1. Public sector innovation definitions from past two decades
Source
Definition
Scope
Lynn (1997)
“Innovation [in government] is properly
defined as an original, disruptive and
fundamental transformation of an
organization’s core tasks. Innovation
changes deep structures and changes them
permanently.”
The definition echoes Wilson’s 1989 use
of the concept and is aimed at
differentiating any change in the public
sector from deeper transformations that
can be called innovations.
Moore et al.
(1997)
”Changes worth recognizing as innovation
should be…new to the organisation, be large
enough and durable enough to appreciably
affect the operations or character of the
organization.”
General definition involving novelty and
change in relation to the organisation and
its overall operations. This means
widespread improvements in both
governance and service performance to
increase public value (Moore, 1995).
Newman et al.
(2001)
Public sector innovation can be constituted
as a “discontinuous or step change, as some-
thing which was completely new to a parti-
cular local authority (though which may
have previously been applied elsewhere),
and a change which had already been imple-
mented rather than just an aspiration or
planned initiative.”
Change-based definition that accentuates
also incremental innovation and puts the
focus on implementation.
Green et al.
(2001)
“/…/ doing something new i.e. introducing a
new practice or process, creating a new
product (good or service), or adopting a new
pattern of intra- or inter-organisational
relationships (including the delivery of goods
and services).”
Emphasizes that simple organisational
change does not equate innovation.
Definition originally concentrates on
service innovation.
Mulgan and
Albury (2003)
“New ideas that work /…/ successful
innovation is the creation and
implementation of new processes, products,
services and methods of delivery which
result in significant improvements in
outcomes efficiency, effectiveness or quality”
Emphasis put on implementation and
successful innovations that have a
significant impact in the public sector
(implying radical change).
7
See Drechsler 2005 on the role academic and policy talk fashion plays in such relabeling practices.

6
Hartley (2005)
The definition starts from the fact that
innovation is not always “a physical artefact
at all, but a change in the relationships
between service providers and their users./..
./ Consequently the public sector innovations
“consider innovations, particular radical or
complex ones, to be multidimensional,
specifying the dimensions (and the size of the
innovation in those dimensions) in the
interests of systematic comparison.”
Takes note from the definition of Moore
(see 1995; 1997) and extends the public
value oriented approach to include
different forms of innovation: product,
service, process, position, strategic,
governance and rhetorical innovations.
Also diffusion and dissemination –
spreading good practice and adop-
ting/adapting existing innovations – is
included as a significant part of public
innovation.
Osborne and
Brown (2005;
2013)
“The introduction of newness into a system
usually, but not always, in a relative terms
and by the application (and occasionally
invention) of a new idea. This produces a
process of transformation that brings about
a discontinuity in terms of the subject itself
(such as a product or service) and/or its
environment (such as an organisation,
market or a community).”
Emphasizes ‘newness’ and discontinuity
of change in the public sector.
Albury (2005)
Public sector innovation is “the creation and
implementation of new processes, products,
services and methods of delivery which
result in significant improvements in
outcomes efficiency, effectiveness or
quality.”
Emphasizes implementation, significant
improvement and creativity.
Koch and
Hauknes
(2005)
“Innovation is a social entity’s
implementation and performance of a new
specific form or repertoire of social action
that is implemented deliberately by the entity
in the context of the objectives and
functionalities of the entity’s activities.”
Functional distinction of public sector
innovation that is shaped within the
context and environment of the agent is
activity/agent specific. Deliberate action
is emphasized.
Halvorsen et
al.(2005)
Public sector innovation as “change in
behaviour”.
Very broad definition concentrated on
change.
Mulgan (2007)
“The simplest definition is that public sector
innovation is about new ideas that work at
creating public value. The ideas have to be
at least in part new (rather than
improvements); they have to be taken up
(rather than just being good ideas); and they
have to be useful.”
The definition makes an additional
requirement implementation – of being
‘taken up’ –, meaning that also in terms
of measurement some time lapse before
change and impact is required.
European
Commission
2013; EU
Expert Group
on Public
Sector
Innovation
“We therefore structure this inventory along
two lines: initiatives that would be readily
considered innovations inside the public
administrations, such as the shift to ICT
tools and HR management (innovation IN);
and initiatives that foster innovation
elsewhere in society, such as the public
procurement of innovation, the unitary
This definition tries to separate different
modalities in public sector innovations
rather define in detail what innovations
are.

7
2013
patent or support to social entrepreneurship
(innovation THROUGH).”
Source: compiled by the authors.
However, in contrast to earlier periods of public sector innovation concepts, and with the
exception of Lynn (1997; see also Lynn 2013), the current period of scholarship pays much
less attention to evolutionary character of changes described as innovations. This is not to say
that there is not an acute awareness that one has to differentiate ordinary change from
innovation. For instance, Osborne and Brown 2013 argue, “the management of innovation is
an entirely different task from the management of developmental change” (2013, 3); Lynn
similarly concurs that all non-transformative change is “’innovation lite’, which is
indistinguishable from ordinary change.” (2013, 32) Yet, how this transformative change in
fact works in the public sector – and differs from typical private sector dynamics – remains
almost always unpacked. Even the most advanced concepts of public sector innovation do not
address in detail how selection mechanisms and other processes take place that would enable
us to distinguish innovations from ordinary changes. What makes one reform or new service
an innovation, and the other not? Often there seem to be normative connotations involved in
distinguishing innovation from change: as innovation is good, a successful reform must be
innovative.
On the other hand, evolutionary dynamics dominate private sector innovation literature,
evident in such concepts as backward and forward linkages, increasing returns to scale, first
mover advantage, winner-takes-all markets, imperfect competition, externalities, etc, (many
of these concept are present already in Schumpeter, especially in 1939 Business Cycles). In
fact, innovation research in the private sector is all about evolutionary change: how and why
certain products, services, technologies, technology systems, but also organizational forms
and institutional frameworks become dominant over others that in turn become obsolete or
vanish altogether. (Nelson and Winter 1982, Perez 2002) The role of technology, particularly
large-scale shifts following technological revolutions that lead to whole new paradigms, is
difficult to underestimate here.
However, such evolutionary practices and processes are simply much less evident or even
lacking in the public sector. Moreover, many of these processes would be also not desirable
in the context of public organizations, such as monopoly rents garnered by first movers, or
undercutting the same first movers by imitation. There is hardly any competition within the
public sector for such evolutionary processes to take place. The way innovations diffuse in
the market environment, via imperfect competition and imitation, is hardly a way for public
sector innovations to emerge and to diffuse. Furthermore, in business innovations, there are
lot of failures at innovations and lot of losses through innovations or imitations by
competitors. Again, these phenomena seem to be not present in the public sector or present
themselves in a different form.
That is not to say there is no evolutionary change in the public sector. As we have seen
above, almost all literature on public sector innovation assumes there is evolutionary change,
but conceptualizing the evolutionary changes in public sector seems to have been lost in
private sector terminology. The key lesson from previous literature, accordingly, seems to be
that we should not attempt to look for similar processes to take place within public sector;
rather we should try to focus on evolutionary processes within public sector that originate

8
from logics of public sector and pertain to such phenomena as power, legitimacy, trust, etc.
This is arguable exactly the topic of perhaps the earliest ‘discussion’ on public sector
innovation, namely between Tocqueville and Weber on the state level public administrations
in the US.
8
Tocqueville’s analysis, and admiration, of state level administration is famous,
Weber’s counterarguments are much more scattered and less well-known (Tocqueville’s was
published in 1835 and 1840;
9
Weber remarks can be found in Wirtschaft und Gesellschaft
from 1922 and elsewhere).
10
Tocqueville’s main question in looking at US state and especially township level
administration was how can diverse townships in New England, without central
administration, still provide relatively uniform public services, especially under an
administrative system where most public functions are fulfilled by elected officials. (1876,
92) He explained this with judicial oversight of administrations, and called both –
decentralized administration and judicial oversight – innovations (ibid.)
11
In Tocqueville’s
view, decentralized administration with elected officials and judicial oversight work better
than centralized administrations (which, he argues, was an innovation of the French
revolution; 121): centralized administrations have more resources, are good at regulating
business, maintaining social order and security but also keep society equally from
improvement and decline (113); centralized administrations are good at mastering resources
to combat problems but they are poor at rejuvenating what might be called socio-political
resources for change (109).
When we jump two thirds of a century further, we can see that all the ills of centralized
administration described by Tocqueville become positives in Weber’s view:
12
in order to
keep social order, that is to retain authority and society functioning, centralized bureaucracy
is the “technically” better instrument over elected officials. (2009, 156; further also 545-550
and 561;) Elected officials and other “’schöpferische’ Betätigung der Beamten” leads rather
to unpredictability and to bureaucracy that seeks to retain its own power, in other words to
rent seeking behaviour.
13
(565)
While Tocqueville and Weber had different normative goals – former describing the benefits
of active civic life, the latter describing benefits of well-functioning and predictable state
apparatus (also Freund 1974) – both discuss eventually how authority, to use Weber’s term, is
maintained in society with competitive interests via institutional and administrative
8
For a longer historical discussion of the concept of innovation, see Godin 2008&2012.
9
We refer here to 1848 French edition, available via Project Gutenberg and to 1876 English translation.
10
In Weber’s case, we use 2009 German edition. For a comparative discussion of Tocqueville’s and Weber’s
discussions of America, see Kalberg 1997.
11
“C'est ce qui ne se découvre pas au premier coup d'œil. Les gouvernants regardent comme une première
concession de rendre les fonctions électives, et comme une seconde concession de soumettre le magistrat élu
aux arrêts des juges. Ils redoutent également ces deux innovations” / “The communities therefore in which the
secondary functionaries of the government are elected are perforce obliged to make great use of judicial
penalties as a means of administration. This is not evident at first sight; for those in power are apt to look upon
the institution of elective functionaries as one concession, and the subjection of the elected magistrate to the
judges of the land as another. They are equally averse to both these innovations”.
12
Weber’s Wirtschaft und Gesellschaft
for 3 months in 1904, see extensive discussion in Kaesler 2014, 563-637.
13
Weber summarizes his discussion of bureaucractic creativity as follows: “Entscheidend ist für uns nur: dass
prinzipiell hinter jeder Tat echt bürokratischer Verwaltung ein system rational diskutabler ‘Gründe’, d.h.
entweder: Subsumtion unter Normen, oder: Abwägung von Zwecken und Mitteln steht.” (565)

9
innovations (although Weber does not use the term). We can paraphrase Weber: modern state
is defined by its authority to use violence to uphold the very same authority. Above all,
Tocqueville and Weber show how such innovations lead to differing socio-political
relationships and networks, institutional and organizational structures and cultures, in other
words: how these innovations drive different evolutionary change. But both also show why
evolutionary processes in public sector are punctured by political, legal, institutional and
administrative constraints. (See also Karo and Kattel 2013) In fact, these very constraints are
part of these evolutionary processes, forming simultaneously internal factors that are changed
and external factors limiting changes. Constraints are intrinsic to public sector. Thus, to use
Tocqueville’s example, judicial oversight in small townships acted as a constraint on elected
officials, yet this same constraint led to better services for the citizens. Weber, on the other
hand, writing two thirds of a century later, argued that modern societies have become
increasingly more complex and thus require centralized administrations that can act
simultaneously as constraints and enablers.
Consequently, following Tocqueville and Weber, we can argue that instead of competition as
driver and diffuser of evolutionary processes as is the case in private sector, intrinsic public
sector features act simultaneously as constraints and enablers and engender punctured
evolutionary processes as consequence of public sector innovations. Notice that in both cases
the innovations influence organizational level capacities, institutional interactions and,
eventually, political authority of a state. Their recommendation, as it were, would be to look
at changes in public sector that lead to 1) changes in constraints and enablers, that relate
directly to how authority is obtained/retained and 2) engender clearly discernible
evolutionary trajectories in their respective ecosystem – such changes could be termed public
sector innovations. Essentially, public sector innovations are such changes in public sector,
according to our interpretation of Tocqueville-Weber debate, that realign enablers and
constraints and one way or another influence authority and legitimacy of the given public
sector actor. This dimension of authority and legitimacy is almost completely missing in all
historical and contemporary debates.
It can be argued that a recently emerging literature on social innovation (see Bekkers et al
2013 for an overview) tries fill the gap in public sector innovation literature by looking at
values and social relevance and thus moves the discussions towards issues of authority, trust,
etc. The concept of social innovation can be construed wholly as value-based: broadly put as
democratic commitment to social change (Andersen and Bilfeldt, 2013). This can be
described as change towards meeting social needs. However, the concept is also widely
applied to the private sector: the effect of corporate social responsibility, business ethics,
social businesses and not to mention the development of the civil society (Osburg, 2013;
Schöning, 2013). Consequently, it is sometimes used as an opposite to what we perceive as
the traditional public sector – the inclusion of market-reliant or philanthropic solutions that
are found to be more inclusive and ‘better’ (Moulaert et al., 2013). First and foremost, the
engagement of citizens is emphasized in social innovation. Nevertheless, when the positive
change in responding to social needs is taken as primary, social innovation can be both
bottom-up and top-down, not to mention output or process related, legislative or cultural. The
important distinction is that social innovation reveals and responds (better) to social needs by
creating new services or expanding equality of access to them (Martinelli, 2013). This can be
done in unison with empowering users or specific social groups and thus, modifying social –
and also power-relations between service users and providers. Hence, it is hard to distinguish
and demarcate the extent of public sector innovation through the concept of social innovation,
however, this can be accompanied into the public value based understanding of

10
transformative change within the public sector. In other words, while conceptually social
innovations are situated between private and public sector innovations – their origins can be
in both sectors –, it is their perceived impact of empowering, of changing power and
authority relations that makes this strand of literature interesting also conceptually.
Furthermore, social innovation as a concept can legitimize more socially oriented and
solidarity based political economy which also creates momentum behind public sector
services (see Fraisse 2013).
Summarizing 150 years of discussion sketched above on conceptualizing public sector
innovations and innovations generally, we can, first, conceptualize the relationships within
the whole innovation arena in one figure as a taxonomy (Figure 1) and, second, draw
following conclusions:
A From the oldest literature discussing public sector innovations (Tocqueville, Weber):
1) Public sector innovations are in the most abstract sense related to public authority;
2) Innovations lead to evolutionary changes in constraints and enablers that are intrinsic to
public sector (rules, relationships, institutions);
B From recent public sector innovation literature:
3) These evolutionary processes use different modalities (innovations within and through public
sector), agency (public sector proactively initiates changes or reacts to technological,
environmental, etc, changes), and morphology (from incremental to discontinuous changes);
4) Literature on public sector innovations rarely deals with authority (and related phenomena
such as legitimacy, trust, etc,) but rather with relatively specific features of these changes, e.g.
with specific modalities (within public sector organizations), agency (reactions to external
stimuli such as technology, politics, social challenges) and morphology (incremental
changes); most of these changes are in fact not evolutionary or their impact remains difficult
to discern;
5) Innovation is too often defined from a normative viewpoint (as something leading to
significant improvement in public service delivery) rather than a process that explains how
profound changes take place in public sector.
6) In defining innovation, the literature has focused mostly on organizational or policy levels,
but in doing so it has neglected the wider, public-sector-level, constraints and enablers. What
is argued here is that there is a need for a systemic perspective that goes beyond single
instruments or decisions and that would offer a framework against which the changes in core
routines on organizational or policy levels can measured against.
7) Accordingly, disproportionally large areas of public sector activity in relations to innovations
are under-researched and, we will argue below, this leads also to relatively simplistic attempts
to measure public sector innovations.
Figure 1. Taxonomy of innovations, 150 years of discussion.

11
Figure 1 is based on the assumption, derived from our literature review, that innovations in a
society take place in two different contexts – technology and authority, to put it very simply –
that lead to different and often parallel evolutionary changes in the society. While in
capitalist economies, technology influences private sector innovations more directly than
public sector ones, then authority (and all that follows from this) influences more
dramatically public sector innovations than private sector ones. Social innovations can be
described in this context as a case of innovations where this parallelism of changing
technology and authority comes most clearly to the fore, at least in our current conceptual
state of the art.
Section 2. Indicators on private sector innovation
The first well known effort to measure private sector innovation took place in the United
States: the National Research Council started to collect statistics that we commonly relate
with innovation today (e.g., on the R&D activities and its impact on changes in volumes of
sales) in 1933 (see Holland and Spraragen 1933). Such measurement activities became more
common in the 1960s when several other organizations – e.g., the Federation of British
Industries – carried our related industrial R&D surveys (for an overview, see Godin 2002).
The linear approach – overemphasizing R&D based creative activities to develop technical
inventions – dominated until the early 1980s when the re-discovery of the Schumpeterian
theories took place (Arundel & Hollanders 2008). As of today there are basically two strands
of literature that deal with indicators and the measurement of innovation in private sector:
first, competitiveness indices and indicators; second innovation and technological dynamics
indices and indicators.
The first set, competitiveness measurements do not have a clear-cut theoretical basis, come
from management research, and are methodologically mostly survey based. The examples of
such approaches include various applications of the Porter’s approach to cluster and
competitiveness (Porter 1990, 1998) to measuring competitiveness, but also the closely
related work of the World Economic Forum (e.g., Porter et al. 2008; Schwab et al. 1999) and

12
the Institute for Management Development (IMD 2012). At the micro (company) level,
competitiveness is generally understood to refer to the ability of a firm to increase in size,
market share and profitability, and such studies have increasingly paid more attention as non-
price factors as being important contributors to competitiveness, including human resource
endowments, technical factors as well as managerial and organizational factors (Clark and
Guy 1998).
The second approach – science, technology and innovation measurements – has as clearer
basis in the Schumpeterian theory and is usually based on empirical measurements and
increasingly also on surveys. The following are indicated as the main areas of indicators: (1)
R&D data, (2) data on patent applications, grants and citations; and (3) bibliometric data, i.e.
data on scientific publication and citation (Smith 2005). While statistical analysis along those
indicators is common in innovation research, the specific databases and surveys have
generated more insights into the matter, e.g., SPRU database on major technical innovation in
the British industry (see Pavitt 1984 for the results) or DISKO surveys on technological
collaboration of the University of Aalborg. Most notably OECD, by synthesizing various
innovation measurements, has developed the so-called Oslo manual (OECD 1992, OECD
2005) that largely guides the measuring of private sector innovation activities today. The
most common application is the Community Innovation Survey (CIS), carried out with
biannual frequency by EU member states and number of other countries, and has given basis
for extensive econometric and statistical analysis of innovation (for an overview of such
studies, see Arundel and Smith 2013).
What is common to both approaches is that impact of the innovation on a firm performance is
expected. The firm specific indicators – the additional turnover generated, impacts on costs
and employment, and on productivity – are generally applied (see OECD 2005), although
several limitations are already proposed in the initial OECD manual (2005, pp. 109-112),
including the multiplicity of factors that influence innovation impacts (can be exogenous to
the company), time lag between an innovation and its impact and difficulty “to ask for
quantitative measures of the effects of innovations in surveys, even for very rough estimates,
as the calculations often require substantial analysis on the part of the enterprise” (p. 109).
The use of such overall proxies to grasp the true nature of the innovation impacts is revealed,
for example, on research on the relationship between innovation and employment as it has
been concluded that “On the whole, economists cannot propose a clear-cut diagnosis about
the employment impact of innovation, either theoretically or empirically” (Vivarelli 2007, p.
729). The locus of such research has moved from the question “does technology create or
destroy jobs?” to questions like “what type of jobs are created or destroyed by innovation?”
and “how does the composition of skills and wage structure change” on firm, industry and
macro levels (Pianta 2004).
Most of the work on the effects of innovation concern productivity – defined as the ratio of
output over input – and more specifically on labor productivity rather than total factor
productivity, due to lack of data on capital and other inputs (Mairesse and Mohnen 2010).
The research is largely based on the framework from Griliches (1979, 1995) linking
investment in knowledge to productivity growth, its elaboration explained in Figure 2 (by
Hall and Mairesse 2006), where the square boxes denote generally measurable quantities,
while key elements – knowledge capital and innovation output – are marked as oval boxes for
which only coarse proxies are used. This has led to the conclusion that although measuring
the effects of innovative activities on firms’ productivity has been an active area for research
for several decades, the literature still does not provide a unique answer in terms of the

13
magnitude of this impact due to variability and uncertainty that is inherent in innovation (Hall
et al. 2009; Mohnen and Hall 2013).
Figure 2. Innovation and productivity
Source: Hall and Mairesse 2006, p. 292.
While the measurement and the improvement of productivity are better established in the
manufacturing sector, measuring productivity of a service is not yet as well-developed or
well-established (Grönroos and Ojasalo 2004), although increasing rapidly in the case of so-
called information intensive (Table 2) industries. Still, services are generally related to
intangibility, simultaneous consumption and production and less R&D content, leaving
several aspects for which well-developed metrics have been developed (Figure 2) uncovered.
In productivity measurements wider view on productivity is called for, i.e. not only
measuring efficiency, but deeper inclusion of the concept of effectiveness – is needed
(Grönroos and Ojasalo 2004; Johnston and Jones 2004; Gallouj and Savona 2010, Gallouj
and Djellal 2010), and remains challenging for measuring public sector services as well.
Both approaches share more general common problems as well. First, a key challenge is
related to what are the most appropriate indicators to include, both generally as well as
specifically. Research on private sector innovation has made is quite clear that companies and
the environments they operate in – the sectors, the national context, etc. – are different,
posing difficulties to measure innovation across companies. Table 2 presents a commonly
accepted view indicating that the main sources of technology and innovation are different
across companies (OECD’s recent innovation strategy (2010) also proposes categories such
as ‘cost-driven’, ‘demand-driven, ‘user-driven’ and ‘employee-driven’ innovations): for
companies in some sectors it is the R&D that matters mostly, while for the others innovation
becomes available once new products (e.g. new seeds) or new production processes (e.g. new
machinery) are available. The nature of innovation can be also different varying from radical
product innovation and process innovations (which are mostly researched) to (less

14
researched) incremental marketing and organizational innovation. This all also leads to
difficulties in measuring innovation in various companies in a consistent and statistically
comparable way.
Table 2. Types of technology firm
Sources: Adopoted from Dodgson et al. 2008, p. 40, based on Pavitt 1984.
To measure sectors and countries in a consistent and statistically comparable way poses even
more complications, related to aggregation problem – firm level management tools and
indicators are difficult to aggregate on the national level as sectors and their dynamics differ
widely. Research on national systems of innovation that emerged in late 1980s (see Freeman
1987, Lundvall 1985; Nelson 1993) gave impetus to advance our understanding on those
national contexts, promoting the idea that economy as ‘an ensemble of connected elements
not an aggregate entity’ (Metcalfe 2002), and ample research has become available since
then. Still, it has remained rather weak theory-wise, posing also measurement problems.
CIS, the major innovation survey, specifically, has been criticized on the basis of definitional
restrictions with respect to innovation inputs and outputs and on limitation of applying the
survey – that has been developed on the basis of manufacturing companies – to the service
sector, as well as weak inclusion of human capital development aspect (Smith 2005). In order
to integrate emerging topics in more recent innovation research, the Oslo Manual has
undergone revisions. First, by extending innovation surveys to the service sector and then, by
acknowledging the importance of non-technological innovation, by adding organizational
(the introduction of new systems and management methods and new types of work
organization and business models) and marketing innovation (the introduction of new
commercial methods, and they include changes in product design, promotional strategies,
etc.). Still, both strands of research face difficulties on measuring and relating to innovation
intangible assets, such as skills, that have a key place in evolutionary work. OECD (2010),
e.g., emphasizes that more attention should be put on measures of education,
entrepreneurship, economic, environmental and social outcomes, and the framework
conditions that support or inhibit innovation.
To measure evolutionary dynamics on sectoral level and even more so on country level are
complicated tasks. In recent years we see how productivity has become increasingly used as
simple one data point proxy for evolutionary dynamics. The problems with this are obvious:
while productivity changes indicate change within a company, sector or country, it does not
actually indicate the sources of the changes (massive lay-offs due to a recession can also lead
to productivity increases without any innovations or skill upgrading). However, it is

15
politically highly poignant and easy to use such figure as it is seemingly understandable also
to non-specialists. As we show below, this logic is also increasingly important in the case of
public sector performance measurement efforts.
In sum, innovation measurement in private sector has developed towards more
multidisciplinary and interdisciplinary approaches, while the impact on productivity is the
key aspect studied. On the policy making level, we see a tendency to emphasize single high
importance figures such as productivity, R&D expenditure in GDP, or rankings in various
competitiveness indices.
Section 3. Measuring change and productivity in public sector
As we argued above, productivity has become one of the main proxy-concepts used in
measuring changes in private sector innovation performance and there exists a rather strong
consensus on what private sector productivity measurements are on company level, less so on
how useful it is on country level. This is, however, more complicated in the case of public
sector organizations and public sector in general.
In the last two decades public sector has increasingly imported values from the private sector.
It is expected that the first focus more on customers, outputs rather than inputs and more
effective and efficient performance (Hoque, 2008). From the early 1990s this process has
gathered steam with the introduction of various performance instruments (incl. many perfor-
mance indicators) to the public sector (see the seminal work by Osborne and Gaebler, 1992).
The goals of this process are usually tied to ideas of advancing transparency, learning, appri-
sing, sanctioning and also showing accountability in the public sector. Simply put, “what gets
measured, gets done” (Osborne and Gaebler, 1992). While decentralization and respon-
siveness were deemed central to this reform trajectory, the idea that accountability equals
performance does not inherently concur with the former ideas (Kelly, 2005). Consequently,
many problems of use of indicators as performance management tools have been brought out
for decades and while the critiques are cyclically repeated and sometimes advances (e.g.
Ridgeway 1956; Smith, 1995; van Thiel and Leeuw, 2002; Miller, 2003; Pidd, 2008), very
few solutions have been brought out to tackle these problems. In line with this, the quality of
data from various sources is a perennial problem for measurement systems and further,
various rationalities can result in even different interpretations of the same data (Townley,
2008).
Many different things can be measured in terms of services provided by the government
ranging from inputs/resources, throughputs/processes, outputs, and outcomes/impacts (see
e.g. Packard, 2010; Kuhlmann, 2010; Sillanpää, 2013). This has also lead to the proliferation
of performance indicators, which does not necessarily mean that the quality of indicators
themselves has improved (Lonti and Gregory, 2007). This can be a source of government
overload (Lewis and Triantafillou, 2012). With the growth of indicators, a high degree of
confusion regarding priorities has also been noted (Micheli and Neely, 2010). Thus, a large
number of indicators in public organisations nowadays may be irrelevant, but very hard to
remove (Fryer et al., 2009). Even when trying to avoid it, usually due to credibility and
comparability more technocratic performance measurement is preferred (for public
administrators usually department workloads are more important than long-term outcomes or
positive changes (Ammons, 2004)). The simpler the representation of results, the easier it is

16
to recall them later (see Hibbard et al., 2002). However, made simple, the measurement
system can severely misrepresent result and thus, become useless.
14
Effectiveness in the public sector is seen in theory as value creation to the citizen, which has
no real maximum and thus, is very hard to quantify (Tangen, 2005). Various measurement
systems have been suggested to the public sector from balanced scorecards to quality
management models to deal with measurement (Hasan and Kerr, 2003; Sahay, 2005), but
they have been also severely critiqued for simplistic, output-centred approaches or
unintended effects. It is very easy to develop ‘tunnel vision,’ myopia, misrepresentation and
misinterpretation, gaming and ossification etc (as outlined by Smith, 1995). While the general
goal of indicators is to enable comparability (to select the good out of the bad), they can also
add to the impetus to ignore local circumstances and tacit knowledge (already brought out by
Carnegie and Wolnizer, 1996).
Consequently, the main problems of measurement in the public sector, not only technical, but
also conceptual, can be summarized as follows:
1. Diverse nature of public sector services, the wide range of users and the difficulties in defining
targets (Arnaboldi and Azzone, 2010). Targets do not adhere to singular profit imperatives in the
public sector (Van Thiel and Leeuw, 2002). When multidimensional impacts – tangible and
intangible, financial and value based, individual and system level – are concerned, prospects on
how to measure these effects are largely missing in academic literature.
2. Many economic impact evaluation methods are almost impossible to use in the public sector
simply because they require that effects (also intangible, e.g. improved health, quality of life etc)
should be monetized. As such, measurement in the public sector is usually limited to the ‘product’
rather than a process, ‘throughput’. Consequently, measures are faced with the problem of
‘product’ definition: for example, how to measure research (by scientific publication?), successful
treatment (reoccurrence?) or even deterred crime. For this, tolerance of multiple definitions has
been suggested as a possible solution that could capture multiple values (de Bruijn, 2002).
3. At the same time, measurement in general is static, while the processes are clearly dynamic. There
are time lags connected to the effects of many policies and also public sector performance. When
a long-term view is taken into consideration, present actions can be questioned (Brax, 2007). Wel-
fare services, with traditionally very high number of performance indicators, are found to lack
measures to demonstrate the various long-term effects (Sillanpää, 2013). These are problematic to
measure, not only because of the unknown, but also most strategic planning cycles are maximally
4-5 years long. However, in many cases, success can only be shown through long-term
effectiveness that is usually only possible to describe in qualitative effects (e.g. Porter, 2010).
15
4. As such, application of quantitative performance measurement usually rests on proper
measurement scales decreasing ambiguity and uncertainty (see Pidd, 2008). However, there are in
this regard clearly situations where quantitative indicators are not the best measures. This occurs
14
This is the usual problem with single point league table ranking systems and indices (e.g., comparing schools
by average exam results (Goldstein and Spiegelhalter, 1996)) that is also present in the private sector (see
previous section). When reasonable estimates of statistical variation would be introduced to these comparisons,
most differences would disappear (Pidd, 2008). This means that the extremes – top and bottom results – can
considerably vary, but for most others there can be very little difference.
15
Moreover, in innovation related long-term goals the static outtakes of the current situation may not be that
informative and they may even stand in the way of innovation. One of the major lines of critique most pertinent
to innovation indicators is the fact that measurement is found to simply block innovation or reproduce the
existent (Behn and Kant, 1999).

17
especially when practices in the public sector are in transition, complex issues are faced and when
standards in place are contested (see Noordegraaf and Abma, 2003). This usually leads to a value-
debate for which quantitative numerical indicators are not useful (they simply represent one
limited dimension of value) and expert opinion based on professional agreement is more
advanced for evaluation.
5. Furthermore, in situations that have been described as ‘wicked problems’ in the public sector
(Rittel and Webber, 1973) – i.e. innovative and sometimes conflicting processes that are not
routine – conventional control systems (including static measurement) are found to make no sense
as a certain level of ambiguity and uncertainty is necessary in the processes (this should not be
confused with statistical uncertainty that comes from the measurement system (data derived from
indicators being neither reliable nor valid)).
16
The quest for measuring public sector productivity only amplifies the conceptual and
technical problems related to general public sector performance (change) indicators. Public
sector productivity has many different meanings and its significance has changed constantly
over time (Andrews and Entwistle 2013, Rutgers and van der Meer 2010, Schachter 2004). It
is today mostly regarded as a technical term, which refers to the ratio of outputs to inputs in
producing public services (Pollitt and Bouckaert 2011, Andrews and Entwistle 2013, Rutgers
and van der Meer 2010, Dunleavy and Carrera 2013). The definition of productivity is often
used as a synonym to public sector technical efficiency and in fact the terms of public sector
productivity and efficiency tend to be used interchangeably in the literature (see various
definitions outlined in Rutgers and van der Meer 2010, but cf. Dunleavy and Carrera 2013).
At the same time, a clear distinction is usually made between productivity and related
concepts. For example, while productivity (efficiency) refers to the “amount of resources
used to produce a unit of output”, effectiveness is mostly understood as “degree to which an
organization realizes its goals” (Etzioni 1964 in Lane 1993, 191). This, in turn, implies the
need to distinguish productivity from other related concepts (in addition to effectiveness, e.g.,
economy, cost-effectiveness, value-for-money) when dealing with public sector performance
(see also Rutgers and van der Meer 2010, Dunleavy and Carrera 2013).
This technical definition of public sector productivity has become firmly rooted in today’s
public administration and management rhetoric (see only Pollitt and Bouckaert 2011). In
2000s the discussion has moved collectively onward to include also public sector productivity
and efficiency frontiers (benchmarking similar organizations in their input/outcome ratio).
Challenges that both the private and public sector face – diminishing manpower in an ageing
16
Eradicating the former from the process by strict, narrowly defined indicators and control system may make
also action impossible. Reducing uncertainty and ambiguity is, however, the usual reaction when dealing with
public sector control systems. The more autonomy is assigned with more complicated tasks – which innovation
related activities surely are – more accountability is of course expected. This is used to evaluate actions of single
organisations, but also ‘benchmark’ the former against each other. Thus, increase in reporting can come at an
expense in taking risks and being innovative (de Bruijn, 2002). Consequently, the whole process can be
subjected to ‘pigeon-holing,’ due to the measurement system thinking and acting in pre-established categories.

18
society and less available capital – are confronted with the solution from the private sector:
increasing productivity growth (e.g. Waller, 2006). As is the case with private sector – mostly
assumed to be more productive than public sector (Roessner 1977) – productivity
measurements holds to allure to express complex dynamic processes in one number that is
equally understandable to experts and lay persons.
At the same time, this technical approach to public sector productivity has been repeatedly
challenged (see Rutgers and van der Meer 2010 for an overview) and in spite of the simple
and eloquent definition of public sector productivity, the concept has proven to be all but
unambiguous and uncontested (see Table 3 for an overview of different concepts). The
central normative debate has been about the proper place of productivity among other public
sector values and whether the technical approach is at all useful for public sector. For
example, Gulick (in Rutgers and van der Meer 2010) has claimed that administrative
productivity should be seen as a fundamental value in itself, a value which may conflict with
other public values. Others (e.g., Waldo) have rejected this idea and insist that productivity
cannot be in conflict with other public purposes; productivity has a meaning only if measured
in terms of other values; it is part of wider value system and can have only subordinate value
for some other (higher) public purposes (ibid.).
For the strongest opponents the general pursuit of productive efficiency should be rejected
altogether as public managers need to foremost realize multiple democratic values rather than
base their decisions on productive rationality (Andrews and Entwistle 2013, Dunleavy and
Carrera 2013). According to this view public administration should be guided by wider public
purposes and related democratic values (such as equity, probity, accountability) that are
necessarily contested and cannot be reduced to the ratio of inputs and outputs. Moore’s
(1994) public value concept, which aims at a shift away from productivity to broader public
value creation, is an example of a more recent approach reflecting the general discontent with
regard to the market-loaded productivity thesis. According to the public value framework it is
the citizens who should decide and express through the democratic process what kind of
values are to be created by public sector and how (O’Flynn 2007). And productivity may not
even be among the parameters according to which the processes and outcomes of delivery of
public values is evaluated against. In short, it is the responsiveness to citizens that matters the
most rather than how to structure for efficiency that the critics insist.
17
With the rise of the debate around public values, the focus in measurement has moved even
more towards outcomes and processes – what value, by whom and how is added to the public
sphere (see Benington and Moore, 2011). This cannot be singularly captured in individual
satisfaction surveys, as many of the effects may be not directly visible to individual service
users. For example, Moore, who has popularized the concept in mid-1990s, has used the
balanced scorecard model for the private sector (Kaplan, Norton) for PV - 'public value
scorecard' (2003). For broader evaluation also the ‘competing values framework’ has been
proposed (Talbot, 2008). However, most studies on public value concern themselves with
measuring public sector employees’ individual values and motivations, mostly in the form of
surveys (e.g., O’Toole et al. 2005; Lyons et al., 2006; Stackman et al., 2006; Buelens and
Van den Broeck, 2007; Meynhardt and Metelmann 2009). The new public value framework
brings out competing priorities in the public sphere based on ‘values’ that also include trade-
offs (Benington and Moore, 2011). In measurement systems in the public sector rarely
answer questions how trade-offs between different values are mitigated and emerging
17
Schachter (2004) provides an interesting historical overview on early debates on public sector productivity
and where this dilemma is of special interest.

19
problems are dealt with (a good overview of current literature (also on measurement, or lack
there of) is Williams and Shearer 2011).
In a rather similar vein, Rutgers and van der Meer (2010) argue that technical concept of
public sector efficiency is a too narrow one to embrace the true meaning of the term, and it is
the substantial efficiency that should be taken as the core value of public administration. For
them substantial efficiency refers to the Aristotelian meaning of efficiency as “the ability to
get things done” (ibid., 772), or to put in other words, “capacity to produce an effect” (ibid.,
772-773). The substantial meaning of efficiency is not related to outputs (which is part of the
technical realm of the concept), but rather on force and ability; that is, “efficiency is the
potential to pursue desired ends” (ibid., 773).
In order to overcome the normative contradictions, Andrews and Entwistle (2013) argue that
instead of abandoning the efficiency concept altogether, one should attempt to incorporate the
fundamental democratic values into the concept of efficiency. They stem from the position
that “public service efficiency is essentially a product of the proper functioning of democratic
institutions and the responsiveness of the state bureaucracy” (2013, 258-259). Building upon
mainstream economic theory they delineate four dimensions of public service efficiency:
productive efficiency, allocative efficiency, distributive efficiency and dynamic efficiency.
Productive efficiency (or productivity) corresponds to the technical approach to public sector
productivity; that is the ratio of outputs to inputs. Both strengthening the bureaucracy (via
centralization and consolidation) as well as breaking up bureaucracies (via agencification and
contracting) may be outlined as central strategies for enhancing public sector productivity.
Allocative efficiency refers to the match between the demand for services and their supply.
This is a complex dimension of efficiency since the citizens’ demand is always mediated (e.g.
through politicians). Main policy options to enhance the allocative efficieny may include
electoral reform, fostering more participative forms of democracy, devolution and
decentralization, and injecting choice and contestability to public service delivery.
Distributive efficiency “refers to the distribution of resources or services between citizens and
the relative cost to government (and society) of that distribution” (255). Policy choices here
include direct transfers via taxes and welfare payments as well as provision of merit goods
(e.g. education). Dynamic efficiency refers to the allocation of resources between current and
future consumption (256). In other words, it refers to efficiency over time. It is a question of
maintaining a balance between current and capital investments and borrowing and debt
repayment. It relates the issue of public sector efficiency to the rate of consumption of non-
renewable resources as well as finding a proper balance between investments in social capital
and ex post service delivery (e.g. health promotion vs providing acute care). Dynamic
efficiency is allegedly directly dependent on institutional design as it needs slack resources
for R&D, innovation and improvement.
The normative claims that call for a much broader efficiency definition in the context of
public sector makes it obviously very difficult to actually measure public sector productivity
and its dynamics. This would require one to considerably widen the scope of measurement
and assume that outcomes and processes were incorporated into the efficiency calculations.
Thus, further conceptualization is needed before one can really apply this wider framework to
study public sector productivity, and its relations with innovation, on operational level. As
there are many boundary crossing activities (incl. synergies, shared objectives and measures)
in the public sector, recently also system level performance measurement systems are called
for (Callender, 2011; Lönnqvist and Laihonen, 2012). Although, as of yet no such tools or
measures for this kind of evaluation are advanced. Thus, measurement should also include

20
information on the ability and support to system coordination (see Lönnqvist and Laihonen
(2012) on welfare service system productivity).
But even if restricted to the narrow/technical definition of productivity, problems related to
measurability of public sector productivity are still abundant (Pollitt and Bouckaert 2011, van
de Walle 2006, Atkinson 2005, Lane 1993, Dunleavy and Carrera 2013). This is mostly so
because of lack of adequate data (static as well as over-time) and problems related to output
valuation. In private sector, price-weighed outputs are used to overcome complexities related
to measuring and aggregating different output volumes and quality. But public services are
delivered either for free or are subsidized, thus there are no price-tags attached to public
services that would provide easy and comparable indicator to value different outputs
(Dunleavy and Carrera 2013). Consequently, even if the technical approach is seemingly
widely used in policy and management rhetoric, it has too often been over-inflated by
including a wide variety of performance criteria, it has had little to do with systematically
measuring productivity change over time (Dunleavy and Carrera 2013) and it has too often
been reduced to simple cost reduction campaigns with no apparent success (Hood and Dixon
2012).
All this has direct systemic level consequences. Due to the measurability problems the
national accounts usually treat public sector outputs as equal with inputs, meaning that
productivity is assumed to be always constant (i.e. the productivity growth is assumed to be
constantly zero).
18
In addition, cross-country comparative analyses that use proxy measures
have not been able to remedy the measurability problems as most of studies have so far failed
to capture the public sector efficiency (productivity) dynamics in meaningful ways (van de
Walle 2006).
19
What becomes evident from above is that although the concept of productivity seems to have
occupied a highly prominent place in contemporary public policy and administration rhetoric,
it has not, for various political, conceptual and analytical reasons, really applied into the
study and practice of public administration. Still, there have been some recent attempts at
refining the narrow approach of public sector productivity (Atkinson 2005, Dunleavy and
Carrera 2013). The central idea in Dunleavy and Carrera (2013) is that public productivity as
a concept of output-input ratio makes only sense if applied to organizational level and clearly
de-linked from other public sector performance indicators (e.g., effectiveness and outcomes)
as well as from macro-level innovations and policy-level or political factors. As they insist,
“public sector productivity is (and must remain) a single, deliberately limited measure,
focusing solely on how many outputs are produced for a given level of inputs” (ibid., 12).
20
They dismiss the idea that policy-level changes could be incorporated into the study of
productivity because there is simply no methodology that would differentiate between policy
change and genuine innovation. Inspired by the methodologies developed for private sector
18
Interestingly, this is often politically rewarding position to have as it inflates GDP and enables to artificially
increase GDP figures especially during economic downturns (Dunleavy and Carrera 2013).
19
Van de Walle (2006) has reviewed four international public sector performance studies (World Bank
Governance Indicators, the European Central Bank’s public sector efficiency study, the Global Competitiveness
Report and the World Competitiveness Yearbook Moreover) that one way or another attempt to measure the
public sector efficiency. As he demonstrates, the aggregate level rankings provide rather unreliable basis for
international comparisons as many of the datasets have inherent quality problems, it is difficult to determine
what the used indicators actually measure, and the datasets often give a partial and biased view (e.g. inputs are
often used as outputs, many important aspects are neglected, efficiency criteria are often assessed only against
free trade and government intervention etc.).
20
This has been also suggested by other authors, e.g., Putnam 2003.

21
total factor productivity and labor productivity measurement and earlier works on public
sector productivity (e.g., Atkinson 2005), they suggest that unit costs can be used to derive a
value of a public sector organization’s output, which then can be divided by the total volume
of inputs of that organization.
21
This is, however, far from being an easy and automatic task. Serious efforts are needed in
striking a proper balance in choosing over activities that represent the organization best and
this assumes collecting good-quality and stable data on government services’ inputs, outputs
and productivity growth. It also assumes from an organization analytical capacity to calculate
unit costs or proxies such as share of administrative costs and evaluate the true impact of
part-time staff, outsourcing and capital consumption on the organization’s productivity.
Moreover, quality adjustments are often needed, especially if dealing with complex services
and services with high variations in quality across public sector. (Ibid.)
The recent developments in public productivity measurement may offer a step forward in
terms of organizational productivity, but the systemic perspective is still largely missing. If
we look at our discussion of public sector innovations above and compare it to productivity
debate, we note a latent conflict: while productivity is best measured on organizational bases,
innovations are systemic, at least on the level of a sector, that is their impact on concrete
organization may be vague and indirect and likely also with serious time lags.
Table 3. Different meanings of productivity and efficiency
Different meanings of
productivity
Definition
Sources
Technical efficiency
(productive efficiency;
productivity)
Ratio of organization’s
outputs to inputs
Dunleavy and Carrera 2013;
Atkinson 2005
Allocative efficiency
Match between the demand
for services and their supply
(responsiveness)
Andrews and Entwistle 2013
Distributive efficiency
Distribution of resources or
services between citizens and
the relative cost to
government (and society) of
that distribution
Andrews and Entwistle 2013
Dynamic efficiency
Allocation of resources
Andrews and Entwistle 2013
21
More specifically, Dunleavy and Carrera (2013, 34) use cost-weighed outputs as a proxy value measure for
private sector price-weighed outputs. They stress the need to consider the total number of the activities
performed by an organization and weigh these activities against each other according to their unit costs (or
shares of administrative costs). In some occasions the outputs should be also adjusted to quality factors, which
is, however difficult to do. To calculate total factor productivity for a given period of time, one should then
aggregate the cost measures and divide these with the total volume of inputs (labor costs, intermediate
administration and capital consumption) of that period. If the volume of output is divided by the total number or
costs of full time equivalent staff then a labor productivity measure is derived. All costs should be deflated if
over-time analysis is carried out. Dunleavy and Carrera also provide an overview on some of the most common
techniques to be used for measuring productivity change: index-based, parametric and non-parametric
techniques (ibid, 47-55).

22
between current and future
consumption
Substantial efficiency
Capacity to produce an effect
Rutgers and van der Meer
2010
Section 4: Overview of different approaches to public sector innovation indicators
across the world
Quantitative large-scale studies aimed at measuring innovation previously mostly targeted
innovation in the private sector (e.g. Community Innovation Survey). Public sector
innovation surveys started to appear only during the recent decade. Arundel and Huber
(2013) through a literature search have identified 18 studies aimed at evaluating innovation in
the public sector using 15 large scale data sets. These studies focused on developed
economies, and also excluded service providers for health and education. Most of the studies
did not cover all five categories of the public sector, mostly focusing on public
administration.
MEPIN (Measuring Public Innovation in the Nordic Countries)
Public sector innovation is high on the policy agenda in the Nordic countries, however until
very recently lack of relevant quantitative evidence covering a wider range of public sector
organisations and innovations, limited the understanding of innovation processes in the public
sector and thus also the possible tools to promote innovation. MEPIN project aimed at
covering this gap by devising a conceptual framework and a survey questionnaire.
Development of an appropriate conceptual framework and survey questionnaire required
understanding of the needs of the potential users of the data, such as industry representatives,
national and regional level policy-makers, trade and public sector organisations, as well as
research-focused organisations. The framework and questionnaire should also take into
account the possible range of potential applications of the gathered data, such as
benchmarking, project evaluation, identification of good practices, monitoring and analysis,
as well as documentation (Bloch, 2011). The project was divided into a number of stages. At
the first stage, a conceptual framework for measuring innovation in the public sector was
devised, The conceptual framework is built upon the insights from the Community
Innovation Survey (CIS) which is intended for measuring innovation in the private sector,
adjusted to suit the needs of the public sector, as well as on the existing work on public sector
innovation. The conceptual framework included the definitional aspects of innovation in the
public sector
22
, as well as a set of indicators to evaluate innovation activities, both stimulating
and retarding factors, and forms of interaction with external actors. The preliminary
conceptual framework was piloted in all five Nordic countries, and concluded that the
framework for measuring innovation in the private sector is not directly applicable to the
public sector.
One of the issues identified as a result of the pilot study was that quantitative measures for
evaluation of innovation activities are often difficult to estimate for the respondent
22
Focusing on four types of innovation, i.e. product innovation, process innovation, organisational innovation,
and communication innovation (for definitions see Bugge et al., 2011, pp.8–9; Bloch, 2011, pp.14–15). Beside
the main four types of innovations, the report also discussed other types of innovations discussed in literature
concerned with innovation in the public sector, such as social innovations, policy innovations, strategic
innovations and systemic innovations, however, the authors of the report did not find sufficient reason to include
these additional types of innovation with an exception to strategic innovation (DIISR, 2011).

23
organisations, as the data is often not available in their accounts, thus bearing implications on
the accuracy and reliability of the data. While for measuring some indicators, it is enough
with rough estimates, when it comes to productivity analysis, more precise data is necessary.
Therefore, an approach that takes into account the existing limitations had to be developed,
which required the respondents to provide a rough estimate of total innovation expenditure in
certain time intervals. This approach made it easier for respondents to provide the data,
however, the results gathered were mixed(Bloch, 2011).
One of the parts of the questionnaire focused on linkages between public sector organisations
and other actors in their innovation activities. A range of indicators was devised in order to
capture these interactions, such as information channels for innovation, innovation
cooperation, innovation procurement, actors involved in innovation processes, financial
support for innovation, as well as drivers of innovation. The Oslo Manual (OECD and
Eurostat, 2005) was used for questions regarding innovation cooperation with some
modifications regarding the sources of information and partners for cooperation to fit the
needs of the public sector. The following list was used in the project: 1) Enterprises as
suppliers (including consultancy services); 2) Enterprises as clients/users; 3) Public sector
organisations as suppliers (e.g. universities; government research organisations); 4) Public
sector organisations as clients/users; 5) Universities/government research organisations; 6)
Other public sector organisations; 7) Citizens as users.
Questions regarding information channels are somewhat distinct from the questions regarding
information sources used within the CIS, thus distancing this indicator from indicator on
cooperation. The following channels were identified in the project: 1) internet and online
discussion forums; 2) user satisfaction surveys; 3) networks, conferences, seminars and other
gatherings; 4) hiring special personnel; 4) different evaluations (e.g. efficiency, quality, etc).
Questions concerning innovation cooperation also included questions on the source of
innovation. While the Oslo Manual distinguishes between innovations developed internally
and externally, in relation to the public sector it is important to distinguish whether
innovation was created by the public or private sector partner. Therefore the following
breakdown has been used: 1) innovation created in your own organisation; 2) innovation
created in your organisation in partnership with business; 3) innovation created in an
organisation together with another public sector entity; 4) innovations created by external
entities (both public and private).
In the recent years public procurement has become an important source of innovation in the
public sector, and also public sector has become an important driver of innovation in the
private sector, particularly so in the Nordic countries, which employ a more strategic
approach to public procurement of innovation. therefore, a number of questions were devised
to measure procurement of innovation through a variety of indicators: 1) acquisition of
components or software from ICT-suppliers; 2) acquisition of other machinery and
equipment; 3) contracting of consultancy services (ICT, management, user studies); 4)
outsourcing of service provision; 5) PPPs.
Due to the fact that public sector organisations are often constrained in their decision making,
significant share of attention has been paid to different drivers of innovation, which can
include people, organisations as well as other factors. Thus, MEPIN project included a wide
range of drivers: 1) internal forces (management, staff); 2) political forces (changes in
organisation’s budget; laws and regulations; changes or innovations implemented in partner
organisations; new policy priorities); 3) public organisations; business (both as suppliers and
customers); citizens as users.

24
As part of the range of indicators, MEPIN project also included a range of indicators aimed at
measuring the output by looking either at objectives, effects or outputs. It is often difficult to
discern and evaluate the effects of innovation, as some effects can require detailed analysis
and evaluation in order to identify the effects that actually took place. In order to provide for
a more reliable measure for the outputs, MEPIN project focused on the objectives of
innovations, devising the following list: 1) address social challenges (e.g. health, inequality);
2) fulfil new regulations, policies and other politically managed changes; 3) improve the
quality of services or goods; 4) increase efficiency of service provision; 5) improve user
satisfaction; 6) improve online services; 7) improve working conditions for employees. In
contrast to the private sector, where more general measures of output are possible (e.g.
revenue, profits), it is often difficult to create such general indicators for the public sector,
thus suggesting that output is likely to be reliably measurable on a sectoral basis (e.g. health
care, education, elderly care, etc.).
Similarly to the private sector, innovation in the public sector is greatly affected by
organisational and innovation culture, including such aspects as attitudes towards risk and
change, incentive structures and perceptions of barriers to innovation. The authors of the
MEPIN project identified four main elements of innovation process in public sector
organisations: 1) positioning of an innovation in the overall strategy of organisation; 2) the
role of management in promoting innovation in the organisation; 3) the structuring of
innovation process; 4) competences available within the organisation. According to the
suggested components of the innovation process, they proposed a list of indicators that
includes: Drivers 1) innovation strategy and organisation (goals/targets for innovation
activities; innovation strategy as part of the overall strategy; development department;
project-based innovation activities steered by a dedicated group; regular evaluation
activities); 2) innovation management and staff (managers prioritise development of new
ideas or new ways of working; top management active in implementation of innovations;
members of staff have part of their time devoted to development/innovation; incentives to
identify or develop new ideas; demographic or educational diversity of staff); as well as
barriers 3) political factors (flexibility or lack of it in laws and regulations; lack of incentives
for the organisation; lack of funding); 4) organisation and culture (risk-averse behaviour; lack
of cooperation within the organisation); 5) other endogenous conditions (lack of time for
innovation; lack of incentives for staff); 6) exogenous conditions (contract clauses limit
collaboration with suppliers; lack of capabilities among suppliers; resistance to change
among users).
Several issues were identified as a result of the pilot study. First, the results gathered using
the quantitative measures to evaluate input, such as innovation personnel and expenditures
were not particularly encouraging. One of the potential issues to occur in reporting was
linking the reported expenditures on innovation for the same unit for which the overall
expenditure is reported. As already mentioned, indicators measuring effects and impacts
encountered some challenges, therefore suggesting using objectives as an indicator for
output. Other challenges included defining the target populations for the questionnaire. Five
Nordic countries were involved in the pilot study, which identified a number of differences
across the countries in the types of organisations that are registered as enterprises, thus being
considered irrelevant. All the limitations described above suggest that a generic questionnaire
applicable to different sectors and different countries is possible with some adjustments and
supplement questions in order to address the specificities of particular countries or specific
range of public sector institutions.

25
European Public Sector Innovation Scoreboard (EPSIS)
As part of a preliminary work in developing public sector innovation indicators, an
Innobarometer survey was conducted in 2010, which consisted of 24 questions. The survey
involved 500 public organisations coming from across public administration, higher
education, local authorities and hospitals from all 27 member states. The results of the
preliminary study suggested that two thirds of respondent organisations have introduced a
new or significantly improved service during the last three years. The results also suggested
that the larger organisations were more likely to introduce an innovation than the smaller
ones; in larger organisations (over 250 employees) budget cuts were the main driver of
innovations. However, the majority of managers of public sector organisations, when asked
about the factors that would positively affect innovation, suggested that it would be more
money rather than less (EC, 2013). The report on a survey conducted in October 2010 in 27
EU Member States as well as in Norway and Switzerland and published in 2011, studied
innovation in public administration in response to changing constraints and opportunities.
EPSIS was developed along the lines of the Innovation Union Scoreboard that targets
innovation in business enterprise, distinguishing between three factors – enablers, activities
and outcomes. However, in contrast to other studies mentioned here, EPSIS also includes
measurements of the impact of public sector innovation on performance of businesses. While
the areas of responsibilities on different levels of government differ significantly, due to
insufficient data availability, the conceptual framework used in the EPSIS project does not
differentiate between different administrative layers of the public sector. Also due to the
limited data availability regarding innovation in the public sector, EPSIS project focuses only
on public administration and does not include the rest of the public sector organisations.
The indicators are used to create a scorecard showing the relative strengths or weaknesses of
a particular country by indicator. Assigning the scores to countries is based on a simple
methodology, using the distribution of countries’ scores, assigning scores to countries as
being below (score is among the lowest 33%) or above (score is among the highest 33%) the
average (score among the middle 33%) performance. The scorecard methodology was
applied to countries for which data on 21 out of 22 indicators was available. The scorecard
provides a possibility to evaluate the relative performance of every country. However, due to
a different nature of single indicators, from which some are based on hard (statistical), while
others are based on soft (opinion-based) data, the scorecard cannot be used to evaluate and
compare the overall relative performance of the responding countries. The scorecard can only
be effectively applied for evaluation of individual indicators, thus allowing identifying
particular dimensions where performance could potentially be improved. Opinion-based
indicators also pose a different challenge, as they would represent not an absolute level for a
particular indicator, but relative (i.e. representing change in a level), thus a high level for one
indicator can represent not a high level in absolute numbers, but the fact that a particular
country have started from a comparatively low level. The Innobarometer 2010 survey that
covered 4000 organisations active in public administration faced several methodological
constraints. Firstly, country-by-country comparisons were difficult due to sampling issues, as
smaller countries would result with a much smaller sample of organisations. Secondly,
country comparisons are challenging due to certain structural differences that exist between
countries, difficulties with sectoral classification of organisations, as well as inability to
construct a sample where countries would represent equal shares due to the absence of
accurate information necessary for determining the precise probability weight for each
sample.

26
The Flash Eurobarometer No 343 (Innobarometer 2011) survey measured the importance of
public sector innovation for business performance. The survey sample included 9,500
randomly selected companies from 27 EU and several non-EU states, and was carried out
using CATI method. The survey inquired firms regarding their opinion on the perceived
impact of improved public services and public sector innovation on business performance.
However, both Innobarometer 2010 and 2011 surveys provide only a snapshot view, and they
have not been replicated on a yearly basis, thus not allowing evaluating the dynamic in public
sector innovation performance. Given the overall quality of data, somewhat weakened by the
comparatively small samples in Innobarometer 2010 as well as the opinion-based data of the
Innobarometer 2011, the EPSIS 2013 can merely be considered as a pilot exploratory
analytical exercise. The value of the public sector innovation scoreboard could be enhanced if
the data would be collected on a regular basis. EPSIS 2013 authors suggest that one of the
options for collecting such data on a regular basis would be by copying the CIS approach,
which is used for measuring innovation in the private sector and which has been adopted by
many countries across the world. (EC, 2013) One should, however, be cautious with adopting
the CIS approach without the necessary adaptations for the needs of the public sector. As it
has been discussed earlier, innovation in the public sector differs significantly from that of
the private sector. Another challenge to more comprehensive and precise approach to
measuring innovation in the public sector is posed by the fact that a significant share of
public services are provided by public sector organisations outside public administration,
however, data on innovation in these organisations have not been collected, therefore
requiring another set of surveys that would provide a more comprehensive view on
innovation in the public sector.
Australian Public Sector Innovation Indicators Project (APSII)
Australian Public Sector Innovation Indicators Project is the latest among the efforts around
the world in measuring innovation in public sector; and it incorporates all the lessons learned
from earlier efforts of other countries discussed here. In contrast to the methodology applied
in the European surveys (mirroring the CIS methodology), the APSII project proposed for a
pilot a methodology based on a survey conducted in two modules - an agency and an
employee survey. The reasoning for using a twofold survey is that it can provide more
comprehensive and detailed data (indicators) on innovation in the public sector, taking into
account employee-level innovation activities. A conceptual framework used in the APSII
project for measuring innovation in the public sector incorporates five main themes: inputs to
innovation, innovation processes, outputs of innovation, outcomes of innovation, and
environmental conditions that affect innovation in the public sector (Arundel and Huber,
2013)
The survey method used for agency-level survey included all Commonwealth government
departments with more than 20 employees. A random sample of 100 participants were sent an
online survey and given six weeks for completion and submission, with agency heads signing
off the final questionnaire before sending out. The employee-level survey includes employees
of Australian Public Service (APS) identified using APS Employment database. Out of the
population a stratified random sample
23
of 9,000 employees is selected. The survey was
delivered via online based survey, as well as in paper form to those APS employees that do
not have access to personal e-mail accounts or have limited access to the Internet. One year
reference period was also used for employee-level survey.
23
Stratified by employee level, agency size, agency and location.

27
In order for the innovation indicators to be practical and useful, a range of application criteria
were proposed:
Measurable: able to be reported on by agencies and employees in a consistent manner;
Informative: provides useful information on innovation performance and capability
within organisations;
Developmental: helps improve innovation performance and build capability for
innovation;
Cost-effective: for individual organisations and the taxpayer, in particular relating to
data collection and indicators construction;
Engaging: simple, attractive and the benefits are self-evident;
Forward-looking: enables changes to current practices;
Comparable: facilitates cross-organisation and time comparisons.
United Kingdom (NESTA)
In 2008-2009 the National Endowment for Science Technology and Arts (NESTA)
commissioned six exploratory studies on public sector innovation with an aim to develop a
new Innovation Index. As a basis for the new public sector innovation indicator, NESTA
proposed to develop:
A set of indicators measuring input, output, adoption and outcome that would allow
measuring some dimensions of innovation in the public sector;
A methodology for capturing data on public sector innovation that would be piloted
by NESTA and later disseminated across other organisations;
Analytical framework for evaluation of framework conditions and public sector
productivity (DIISR, 2011).
London School of Economics Public Policy Group (LSEPPG) developed its public sector
innovation index on the basis of a survey of studies performed previously, thus taking into
account the possible shortcomings and benefits of the previous efforts. The approach uses a
mix of both published quantitative and qualitative as well as survey-based data, thus
providing more comprehensive data than the available studies (e.g., the innovation index
developed by Korean Government, see below). The authors do, however, acknowledge that
the use of publicly available data might be problematic due to the inconsistencies, instances
of misreporting and incomparability across years.
The pilot survey was conducted using telephone interviews. After piloting the survey, a range
of limitations and issues were identified. Most importantly, use of telephone interviews
limited interviewees’ ability to access organisational data, therefore limiting the validity of
data, as interviewees had to rely on their personal perceptions. Therefore, it was suggested
that the most of potential of the index tool can be realised through an online benchmarking
tool. Other issues, such as necessity to carry out the survey repeatedly, survey delivery
mechanisms, timing and context for dissemination of questionnaire need to be taken into
account in order to ensure maximum participation and response rates.
There were other challenges that NESTA identified as important on the basis of reports of the
exploratory studies. Firstly, if the index would mirror the CIS, subjectivity would affect the
outcomes of the survey. It was suggested that responding organisations would be reluctant to

28
answer subjective questions due to the scrutiny of the central government. One way to deal
with this issue would be to devise questions that are less subjective and can be confirmed
using other data sources. Similarly anonymity could potentially help in resolving this issue. If
questionnaire methodology would closely resemble the CIS methodology, it is likely to omit
open and hidden innovation, which is particularly important in context of public sector
innovation. Thus, NESTA suggested to incorporate questions that will specifically target
hidden, process, and open innovation.
Korean Government Innovation Index
Government Innovation Index (GII) is an online innovation measurement tool devised by the
Headquarters for Government Innovation under the Ministry of Government Administration
and Home Affairs of Korea in 2005. The GII is one of the early efforts aimed at measuring
innovation in the public sector thus providing insights for the subsequent innovation indices
developed elsewhere. The GII is designed to measure innovation in government agencies,
focusing on a set of innovation management components, including: innovation leadership;
vision and strategy; personnel capacity; implementation of innovation; improvement of
performance; and barriers to innovation (LSEPPG, 2008). Data collection process required
three randomly selected representatives of respondent agencies to fill in the online
questionnaire. Data on clear outliers was subsequently proofed using government documents,
as well as by conducting interviews with relevant officials. In order to avoid subjective
interpretations, survey relied heavily on factual data and less on Likert Scale-based questions.
The questionnaire also reduced the possibility of perceptual biases and inflation of
correlations possible due to the same respondents being sources of data on both dependent
and independent variables, through rigorous data filtering. The mandatory requirement for the
respondents of the GII questionnaire ensured 100 per cent response rate. (Eun Kim et al.,
2007)
While the index was devised to include a comprehensive range of components, the categories
suggest a certain focus on inputs to innovation processes, enabling factors as well as impact
of innovation, whereas measurement of innovation outputs is somewhat omitted (LSEPPG,
2008). This can also be explained by the previously mentioned reliance on measurable data
and an attempt to avoid subjective interpretations. One could assert that the authors of the
index favoured outcomes over outputs due to the difficulties related to measurement of the
latter. However, as argued widely in the innovation literature covering public sector
innovation, public sector organisations (as well as service industries in the private sector)
often innovate not only in terms of outputs or outcomes, but in terms of provision of existing
or new services (i.e. process innovation), or optimising the effectiveness of the provided
services (i.e. maximising the impact/output to the unit of input) (LSEPPG, 2008).
Table 4. Different approaches to public sector innovation measurement adopted in a range of
studies

29
Study/Aut
hors
Inputs/Enablers
Activities
/Processes
Outputs
Outcomes
Impacts/Effects/
other
EPSIS
1. Human
Resources
Share of ‘creative
occupations’;
Share of employees
with a university
degree;
2. Quality of public
services
Government
effectiveness;
Regulatory quality;
Increased efficiency
of gov services due
to ICT;
Online availability of
public services;
e-government
development index.
1. Capacities
Share of service
innovations in-
house;
Share of process
innovations in-
house;
2. Drivers and
barriers
Internal barriers to
innov;
External barriers to
innov;
Management
involvement in
innovation;
Importance of
external
knowledge;
Share of employees
involved in
innovation.
1. Innovators
Share of
organisations in
public
administration with
different types of
innovation;
Share of new
services out of all
services
innovations;
Public sector
productivity;
2. Effects on
business
performance
Improvements in
public services for
business;
Impact of
innovative public
services on
business;
3. Gov
procurement
Gov procurement as
a driver of business
innovation;
Gov proc. Of
advanced
technology
products;
Importance of
innovation in
procurement.
UK
NESTA
Innovation
capability:
leadership and
culture; management
of innovation;
organisational
enablers of
innovation;
Wider sector
conditions for
innovation:
incentives;
autonomy;
leadership and
Accessing new
ideas;
Selecting and
developing ideas;
implementing
ideas; diffusing
what works.
Improvement in
organisational
key performance
indicators;
improvement in
service
evaluation;
improvement in
efficiency;
improvement
context.

30
culture; enablers.
AUSTRA
LIA
Investment in
intangible assets;
Innovation
expenditures;
Human resources
devoted to
innovation;
Staff skills for
innovation;
Staff attitudes and
attributes to
innovation;
Sources of
information;
Technological
infrastructure for
innovation.
Explicit innovation
strategy and
targets;
Systematic, internal
measurement and
evaluation of
innovation;
Management
practices for
innovation;
Incentive and
reward structures;
Practices for
learning and
diffusing
knowledge and
innovations;
Innovation
collaboration and
alliances;
Perception of
enablers and
barriers to
innovation.
On-going
innovation projects;
Types of
innovations;
Degree of novelty
and scope of
innovations
(incremental vs
radical innov);
Innovation
intensity;
Related, intangible
outputs (patents,
trademarks).
Quality and
efficiency;
Productivity
;
User
satisfaction;
Employee
satisfaction;
Societal and
environment
al impacts;
Other
intangible
effects
(increased
trust,
legitimacy);
Effects of
innovations;
Environmental
conditions:
User innovation;
Supplier
innovation;
Wider public
sector culture
and leadership
identified as
drivers or
barriers;
External political
and legislative
factors identified
as drivers or
barriers;
Leadership and
culture;
Public tolerance
of risk.
LSEPPG
R&D activities (e.g.
dedicated innovation
or R&D unit;
dedicated strategy
unit; expenditure on
consumer or market
research; expenditure
on development and
implementation of
innovations);
Consultancy and
strategic alliances
(e.g. No of joint
ventures; consulting
expenditures;
collaboration
programmes with
universities);
Intangible assets
(e.g. patents; IPR
development
activities; unit
responsible for IPR;
trademarks);
ICT infrastructure;
Human resources
(e.g. staff with grad
education; job
satisfaction;
performance-based
Institutional
performance (e.g.
percentage of
targets met;
average time to
deliver outputs;
change programme
in place; prizes and
awards for
innovations, etc.)
e-government,
online services
(percentage of
services that can be
requested online; %
of serv that can be
delivered online);
origins of
innovation (e.g.
how many:
innovations as a
result of EU
regulations;
innovations as a
result of
ministerial/political
suggestions;
customer
suggestions;
management
suggestions)
Number of
innovations
developed for
delivery of new
outputs; number of
innovations
improving existing
outputs; number of
innovations
altogether;
New outcomes;
Number of
innovations
joining-up across
other gov
organisations;
number of
innovations
improving
performance;
Number of
people that have
been affected by
innovations
introduced in the
government
organisation for
provision of new
or existing
outputs.

31
promotion system;
churn of personnel;
etc.)
MEPIN
Information
sources: channels of
knowledge transfer;
interactions between
public organisations
and other actors (i.e.
enterprises, citizens);
Driving forces:
people, organisations
and other factors that
push organisations to
innovate (e.g. new
policy priorities,
regulations, citizen
feedback, staff,
management)
The role of ICT;
Barriers: political
factors, bureaucracy,
other internal
conditions such as
lack of incentives for
staff to innovate;
external conditions
such as resistance of
users to change.
In-house
activities: in-house
R&D; internal or
external training
and education of
staff for innovation
activities; other in-
house innovation
activities (e.g.
planning and
design; market
research; feasibility
studies, testing and
other preparatory
work for
implementation of
innovations;
External
activities: external
R&D; other
consultancy
services;
acquisition of
external know-how
(patents, licenses,
etc.); acquisition of
equipment/software
.
Procurement
practices -
acquisition of
services,
components or
software from ICT
suppliers,
contracting for
management
services.
Organizing
innovation:
innovation strategy;
the role of
management;
organizing
innovation
activities; and
organizing
competences.
Measuring
effects or
objectives:
efficiency,
quality, ICT,
organisations
and staff, other
factors – health
and safety.
Source: authors’ elaboration on the basis of LSEPPG, 2008; Bloch, 2011; Bugge et al., 2011; DIISR, 2011;
Arundel and Huber, 2013; Allman et al., 2011; EC, 2013; Eun Kim et al., 2007.
From these exercises, admittedly preliminary and exploratory in their nature, we can
draw following equally preliminary conclusions:
1. All exercises are aimed at comparing innovativeness, either across
organizations, sectors, or countries. As we have seen above, performance and

32
productivity measurement literature tends to more and more critical whether
such exercises make sense in the case of public sector.
2. All exercises seek to combine multitude of variables (from funding, to
external/internal constraints to impact) and sources (from data to surveys to
self-assessment tools).
3. Both these tendencies show that all the exercises are still dominated – despite
many efforts to the contrary – by private sector innovations logics, especially
in as far as evolutionary changes are concerned. That is, how the exercises
envision innovations emerge and especially impact (organizations,
businesses), is still dominantly efficiency driven.
Section 5. Attempts to measure social innovations
The main problems of measuring productivity, change, and innovation in the public
sector identified earlier are to a similar extent applicable in relation to social
innovation in the public sector. Due to the relative newness of the concept of social
innovation in the public sector the academic literature on measuring and evaluating
social innovation is scant, if not entirely wanting. As already mentioned above and
emphasised by Mulgan and colleagues (2013), there is no consensus on a single
theory of social innovation as there are few if any analytical frameworks that would
help to make sense of the available data. As is the case with public sector innovation
more generally, quantitative measurements for the success of innovation – such as
used for measurement and evaluation in the private sector (e.g. increase in
profitability, market share, productivity) – are often not applicable, and even if
applicable should be used with certain caveats. As argued earlier social innovation is
situated somewhere in between the public and the private sector and the process of
social innovation often involves actors representing all of the above, including public
service providers, private service providers, NGOs, non-profits and citizens on the
receiving part of the supply chain. The sheer complexity of the innovation process
due to the number of actors involved makes the evaluation of social innovation a
complex and challenging problem in itself. Another level of complexity to public
sector innovation is contributed by the fact that often times public sector aims at
addressing complex (or ‘wicked’) problems.
A successful social innovation combines and exploits the strengths and resources of
different actors, including the competences of the third sector, networks and linkages
that exist in the informal sector (local communities), the market acumen of the private
sector as well as experience, capabilities, knowledge and leadership of the public
sector in addressing social problems (Lauritzen, 2013). Thus, in the most common
performance (inputs/outputs) measurement framework, all the inputs provided (i.e.
finance, knowledge, time) by the parties involved need to be accounted. If one
approaches evaluation and measurement of social innovation from a purely
quantitative perspective, disregarding qualitative properties, then it is likely that a
range of factors facilitating social innovation (e.g. social networks and trust existing
in local communities) will not be accounted for.
Another issue lies in measuring the outputs and outcomes, or the financial and social
effects of social innovation. One of the challenges, similarly to economic externalities
such as pollution, is in attaching monetary value to the effects of social innovation,
which can be further defined as the potential savings for the public sector as a

33
provider of a service. What would be the value of a happy retired person offered an
opportunity to provide assistance to other seniors thus creating an opportunity for
being active and engaged with community? What would be the value of a better social
environment created by the joint effort of the local municipality as well as local
community and relevant NGOs in an economically disadvantaged neighbourhood?
These are all issues that have a potentially profound effect, which cannot be measured
in monetary or any other quantitative terms.
Similarly, as argued previously, social innovation stands at the intersection of private
and public sectors; it also can follow the path of technological innovation, as well as
innovation in terms of citizen empowerment and shift in social and power relations
between the service user and service provider. These changes are equally difficult to
measure, particularly in short term, as societal transformations of this kind can often
take significant time spans. Despite all the differences outlined above, there have been
recently several attempts in evaluation and measuring social innovation, which will be
summarised in what follows.
The “Social Innovation Europe Initiative” launched by the European Commission and
led by “Social Innovation eXchange” in their overview of different approaches to
appraising social innovation distinguish two groups of metrics: ones that support
policy making and metrics aimed at measuring performance of social innovations.
Apart from the efforts to measure innovation in the public sector described earlier, the
report (Reeder et al., 2012) identified a number of projects that focus on social
innovation or social aspects of innovation, such as: the WARM Wellbeing and
Assessment Model tool for assessing social capital and wellbeing of local areas;
TEPSIE – a EU FP7 project aimed at creating theoretical, empirical and policy
foundations for social innovation in Europe, as well as two reports commissioned by
NESTA, “Hidden Innovation” (Harris and Halkett, 2007) and “Innovation in public
sector organisations: a pilot survey for measuring innovation across the public
sector”.
While there is still long way to go until methods for gathering meaningful data on
effects of social innovation on a macro scale will be developed, evaluation of social
innovations on a micro level is crucial, as it provides evidence for the subsequent
decision regarding a wider uptake of innovations across relevant public sector
organisations. Here experimental or quasi-experimental methods, such as randomized
controlled trials, can potentially contribute to evaluation and appraisal of social
innovation in the public sector. Another key development could prove to be use of big
data in evaluating social innovations.
When it comes to measuring or evaluating social innovation in the public sector, both
academic and grey literature is rather quiet, hence most of the discussion is around
social innovation that originates from the private sector. Micro-level metrics play an
important role in defining whether the innovation will be scaled up and therefore
succeed on a larger scale. The metrics that are used for measuring the impact of social
innovations, however, serve different objectives. Some metrics are designed to help
the funders of social programmes and social innovations to gather the data on impact
necessary for cost-benefit analysis. Other metrics help the organisations themselves to
identify whether the choices that were made regarding implementation of one or

34
another initiative, were appropriate. Yet other metrics help in understanding the
broader impacts on social change in longer term perspective. While the purposes
overlap to a certain extent, none of the methods used for measuring the impacts are
suitable to answer the questions of all parties concerned. Besides, there are also direct
conflicts of interest between all the parties involved. (Murray et al., 2010) Murray and
associates (2010) as well as Mulgan and colleagues (2013) provide an overview of the
methods most commonly used for appraisal of social programmes and social
innovations. Mulgan et al. (ibid.) have grouped different methods according to their
purpose, namely those that measure impact; those that measure broad social
outcomes; innovation; and civil society.
Given the origins in the private sector, most of the tools used to evaluate impact try to
capture the monetary value of a project or a programme. The standard cost-benefit (or
cost-effectiveness) is one of the most commonly used, despite being notorious for
overestimating the benefits and underestimating the costs (Flyvbjerg et al., 2003).
The revealed and stated preference methods try to identify the monetary value of a
particular outcome or service by either studying the choices of individuals in similar
or related fields, or asking individuals about the price they would pay for a service or
outcome. These two methods are based on assumptions that individuals possess
perfect information about the possible outcomes and prices, are able to process this
information, as well as are perfectly rational. If these assumptions are relaxed, the
impact data produced by such methods is questionable. A number of methods were
developed by and for NGOs and charitable organisations in order to measure the
impact of charity programmes and NGO work. The Social Return On Investment
(SROI) framework is based on the traditional cost-benefit analysis and uses accepted
accounting principles, however it addresses previously uncovered fields, such as
environment and society by assigning monetary value to social and environmental
outcomes. SROI methodology has gained substantial following among non-profit
organisations. While in traditional financial accounting calculating return on
investment is relatively straightforward, it is not so in relation to social and
environmental returns, which need to have a monetary value assigned, which in some
cases is problematic from the ethical and moral standpoint. Among the numerous
different approaches to applying SROI methodology in practice a part of the
European Union’s EQUAL
24
programme developed in Finland, the ‘SYTA method’
25
stands out. SYTA methodology has been developed to analyse the effects of
operations of social firms or communities. SYTA method not only looks at the
financial performance of a social enterprise, but also at whether the programme has
been implemented according to the initial business idea, what kind of effects it aimed
at achieving within the target population as well as whether those effects have been
achieved. Thus combining both financial and qualitative performance sides it makes
cost-effectiveness evaluation of the operation possible. However, since SROI
24
http://ec.europa.eu/employment_social/equal_consolidated/
25
http://www.syta.fi/indexE.html

35
framework has been originally proposed, both the original authors of the methodology
REDF
26
as well as others have retreated from claiming that SROI is capable of
providing reliable quantitative measures (Murray et al., 2010). While the evaluation
framework is qualitative, relying on such methods as interviews and questionnaires in
order to identify the effects of a programme on the organisation or recipient of a
service, at the final stage of evaluation financial proxies are used to define the value
to the stakeholder (The SROI Network, 2011).
There is a family of methods that were developed to evaluate the effects of enterprises
on society or community, such as blended value
27
or social accounting
28
, the
relevance of those in the context of the public sector and more so their applicability
for evaluation of social innovation in the public sector is questionable.
The most recent attempt at developing a set of indicators for measuring social
innovation on a macro level, however without a particular reference to the public
sector, has been undertaken within the TEPSIE project
29
.
One of the important criteria in evaluation of innovations in the public sector, where
the traditional mechanisms of market selection do not apply, is sustainability and
diffusion of innovations. Many of the initiatives started on a project basis fail to
secure sustainability and are often abandoned as soon as funding ends, not reaching
the moment when those can be successfully disseminated across the population of
organisations. Where standard evolutionary selection through market is not an option,
experimental or quasi-experimental designs offer an opportunity to test a new service
in a particular environment, which can shed light on the feasibility of the project, as
well as on the effects of this service on an organisation providing the service and on
service recipients. Experimental designs, such as randomised controlled trials,
originating from the medicinal drug trials, have recently gained significant traction in
development economics (e.g. Banerjee and Duflo, 2008; Banerjee et al., 2013) as well
as in education and social science research(Torgerson and Torgerson, 2008).
Section 6. Discussion: what can we measure in public sector innovations
The state of the art of scholarship on public sector innovations, what it is and how it
can be measured, seems to suggest that there are four key tendencies at work and
often they cannot be clearly separated from each other:
1. Public sector innovation literature is seemingly at odds how to conceptualize
innovations in the public sector. Many researchers, similarly to policy makers and
opinion leaders, tend to have strongly normative approach to innovation (it is a good
we should aspire towards in any case). This leads to what can be called overuse in
labelling any seemingly significant change in public services delivery, organizational
26
Redf.org
27
www.blendedvalue.org
28
Here one of the best examples is the French Bilan Sociétal, a set of over a 100 indicators measuring
the impact of enterprises on society.
29
Theoretical, Empirical and Policy Foundations for Social Innovation in Europe, www.tepsie.eu

36
setup, etc, as innovation. Such indiscriminate use makes innovation as a powerful
rhetorical tool in discussions but at the same time makes it virtually impossible to
measure it. Above all, in such rhetorical exercises any evolutionary processes seem to
get almost completely lost -- although they are assumed to exists but conceptual
incoherence makes it nearly impossible to track evolutionary processes. This also
means that often events are discussed as innovations that at closer look are more or
less ordinary changes within an organization that just becomes better at what it does.
2. Most public sector innovation research does not relate innovations to public sector
logics such as authority, institutional interactions, legitimacy, trust (and related issues
such as capacity). Thus public sector innovation conceptualizations remain stuck
between implicitly copying private sector concepts while explicitly trying to move
away from them. This is most clearly evident in emerging productivity discussion:
copying private section indicators that have also a limited if politically highly visible
use in private sector discussions.
3. Measuring performance in the public sector in general is complex and complicated
matter where recent research has brought out more problems than solutions.
However, there is a clear tendency to use more performance measurements (such as
in budgeting) than less; same processes lead to increasing pressures to measure
productivity and efficiency (via such things as efficiency frontiers as benchmarks) in
order to enforce innovative practices in public organizations. Critical assessments,
however, tell us that both performance and productivity measurements tools should
be used with cautions and in particularly comparison should be used with extreme
caution.
4. All three tendencies come crashing together is measuring public sector innovations.
First, all problems prevalent in performance and productivity measurements are
compounded by conceptual confusion described above and most of all, the lack of
evolutionary dynamics in most public sector innovation conceptualizations means
that what is the measured based on these concepts is almost by definition relatively
worthless to the organizations, policies makers and to citizens. Most projects that are
currently under way as described above, indicate that difficulties increase almost
exponentially when we move from organizations (such as agencies, departments,
hospitals) to larger units (sectors, countries).
7. Bibliography
Adcroft, A., & Willis, R. (2005). The (un) intended outcome of public sector
performance measurement. International Journal of Public Sector
Management,18(5), 386-400.
Albury, D. (2005): ‘Fostering Innovation in Public Services’: Public Money and
Management: January 2005: Pages 51 – 56.
Allman, K. et al. (2011) Measuring Wider Framework Conditions for successful
innovation. A system’s review of UK and international innovation data.
Ammons, D. (2004). Productivity barriers in the public sector. In: Holzer, M. and
Seok-Hwan, L. (Eds), Public Productivity Handbook, 2nd ed., Marcel Dekker,
New York, NY
Andersen, E. A. (2009) Schumpeter's Evolutionary Economics. A Theoretical,
Historical and Statistical Analysis of the Engine of Capitalism. London:
Anthem.

37
Andersen, J., & Bilfeldt, A. (2013). Chapter 24. Social innovation in public elder
care: the role of. The International Handbook on Social Innovation: Collective
Action, Social Learning and Transdisciplinary Research, 320.
Andrews, R., & Entwistle, T. (2013). Four Faces of Public Service Efficiency: What,
how, when and for whom to produce. Public Management Review, 15(2), 246-
264.
Aristigueta, M. P. (2004). Indicators of Living Conditions: Challenges for
Accountability and Performance Marc Holzer and Seok-Hwan Lee (eds.)
Public Productivity and Performance Handbook, 431-446.
Arnaboldi, M., & Azzone, G. (2010). Constructing performance measurement in the
public sector. Critical perspectives on accounting, 21(4), 266-282.
Arundel, A. & Huber, D. (2013) 'From too little to too much innovation? Issues in
measuring innovation in the public sector', in 2013 Urbino: University of
Urbino
Arundel, A. & Smith, K.H. (2013). History of the Community Innovation Survey, In:
Gault, F. (ed), Handbook of Innovation Indicators and Measurement,
Cheltenham, UK: Edward Elgar Publishing, pp. 60-87.
Arundel, A. & Hollanders, H. (2008) Innovation Scoreboards: Indicators and Policy
Use, In: Nauwelaers, C. and R. Wintjes (ed.), Innovation Policy in Europe:
Measurement and Strategy. Cheltenham, UK: Edward Elgar, pp 29 – 52.
Atkinson, A. B. (2005). The Atkinson review: final report. Measurement of
government output and productivity for the national accounts. Palgrave
Mcmillan.
Banerjee, A. et al. (2013) The miracle of microfinance? Evidence from a randomized
evaluation. [online]. Available from:
http://papers.ssrn.com/sol3/papers.cfm?abstract_id=2250500 (Accessed 16
January 2014).
Banerjee, A. V. & Duflo, E. (2008) The experimental approach to development
economics. [online]. Available from: http://www.nber.org/papers/w14467
(Accessed 16 January 2014).
Becker, S.W., and T.L. Whisler (1967), The Innovative Organization: A Selective
View of Current Theory and Research, Journal of Business, Vol. 4, pp. 462-469.
Behn, R. D., & Kant, P. A. (1999). Strategies for avoiding the pitfalls of performance
contracting. Public Productivity & Management Review, 470-489.
Bekkers, V.J.J.M., Tummers, L.G., Voorberg, W.H. (2013). From public innovation
to social innovation in the public sector: A literature review of relevant drivers
and barriers. Rotterdam: Erasmus University Rotterdam
Benington, J. (2009). Creating the public in order to create public value?.
International Journal of Public Administration, 32(3-4), 232-249.
Benington, J., & Moore, M. H. (2011). Public value in complex and changing
times. Public Value: Theory and Practice, 1-30.
Buelens, M., & van den Broeck, H. (2007). An Analysis of Differences in Work
Motivation Between Public and Private Sector Organizations. Public
Administration Review, 67(1), 65–74.
Bloch, C. (2011) Measuring Public Innovation in the Nordic Countries: Copenhagen
Manual.
Boyle, R. (2006). Measuring public sector productivity: lessons from international
experience. CPRM Discussion Paper 35, Institute of Public Administration
(IPA), Dublin.

38
Brax, S.A. (2007). Palvelut ja tuottavuus, Teknologiakatsaus 204/2007, Tekes,
Helsinki (in Finnish).
Bugge, M. M. et al. (2011) Measuring Public Innovation in Nordic Countries. Report
on the Nordic Pilot studies - Analyses of methodology and results.
Byus, K., & Lomerson, W. L. (2004). Consumer originated value: a framework for
performance analysis. Journal of Intellectual Capital, 5(3), 464-477.
Callender, G. (2011). Alignment of inter-agency supply chains to enhance public
sector performance management. International Journal of Productivity and
Performance Management, 60(1), 9-23.
Carnegie, G. D., & Wolnizer, P. W. (1996). Enabling accountability in museums.
Accounting, Auditing & Accountability Journal, 9(5), 84-99.
Clark, J. & Guy, K. (1998) Innovation and competitiveness: a review, Technology
Analysis & Strategic Management, 10(3), 363-395.
Godin, B. (2002). The rise of innovation surveys: Measuring a fuzzy concept.
Working Paper for the Canadian Science and Innovation Indicators
Consortium, Project on the History and Sociology of S&T Statistics,
http://www.csiic.ca/pdf/godin_16.pdf.
de Bruijn, H. (2002). Performance measurement in the public sector: strategies to
cope with the risks of performance measurement. International Journal of
Public Sector Management, 15(7), 578-594.
DIISR (2011) Measuring Innovation in the Public Sector: A literature review.
Dodgson, M., Gann, D., & A. Salter (2008). The management of technological
innovation. Oxford: Oxford University Press.
Dosi, G. (1984) Technical change and economic performance, London, Macmillan.
Draghi, M. (2014) Unemployment in the euro area. Available at
http://www.ecb.europa.eu/press/key/date/2014/html/sp140822.en.html.
Drechsler, W. (2005): The Rise and Demise of the New Public Management. Post-
autistic economics review, 33.
Dunleavy, P., & Carrera, L. (2013). Growing the productivity of government services.
Edward Elgar Publishing.
European Commission (2013). Overview of European Commission initiatives related
to Public Sector Innovation. Directorate C: Research and Innovation, C.1.
Innovation Policy.
EC (2013) European Public Sector Innovation Scoreboard. A Pilot Exercise.
EU expert Group on Public Sector Innovation (2013). Interim Repor.
Eun Kim, S. et al. (2007) 'The Quality of Management and Government Innovation:
An Empirical Study', in 2007 Tucson, AZ: University of Arizona.
Fiske, E. B., & Ladd, H. F. (2000). When schools compete: A cautionary tale.
Brookings Institution Press.
Flyvbjerg, B. et al. (2003) Megaprojects and risk: An anatomy of ambition.
Cambridge University Press.
Fraisse, L. (2013). 27. The social and solidarity-based economy as a new field of
public action: a policy and method for promoting social innovation. The
International Handbook on Social Innovation: Collective Action, Social
Learning and Transdisciplinary Research, 361.
Freeman, C. (1982). ‘Technological infrastructure and international competitiveness’,
Draft paper submitted to the OECD Ad hoc-group on Science, technology and
competitiveness, August 1982, mimeo.
Freeman, C. (1987). Technology policy and economic performance: Lessons from
Japan, London, Pinter Publishers.

39
Freund, D. (1974). Max Weber and Alexis de Tocqueville. Archiv für
Kulturgeschichte, 56, 2, 457-464.
Fryer, K., Antony, J., & Ogden, S. (2009). Performance management in the public
sector. International Journal of Public Sector Management, 22(6), 478-498.
Gallouj, F. & F. Djellal (eds). The Handbook of Innovation and Services. Cheltenham,
UK and Northampton, MA, USA: Edward Elgar.
Gallouj, F. & M. Savona (2010). Towards a Theory of Innovation in Services: A State
of the Art. In: Gallouj, F. & F. Djellal (eds). The Handbook of Innovation and
Services. Cheltenham, UK and Northampton, MA, USA: Edward Elgar, pp.
27-48.
Goldstein, H., & Spiegelhalter, D. J. (1996). League tables and their limitations:
statistical issues in comparisons of institutional performance. Journal of the
Royal Statistical Society. Series A (Statistics in Society), 385-443.
Green L., et al. (2001): ‘Services and Innovation: Dynamics of Service Innovation in
the European Union’: Final Report December 2001 PREST and CRIC
University of Manchester, England.
Griliches, Z. (1995). R&D and productivity: Econometric Results and Measurement
Issues. In Stoneman, P. (Ed.), Handbook of the economics of innovation and
technological change. Oxford: Blackwell, pp. 52–89.
Griliches, Z. (1979). Issues in Assessing the Contribution of R&D to Productivity
Growth. Bell Journal of Economics, 10, pp. 92–116.
Grönross, C. and Ojasalo, K. (2004). Service productivity: Towards a conceptua-
lization of the transformation of inputs into economic results in services.
Journal of Business Research, 57, pp. 414 - 423.
Hall, B.H., & Mairesse, J. (2006). Empirical studies of innovation in the knowledge-
driven economy. Economics of Innovation and New Technology, 15(4-5), pp.
289-299.
Halvorsen, T., Hauknes, J., Miles, I., & Røste, R. (2005). On the differences between
public and private sector innovation. Publin Report D, 9, NIFU STEP, Oslo,
Norway.
Harris, D. M. & Halkett, R. (2007) Hidden innovation: how innovation happens in six
low innovation sectors.
Hartley, J. (2005). Innovation in governance and public services: Past and present.
Public money and management, 25(1), 27-34.
Hasan, M., & Kerr, R. M. (2003). The relationship between total quality management
practices and organisational performance in service organisations.The TQM
Magazine, 15(4), 286-291.
Hibbard, J. H., Slovic, P., Peters, E., & Finucane, M. L. (2002). Strategies for
reporting health plan performance information to consumers: evidence from
controlled studies. Health services research, 37(2), 291-313.
Holland, M. & Spraragen, W. (1933). Research in Hard Times. Washington: National
Research Council, Division of Engineering and Industrial Research.
Hood, C., & Dixon, R. (2012). A Model od Cost-Cutting in Government? The Great
Management Revolution in UK Central Government Reconsidered.. Public
Administration, 19(1), 114–134.
Hoque, Z. (2008). Measuring and reporting public sector outputs/outcomes:
exploratory evidence from Australia.International Journal of Public Sector
Management, 21(5), 468-93.
IMD (2012). World Competitiveness Yearbook 2012, IMD World Competitiveness
Center.

40
Johnston, R. and Jones, P. (2004). Service productivity: Towards understanding the
relationship between operational and customer productivity. International
Journal of Productivity and Performance Management, 53(3), pp. 201-213.
Kaesler, D. (2014). Max Weber. Preusse, Denker, Muttersohn. Eine Biographie,
München: Beck.
Kalberg, S. (1997). Tocqueville and Weber on the sociological origins of citizenship:
The political culture of American democracy. Citizenship Studies, 1:2, 199-
222.
Karo, E. and R. Kattel (2013) “Technological change and public management
reforms”, paper presented at Transatlantic Perspectives in Public Affairs:
Technology, Safety, Security, The Hague, May 2013
Kelly, J. M. (2005). The dilemma of the unsatisfied customer in a market model of
public administration. Public Administration Review, 65(1), 76-84.
Koch, P. & J. Hauknes. 2005. Innovation in the public sector. Report no D20. Publin
project under the EU 5th Framework Programme. Oslo: NIFU STEP.
Kuhlmann, S. (2010). Performance Measurement in European local governments: a
comparative analysis of reform experiences in Great Britain, France, Sweden
and Germany. International Review of Administrative Sciences, 76(2), 331-
345.
Lall, S. (2001). Competitiveness Indices and Developing Countries: An Economic
Evaluation of the Global Competitiveness Report. World Development, 29(9),
1501-1525.
Lane, J. E. (1993). The public sector: concepts, models and approaches. Sage
Lauritzen, J. R. K. (2013) Social innovation in Local Government - Experiences from
Denmark
LeRoux, K., & Wright, N. S. (2010). Does performance measurement improve
strategic decision making? Findings from a national survey of nonprofit social
service agencies. Nonprofit and Voluntary Sector Quarterly, 39(4), 571-587.
Lewis, J. M., & Triantafillou, P. (2012). From performance measurement to learning:
a new source of government overload?. International Review of
Administrative Sciences, 78(4), 597-614.
Linna, P., Pekkola, S., Ukko, J., & Melkas, H. (2010). Defining and measuring
productivity in the public sector: managerial perceptions. International
Journal of Public Sector Management, 23(5), 479-499.
Lodge, G., & Kalitowski, S. (2007). Innovations in Government. International
Perspectives on Civil Service Reform. Institute for Public Policy Research
(IPPR), London.
Lundvall, B.-A. (1985). Product Innovation and User-Producer Interaction. Aalborg:
Aalborg University Press.
Lönnqvist, A., & Laihonen, H. (2012). Welfare service system productivity: the
concept and its application. International Journal of Productivity and
Performance Management, 61(2), 128-141.
Lonti, Z., & Gregory, R. (2007). Accountability or countability? Performance
measurement in the New Zealand public service, 1992–2002. Australian
Journal of Public Administration, 66(4), 468-484.
LSEPPG (2008) Innovation in Government Organizations, Public Sector Agencies
and Public Service NGOs.
Lundvall, B.-Å. (ed.) (1992), National Innovation Systems: Towards a Theory of
Innovation and Interactive Learning, London, Pinter Publishers.

41
Lynn, L. (1997) “Innovation and the public interest. Insights from the private sector”,
in A. A. Altshuler and R. D. Behn, eds, Innovation in American Government.
Challenges, Opportunities, and Dilemmas. Washington, DC: The Brookings
Institution, 83-103.
Lynn, L. (2013) “Innovation and Reform in Public Administration: One Subject or
Two?” in Osborne & Brown, eds, Handbook Of Innovation In Public Services,
Cheltenham: Elgar.
Lyons, S. T., Duxbury, L. E., & Higgins, C. A. (2006). A comparison of the values
and commitment of private sector, public sector, and parapublic sector
employees. Public Administration Review, 66 (4), 605 618.
Mairesse J. & P. Mohnen (2010) Using Innovations Surveys for Econometric
Analysis. NBER Working Papers 15857. National Bureau of Economic
Research.
Martinelli, F. (2013). Chapter 26. Learning from case studies of social innovation in
the field of social services: creatively balancing top-down universalism with
bottom-up democracy. The International Handbook on Social Innovation:
Collective Action, Social Learning and Transdisciplinary Research, 346.
Meinhardt, T. and J. Metelmann. 2009. ‘Pushing the Envelope: Creating Public Value
in the Labor Market: An Empirical Study on the Role of Middle Managers’,
International Journal of Public Administration, 32, 3–4, 274–312.
Metcalfe, J.S. (2002). Knowledge of growth and the growth of knowledge, Journal of
Evolutionary Economics, 12, pp.3-15.
Micheli, P., & Neely, A. (2010). Performance measurement in the public sector in
England: searching for the golden thread. Public Administration Review,70(4),
591-600.
Miller, D. (2003). The virtual moment. Journal of the Royal Anthropological
Institute, 9(1), 57-75.
Mintzberg, H., 1996. Managing Government – Governing Management. Harvard
Business Review May-June, 75-83.
Mohnen, P. & Hall, B.H. (2013). Innovation and Productivity: An Update, Eurasian
Business Review, 3 (1), pp. 47-65.
Mohr, L. B. (1969) “Determinants of Innovation in Organizations”, The American
Political Science Review, Vol. 63, No. 1 (Mar., 1969), pp. 111-126
Moore, M. (2003) The Public Value Scorecard: A Rejoinder and an Alternative to
‘Strategic Performance Measurement and Management in Non-Profit
Organizations’ by Robert Kaplan – Working Paper #18. Boston, Harvard
University.
Moore, M.H. 1995. Creating Public Value: Strategic Management in Government.
Cambridge: Harvard University Press.
Moore, M. (1994). Public value as the focus of strategy. Australian Journal of Public
Administration, 53(3), 296-303.
Moore, M. and Hartley, J. (2008): Innovations in governance, Public Management
Review, 10:1, pp. 3-20.
Moore, M. H., Sparrow, M. & Spelman, W. (1997), Innovation in policing: From
production line to jobs shops. In Altchuler, A. and Behn, R. (Eds), Innovation
in American Government (Brookings Institution, Washington, D.C.).
Moulaert, F., MacCullum, D., Mehmood, A., Hamdouch, A. (2013) General
introduction: the return of social innovation as a scientific concept and a social
practise. In: Moulaert, F. (Ed.). The International Handbook on Social

42
Innovation: Collective Action, Social Learning and Transdisciplinary
Research. Edward Elgar Publishing.
Mulgan, G.& Albury, D. (2003), Innovations in the Public Sector (Cabinet Office,
London). Newman, J. (2001), Modernizing Governance: New Labour, Policy
and Society (Sage, London).
Mulgan. (2007). Ready or Not? Taking Innovation in the Public Sector Seriously. In
NESTA Provocation 03.
Mulgan, G. et al. (2013) 'Indicators for social innovation', in Fred Gault (ed.)
Handbook of Innovation Indicators and Measurement. Cheltenham: Edward
Elgar Publishing. pp. 420–437.
Murray, R. et al. (2010) The open book of social innovation. National Endowment for
Science, Technology and the Art. [online]. Available from: http://desis-
dop.org/documents/10157/12818/Murray,+Caulier-
Grice,+Mulgan+(2010),+The+Book+of+Social+Innovation.pdf (Accessed 16
January 2014).
Nelson, R.R. (Ed.) (1993). National Innovation Systems: A Comparative Analysis,
Oxford: Oxford University Press
Nelson, R. R. and Winter, S. G. (1982), An Evolutionary Theory of Economic
Change, Cambridge, Mass., The Belknap Press of Harvard University Press.
Noordegraaf, M., & Abma, T. (2003). Management by measurement? Public
management practices amidst ambiguity. Public Administration, 81(4), 853-
871.
OECD (2013) Glossary of Statistical Terms. [online]. Available from:
http://stats.oecd.org/glossary/detail.asp?ID=2199. [online]. Available from:
http://stats.oecd.org/glossary/detail.asp?ID=2199.
OECD & Eurostat (2005) Oslo Manual. Guidelines for Collecting and Interpreting
Innovation Data. 3rd edition. OECD Publishing. [online]. Available from:
http://epp.eurostat.ec.europa.eu/cache/ITY_PUBLIC/OSLO/EN/OSLO-
EN.PDF (Accessed 19 May 2012).
OECD. (2010). Measuring innovation. A new perspective. Paris: OECD Publishing.
OECD. (2005). Oslo Manual. Guidelines for Collecting and Interpreting Innovation
Data. 3rd edition. Paris: OECD Publishing.
OECD. (1992). Oslo Manual. Guidelines for Collecting and Interpreting Innovation
Data. Paris: OECD Publishing.
O'Flynn, J. (2007). From new public management to public value: Paradigmatic
change and managerial implications. Australian Journal of Public
Administration, 66(3), 353-366.
Osborne, D. & Gaebler, T. (1992). Reinventing Government, Lexington, MA:
Addison-Wesley.
Osborne, D. E., & Plastrik, P. (2000). The reinventor's fieldbook: Tools for
transforming your government. San Francisco: Jossey-Bass.
Osborne, S., P. & Brown, K. (2005): ‘Managing Change and Innovation in Public
Service Organisations’: Routledge, Abington, England.
Osborne, S. P. & K. Brown (2013) “Introduction: Innovation in Public Services”, in
Osborne & Brown, eds, Handbook Of Innovation In Public Services,
Cheltenham: Elgar.
Osburg, T. (2013). Social Innovation to Drive Corporate Sustainability. In Social
Innovation (pp. 13-22). Springer Berlin Heidelberg.

43
O’Toole, L.J., K.J. Meier and S. Nicholson-Crotty. 2005. ‘Managing Upward,
Downward and Outward: Networks, Hierarchical Relationships and
Performance’, Public Management Review, 7, 1, 45–68.
Packard, T. (2010). Staff perceptions of variables affecting performance in human
service organizations. Nonprofit and Voluntary Sector Quarterly, 39(6), 971-
990.
Pavitt, K. (1984). Sectoral Patterns of Technical Change - Towards a Taxonomy and a
Theory. Research Policy, 13(6), 343-373.
Perez, C. (2002): Technological Revolutions and Financial Capital: The Dynamics of
Bubbles and Golden Ages. Cheltenham: Elgar.
Pianta. M. (2004) Innovation and employment. In Fagerberg, J., Mowery, D. & R.
Nelson (eds). The Oxford Handbook of Innovation. Oxford: Oxford University
Press, 568-598.
Pidd, M (2008). A criticalassessment of performance measurement for policy making.
In: Managing to Improve Public Services. Cambridge University Press,
Cambridge.
Pollitt, C. (2011) ‘Innovation in the public sector: an innovatory overview’, in
V.Bekkers, J.Edelenbos and B.Steijn (eds.) Innovation in the public sector:
linking capacity and leadership, Basingstoke, Palgrave/Macmillan, 35-43 .
Pollitt, C., & Bouckaert, G. (2011). Public management reform: A comparative
analysis-new public management, governance, and the Neo-Weberian state.
Oxford University Press.
Porter, M.E. (2010). What is value in health care? The New England Journal of
Medicine, 2477-2481.
Porter, M.E., M. Delgado, C. Ketels, and S. Stern. (2008). Moving to a New Global
Competitiveness Index, in World Economic Forum (ed.), The Global
Competitiveness Report 2008-2009.
Porter, M.E. (1998). Clusters and Competition: New Agendas for Companies,
Governments, and Institutions. Harvard Business School Working Paper, No.
98-080.
Porter M.E. (1990). The Competitive Advantage of Nations. New York: Free Press.
Potts, J. & Kastelle, T. (2010) Public sector innovation research: What’s next?
Innovation: management, policy & practice. 12 (2), 122–137.
Putnam, R. (1993). Making Democracy Work: Civic Traditions in Modern Italy.
Princeton, New Jersey: Princeton University Press.
Rantanen, H., Leva¨, K. and Pekkola, S. (2007). Performance measurement
implementation in a knowledge-based public organisation. International
Journal of Business and Systems Research, 1(3), 343-53.
Reeder, N. et al. (2012) Strengthening social innovation in Europe: journey to
effective assessment and metrics. [online]. Available from:
http://ec.europa.eu/enterprise/policies/innovation/files/social-
innovation/strengthening-social-innovation_en.pdf (Accessed 29 November
2013).
Ridgway, V. F. (1956). Dysfunctional consequences of performance
measurements. Administrative Science Quarterly, 1(2), 240-247.
Rittel, H. W., & Webber, M. M. (1973). Dilemmas in a general theory of
planning. Policy sciences, 4(2), 155-169.
Roessner, J. D. (1977). Incentives to innovate in public and private
organizations. Administration & society, 9(3), 341-365.

44
Rutgers, M. R., & van der Meer, H. (2010). The Origins and Restriction of Efficiency
in Public Administration Regaining Efficiency as the Core Value of Public
Administration. Administration & Society, 42(7), 755-779.
Sahay, B. S. (2005). Multi-factor productivity measurement model for service
organisation. International Journal of Productivity and Performance
Management, 54(1), 7-22.
Schöning, M. (2013). Social Entrepreneurs as Main Drivers of Social Innovation. In
Social Innovation (pp. 111-118). Springer Berlin Heidelberg.
Schumpeter, J. A. (1912), Theorie der wirtschaftlichen Entwicklung, Berlin:
Duncker&Humblot
Schumpeter, J. A. (1939) Business Cycles. A Theoretical, Historical, and Statistical
Analysis of the Capitalist Process, Vol 1. New York: McGraw-Hill.
Schwab, K., Porter, M. E., Sachs, J. D., Warner, A. M., & Levinson M. (Eds.), The
global competitiveness report 1999. New York: Oxford University Press for
World Economic Forum.
Sillanpää, V. (2013). Measuring the impacts of welfare service innovations.
International Journal of Productivity and Performance Management, 62(5),
474-489.
Smith, K. (2005) Measuring innovation, in Fagerberg, J., Mowery, D.C. & Nelson,
R.R. (Eds). The Oxford Handbook of Innovation. Oxford, UK: Oxford
University Press, pp. 148-177.
Smith, P. (1995). On the unintended consequences of publishing performance data in
the public sector. International Journal of Public Administration, 18, 277-310.
Stackman, R. W., Connor, P. E., & Becker, B. W. (2006). Sectoral Ethos: An
Comparison of the Personal Values Systems of Female and Male Managers in
the Public and Private Sectors. Journal of Public Administration Research and
Theory, 16(4), 577–597.
The SROI Network (2011) The economic and social return of Action for Children’s
Family Intervention Project, Northamptonshire. [online]. Available from:
http://www.thesroinetwork.org/publications/cat_view/211-sroi-case-studies-
assured (Accessed 27 January 2014).
Talbot, C. 2008. Measuring Public Value—A Competing Values Approach, London:
The Work Foundation.
Tangen, S. (2005). Demystifying productivity and performance. International Journal
of Productivity and Performance Measurement, 54(1), 34-46.
Thompson, V. A. (1965) Bureaucracy and Innovation. Administrative Science
Quarterly, Vol. 10, No. 1, Special Issue on Professionals in Organizations
(Jun., 1965), pp. 1-20.
De Tocqueville, A. (1876) Democracy in America, Vol 1. Boston: John Allyn.
Torgerson, D. J. & Torgerson, C. (2008) Designing randomised trials in health,
education and the social sciences: an introduction. Palgrave Macmillan.
Townley, B. (2008). Reason's Neglect: Rationality and Organizing. Oxford: Oxford
University Press.
Van de Walle, S. (2008). Comparing the performance of national public sectors:
conceptual problems. International Journal of Productivity and Performance
Management, 57(4), 329-338.
Van Thiel, S., & Leeuw, F. L. (2002). The performance paradox in the public sector.
Public Performance & Management Review, 267-281.
Vargo, S. L., & Lusch, R. F. (2008). Why “service”?. Journal of the Academy of
marketing Science, 36(1), 25-38.

45
Verhoest, K., B. Verschuere, G. Bouckaert and G. B. Peter (2006). “Innovative Public
Sector Organizations”, in Colin Campell et al., Comparative Trends in Public
Management. Smart Practices Toward Blending Policy and Administration,
Ottawa: Canada School of Public Service, 106-118.
Vivarelli, M. (2007). Innovation and Employment: A Survey. In: Hanusch, H. & A.
Pyka (eds.) Elgar Companion to Neo-Schumpeterian Economics. Cheltenham:
Elgar, 719-732.
Waller, M. (2006). Fast track municipal and service restructuring to help meet new
challenges, Demo – Journal for Local Democracy Development, 10-11.
Weber, M (2009). Wirtschaft und Gesellschaft. Grundriß der Verstehenden
Soziologie. 5th edition, Mohr Siebeck.
Williams, I., & Shearer, H. (2011). Appraising public value: past, present and futures.
Public administration, 89(4), 1367-1384.
Wilson, J.Q. (1989) Bureaucracy: what government agencies do and why they do it,
New York: Basic Books
Wolf, M. (2013) “How austerity has failed”, New York Review of Books,
http://www.nybooks.com/articles/archives/2013/jul/11/how-austerity-has-
failed/?pagination=false.