ArticlePDF Available

Public Managers Amidst Ambiguity Towards a Typology of Evaluative Practices in Public Management

Authors:

Abstract and Figures

Contemporary public management is characterized by a strong tendency to introduce performance measurement in order to reduce complexity. Public managers face two challenges when performing their work: uncertainty and ambiguity. Ambiguity is understood as the absence of or contradictory interpretations about what needs to, can and should be done, when and where. In this article we argue that the intensity and nature of ambiguity vary, depending on the public management setting. This has serious implications for the type of evaluation chosen. Performance measurement may be appropriate when ambiguity is relatively low, but it is difficult and potentially damaging in settings marked by a high degree of ambiguity. In these latter cases, evaluation approaches that acknowledge ambiguity through dialogue are more suitable. To structure this line of reasoning, we distinguish four public management settings (industrial, enforcing, professional and strategic) and relate this to different evaluation approaches.
Content may be subject to copyright.
Public Managers amidst Ambiguity
Towards a Typology of Evaluative Practices in
Public Management
TINEKE A. ABMA
University of Maastricht, The Netherlands
MIRKO NOORDEGRAAF
University of Utrecht, The Netherlands
Contemporary public management is characterized by a strong tendency to
introduce performance measurement in order to reduce complexity. Public
managers face two challenges when performing their work:uncertainty and
ambiguity. Ambiguity is understood as the absence of or contradictory
interpretations about what needs to, can and should be done, when and
where. In this article we argue that the intensity and nature of ambiguity
vary, depending on the public management setting. This has serious
implications for the type of evaluation chosen. Performance measurement
may be appropriate when ambiguity is relatively low, but it is difficult and
potentially damaging in settings marked by a high degree of ambiguity. In
these latter cases, evaluation approaches that acknowledge ambiguity
through dialogue are more suitable. To structure this line of reasoning, we
distinguish four public management settings (industrial, enforcing,
professional and strategic) and relate this to different evaluation approaches.
KEYWORDS: ambiguity;new public management;performance
measurement;public management settings;typology of evaluation practices
New Public Management and the Role of Performance
Measurement
1
Over the last 10 or 15 years a revolution has taken place within the administra-
tive world. The new public management movement has played a key role in the
transition towards decentralization, public–private partnerships, marketization,
privatization, competitive tendering and new forms of regulation (Hood, 1991;
Kickert, 1997; Power, 1999; Pollitt and Bouckaert, 2000). The state has turned
into a ‘managerial state’ (Clarke and Newman, 1997). Alongside these develop-
ments, new forms of accountability – citizen’s charters, performance tables, new
forms of direct democracy – have come to the fore to make governments more
Evaluation
Copyright © 2003
SAGE Publications (London,
Thousand Oaks and New Delhi)
[1356–3890 (200307)9:3;285–306;036676]
Vol 9(3): 285–306
285
04Noordegraaf (bc/d) 8/7/03 10:50 am Page 285
accountable and accessible. One of the best Dutch examples of this transition is
the Van Beleidsbegroting Tot Beleidsverantwoording (VBTB) [From Policy
Budgeting to Policy Accountability], a large-scale attempt to introduce budgetary
reform within central government, led by the Ministry of Finance. Since 2000, all
departments produce performance-based ‘policy budgets’ with ‘policy articles’,
in which objectives, performances and means are tightly linked. VBTB does not
stand alone. It is influenced by earlier operations, most notably the American
Planning Programming Budgeting System (PPBS) system, as well as other
reforms, both inside and outside the Netherlands (e.g. Pollitt and Bouckaert,
2000).
The Tilburg model is an earlier Dutch example. In the 1980s, the city of Tilburg
decided to introduce a performance-driven model in order to deal with severe
financial difficulties (Krosse, 1995; Vugt and Tops, 1998). Public administration
must perform, it was argued. It must: find out what its ‘customers’ (citizens) want;
determine – in a precise and measurable way – what it will ‘produce’; then
produce; and evaluate if production ‘targets’ have been met. In order to realize
this, a contract-based ‘planning and control’ logic was introduced. Performance
targets were identified and operationalized in terms of performance indicators,
performance contracts were established, performance was measured and
controlled by comparing indicator-based performance ‘facts’ with performance
targets. Such logic was matched with a business-like organizational structure, in
which the roles and responsibilities of key players were well-defined. Elected
council members determined why things were done, the mayor and aldermen
determined what things were done, and civil servants determined how things
were done. A traditional functional structure, which is considered to be hierar-
chical and bureaucratic and therefore too inhibiting of decisive action, was
replaced by a modern ‘sectoral’ structure, in which the key players were partly
autonomous and tightly connected by way of contracts.
The examples indicate that with the introduction of new public management,
the world of public administration has, first and foremost, become a world of
measurement. The ‘new’ ideology is reflected in the use of performance-driven
financial systems and organizational structures, such as the VBTB operation and
the Tilburg model.
2
We should, however, not overlook the cultural and symbolic
impact of this movement. This development has also affected the organizational
values and discourse. Objectivity, rationality, productivity, effectiveness,
efficiency and transparency have become dominant values in many public-
management organizations. The economic discourse in terms of products,
consumers, targets, facts, evidence, planning and control also reflects the domi-
nance of this line of thinking (Grit, 2000).
The popularity of performance measurement and the emphasis on trans-
parency and effectiveness are understandable in the societal and historical
context. Governments have been heavily attacked in the final decades of the
last century. They are now seeking tools and concepts to cope with scarcity, as
they face increasing problems and decreasing means. They also have to cope
with critical citizens and individuals who want value for money and whose
unique identities and accompanying needs and wants can no longer be reduced
Evaluation 9(3)
286
04Noordegraaf (bc/d) 8/7/03 10:50 am Page 286
to preordained categories (van Gunsteren, 1992). Technological improvements,
such as improved ICT systems and greater (informational) complexities,
strengthen expert power. Governing is an extremely difficult task in this context.
Within individualistic, capitalist societies, guidance and compliance are found in
professional evaluation and other scientific practices such as performance
measurement (House, 1993). Measurement functions as a source of authority to
inform, legitimate and control managerial decisions. Performance measurement
is perceived to be the way out, as it plays a dual role. It fits with the climate in
which citizens and experts find themselves and it introduces new control modes
within a performance-oriented climate.
Despite the popularity of the new public management, several scholars have
criticized the ‘misplaced comparison’ and ‘misplaced generalisation’ that char-
acterize the movement (Noordegraaf, 1999, 2000). First, proponents of the new
public management compare the public sector with one private company, whilst
a comparison with the private sector seems more reasonable. Second, these
proponents argue that one and the same management model is appropriate for
the wholepublic sector. Performance measurement, however, runs into problems,
because this method cannot deal with contradictory preferences, contested
knowledge, fuzzy means–ends relations and unclear relations between outputs
and outcomes that characterize certain management settings (e.g. March and
Olsen, 1979; Wilson, 1989; Weick, 1995). The use of performance indicators is
then questionable: whose values and criteria are represented, and whose values
and voices are subsequently neglected or ignored (Greene, 1999)? Others have
noticed that in these cases performance measurement may give rise to the so-
called ‘performance paradox’ (Meyer and Gupta, 1994; Leeuw, 2001). Perform-
ance indicators are introduced in areas where performance indicators fail, such
as welfare or education, so that, in the end, performance might be harmed instead
of improved. Others have criticized business-like measurement models, and
accompanying notions like ‘transparency’ and ‘effectiveness’ and underscored
the complex, political nature of public-sector performance (e.g. Moore, 1995;
Kickert, 1997; Pollitt and Bouckaert, 2000).
We share the critique on the unreflective application of performance measure-
ment in the public domain and argue for a differentiation in terms of manage-
ment conditions or settings. It seems reasonable to acknowledge the nature of
multiple management settings in order to cope with problems of performance
measurement and evaluation to improve practice. The rest of this article will be
devoted to a detailed analysis of this critical move. We set ourselves three main
tasks. First, we will try to establish a conceptual understanding of multiple
management settings. We will critically reflect on the assumptions in which
performance measurement is grounded as well as giving a theoretical account of
the problems of performance measurement, using the concepts of uncertaintyand
ambiguity for this task. Second, we will strengthen the aforementioned differen-
tiation in terms of policy/management settings. We will provide a theoretically
grounded typology in order to counteract the misplaced generalization of
performance measurement. We use the notion of public management settings to
this end. Third, we will distinguish between different types of evaluation
Abma and Noordegraaf:Public Managers amidst Ambiguity
287
04Noordegraaf (bc/d) 8/7/03 10:50 am Page 287
methods. We argue that specific public management settings require specific
evaluative practices.
Multiple Settings of Public Management
Uncertainty versus Ambiguity
The managerial world is a disorderly world: managerial working days are hectic
and chaotic, issues are unstructured and complex, attempts to influence others
are difficult, and the effects of managerial interventions are hard to predict. Most
(public) managers are aware of this – they really ‘feel’ it – and a large part of the
(public) management literature acknowledges it, in one way or another. How this
is done varies, in two crucial ways:
1. the nature of disorder can be conceptualized differently; and
2. the ways to cope with disorder can be treated differently (Noordegraaf,
2000).
Most analyses of management conditions perceive disorder or complexity in
terms of uncertainty or information shortage or ‘imprecision in estimates of
future consequences conditional on present actions’ (March, 1994). They notice
the lack of understanding of how environmental elements are changing (state
uncertainty), the unknown impact of environmental changes on the organization
(effect uncertainty), and the various response options that are open to the
organization (response uncertainty) (Weick, 1995). Such a perception of disorder
rests upon three assumptions:
a manager knows what he wants or what is important, so he knows what he
does not know;
a manager can collect the right information about what has happened or is
happening; and
a manager knows what to do with this information, not in the least because
the collected information can be information about causal relations.
Management complexity can also be conceptualized differently, in terms of
ambiguity or ‘equivocality’ or ‘an ongoing stream that supports several different
interpretations at the same time’ (Weick, 1995). This turns the aforementioned
assumptions upside down. Preferences, goals, priorities might be absent or
shifting or contradictory. A manager might not know what he wants or what he
does not know. Information is inherently contested: a manager might not know
which information is relevant or how it might be interpreted, or different
managers see different information, or if they see comparable information, they
might ‘see’ something different. It might be unclear how to use information:
causal links might be interpreted in different ways. The concept of ambiguity
highlights the interpretive nature of the managerial world, as it distinguishes
between informational stimuli and the meaning of stimuli. The informational
stimulus might be ‘out there’, but the perception of it and the establishment of
its meaning occur during managers’ dialogues and interactions, when diverging
interests, values and interpretations rise to the surface. That explains why
Evaluation 9(3)
288
04Noordegraaf (bc/d) 8/7/03 10:50 am Page 288
management is an unending process, why managers are not all-powerful ‘orches-
tra conductors’ and why managerial working days are ‘socio-chaotic’ (Noorde-
graaf, 2000). So-called managerial behaviour research has convincingly shown
this (Mintzberg, 1973; Cohen and March, 1974; Kaufman, 1980; Kotter, 1982. For
a summary, see Noordegraaf and Stewart, 2000).
This leads us to the second variation: the ways managers might cope with such
complexity are treated differently. Most argue that ‘subjective’ elements must
be squeezed out as much as possible: managers must bring clarity, consistency
and order. Most management textbooks present neat models and techniques
for bringing this about and are still inspired by age-old Wilsonian, Tayloristic
and Gulickian management insights. The new public management might be said
to fall back on these ingredients (e.g. Pollitt, 1993) in order to create smooth
running output-oriented machines.
These basic ingredients are mixed with a bit of ‘bounded rationality’, a touch of ‘contin-
gency’ thinking and a pinch of ‘culture’, and there you have it – the present-day recipe
for anti-bureaucratic tastes. (Noordegraaf, 2000: 3)
Some, on the other hand (e.g. March and Olsen, 1989; Forester, 1993; Weick,
1995), argue that management will always be interpretive and that the presence
of ambiguity is a precondition for creativity, innovation and survival – it might
be normatively desirable. John Forester (1993) relates the focus on uncertainty
versus ambiguity to the instrumental versus communicative conception of public
management and planning. If management is perceived as ‘context-free techni-
cal problem solving’ emphasis will be placed on reducing uncertainty through
evidence. On the other hand, if management is understood as ‘context-dependent
practical action’, the emphasis is placed on dealing with ambiguity and un-
certainty through communicative processes of dialogue, argumentation and
social learning.
The conceptualization of the public management practice in terms of an inter-
pretative enterprise is an important scholarly position, especially for evaluative
purposes. It raises several questions:
If management is an interpretive enterprise, how do you know what must
be done? What is effective managerial action if the objective grounds for
assessing ‘effectiveness’ are missing?
Is ambiguity always desirable? Can we think of situations in which ambi-
guity is or might be largely reduced?
The first question is relatively easy to answer, as leading scholars (such as March
and Olsen and Weick) have indicated how managers deal with ambiguity: they
do not fall back on objective but social grounds – they do what is appropriate,
given the socio-cognitive or institutional setting in which they operate. March
and Olsen (1989) speak of a ‘logic of appropriateness’, as opposed to an instru-
mental ‘logic of consequence’ (that pervades the new public management), which
is grounded by rules of appropriateness. Social psychologist Karl Weick (1979)
shows how individuals ‘enact’ sensible realities, thus creating ‘consensually vali-
dated grammars for reducing equivocality by means of sensible interlocked
Abma and Noordegraaf:Public Managers amidst Ambiguity
289
04Noordegraaf (bc/d) 8/7/03 10:50 am Page 289
behaviour’. Institutional scholars explore the ‘normative [March and Olsen]
and cognitive [Weick] structures that provide stability and meaning to social
behaviour’ (Scott, 1995). Such a socio-cognitive picture of ambiguity resolution
can also be found elsewhere, for instance in constructivist and discursive
approaches to public administration (van Twist and Termeer, 1991; Hajer, 1993;
Lips, 1996).
The second question is harder to answer, as these authors are less explicit
about different levels of ambiguity. It is our view, however, that this question
needs an answer, as it would be another ‘misplaced generalisation’ if we were to
conclude that all is ambiguous and performance measurement is always imposs-
ible. We need some sort of classification of different levels of ambiguity. We have
constructed such a classification, based on a typology of management settings.
These settings, we argue, differ in two ways:
the intensity of ambiguity – settings are more or less ambiguous, or in other
words, there is a stronger or weaker social grounding; and
the nature of ambiguity – ambiguity takes on special characteristics in
different settings. Settings might, for instance, differ in terms of measura-
bility: some issues might be directly measurable, some only in the long run,
and some might be unmeasurable. The meaning of measurability might
also vary.
Multiple Public Management Settings
There are several possible ways to classify settings. Some authors distinguish
between different types of public organizations, based upon primary activity (e.g.
policy making or policy implementation) or upon output/outcome features.
Wilson (1989) distinguishes between four types of organizations, based upon the
(un)measurability of outputs and outcomes. Others distinguish between types of
activities (Hofstede, 1981) or between different kinds of users that are served by
public organizations: Mintzberg (1996) uses the consumer/client/citizen/subject
classification to distinguish between four types of public organizations (see
Table 1). These distinctions are valuable, and they might be used to put together
a typology that is ‘appropriate’ for our purposes. This can be done as outlined
in Table 1 (for a fuller account, see Noordegraaf [1999]).
Evaluation 9(3)
290
Table 1. User Roles
Role Definition
Consumer A consumer buys identifiable products and can choose whether and where to buy.
Client A client receives professional help and is dependent on professional expertise.
Citizen A citizen participates in arranging, organizing and ordering public infrastructure,
together with politicians.
Subject A subject is subjected to public authority, which might be accompanied by the use
of force.
Source: Based upon Mintzberg (1996)
04Noordegraaf (bc/d) 8/7/03 10:50 am Page 290
Firstly, we do not start with ‘organizations’ but with ‘primary’ or ‘production’
processes that occur within, around and between formal organizations. Secondly,
we identify the main dimensions that run through the earlier classifications and
that might be used to assess the intensity and nature of ambiguity. The follow-
ing two dimensions seem crucial:
The nature of interaction between ‘producers’ and those for whom the pro-
ducer produces. The consumer/client/citizen/subject classification can be
used in this case. See Table 1 for a fuller summary: interactions with cus-
tomers and subjects are one-sided; interactions with clients and citizens are
one-sided.
The nature of the production process. Some processes are routine: repeti-
tive, standardized; they flow according to given algorithms (if . . . then
reasoning). Other processes are non-routine: unstandardized, incidental;
algorithms break down or are absent.
3
Both dimensions enable us to construct a two-by-two table, in which four types
of production processes are distinguished (see Table 2). These settings will be
used to explore the intensity and nature of ambiguity, as well as the consequences
for evaluation. In short, each setting requires appropriate evaluation methods.
Nature and Levels of Ambiguity in Multiple Public
Management Settings
We have distinguished four settings of public management. Each setting has its
own features and ‘logic’, and related forms of confusion and contradictions. In
this section we will further identify the nature and intensity of ambiguity in these
settings.
One-sided Interaction: Predefined ‘Product’
Industrial and enforcing processes are both characterized by a one-sided inter-
action between producer and consumer/subject. The interaction is ‘one-sided’ in
the sense that the provider offers a predefined product. The product may change
over time taking into account consumers’ responses, but the provider can and will
not directly take into account the response from the consumer. The consumers
Abma and Noordegraaf:Public Managers amidst Ambiguity
291
Table 2. Public Management Settings
One-sided interaction Two-sided interaction
Routine production Industrial processes Professional processes
e.g. public transport, e.g. healthcare, higher
permits, welfare benefits education
Non-routine production Enforcing processes Strategic processes
e.g. police, defence, e.g. physical, social and
inspection economic infrastructure
Source: Adapted from Noordegraaf (1999)
04Noordegraaf (bc/d) 8/7/03 10:50 am Page 291
act as receivers, not as co-producers. This implies that the provider largely defines
and controls the product and that the product is relatively insensitive to varying
expectations and interpretations.
In the case of industrial processes, consumption of the products of many
private companies operating on the free market serves as a frame of reference:
the products are tangible, can be stored and identified and consumers are free
to choose a product among a group of providers. Furthermore, outcomes are
knowable; buying and using toothpaste will result in healthier teeth and less
decay; running trains will lead to an improved transport system and mobility; and
allocating welfare funds will improve the financial position of individuals. Such
an ideal-typical production is seldom seen within the public sector, not least
because direct competition is often missing and because consumption is not
always a voluntary act. Another reason for the rarity of industrial processes in
the public sector is the fact that state ‘industries’ are the first to become priva-
tized. Examples of public production that can be typified as industrial include
the delivery of post and telecom services, the production of energy and public
transport. Examples where the industrial character is less obvious include the
delivery of licences and permits and the payment of benefits by social and other
services. In the latter case it will be clear that the ‘real’ equality is often absent:
rights are given in return for certain duties.
In the case of enforcing processes, individuals are the subjects of government
authority. Citizens (as subjects) do not have freedom of choice. Legitimized
power is hard to dismiss; individuals can be arrested and locked up. Enforcing
processes are meant to control the boundaries and demarcations in and of the
public space. That is the reason why one can also speak of ‘integer’ processes:
the least one may expect of practitioners of these processes is that they do not
violate the boundaries and demarcations themselves. This is especially the case
when it concerns policing activities, the accusing and judging of individuals and
the imprisonment of sentenced citizens. Other examples of the establishment of
boundaries are the inspection of public service delivering, national defence and
‘peace-keeping’ activities. The daily operations and activities of enforcing
processes are measurable. Think of criminal acts policemen encounter. The
effects, e.g. volume of theft, are to a certain degree also known. On the other
hand, the processes are inherently disputable and the ambiguity is of a ‘special’
nature. Firstly, we notice dramatic ‘policy conflicts’ that are hidden behind the
screens of the daily production. Consider questions such as:
Should the police primarily function proactively or act as a reactive, repres-
sive force?
Is the army’s function defensive, peace-enforcing or peace-keeping?
Is the intention behind the imprisonment of citizens to lock people up or
to re-educate them?
Secondly, the nature of the production is such that it is always symptomatic and
partly invisible: suspects act strategically, prisoners try to outsmart their ‘captors’.
The process has the character of a cat-and-mouse game.
Evaluation 9(3)
292
04Noordegraaf (bc/d) 8/7/03 10:50 am Page 292
Two-sided Interaction: Negotiated ‘Product’
Professional and strategic processes are characterized by two-way interactions.
The client and citizens act as co-producers and actively influence the service and
this implies that the ‘product’ is a negotiated outcome and less controlled by the
provider. As a negotiated outcome the service will vary depending on the actors’
expectations and context.
In the case of professional practices, the client depends on the expert and the
judgement of professionals. Professionals, in the classical sense of the word, are
individuals who have followed a professional education and training, who are
members of professional associations, who read professional journals, and who
are subject to professional codes and legal procedures. Their expertise and
autonomy are in part the result of the ambition to build power, gather money
and receive status, but also relate to the specific nature of their work. The private
and confidential character of knowledge about clients gives professionals a
discretionary space to act without the interference of third parties. Examples of
professional processes can be found in healthcare, social work and educational
settings.
Another feature of professional work is that the ‘product’ is intangible
(health; welfare; education) and negotiated between professional and client.
Without the compliance of the patient a medical treatment will, for example,
have no result. Traditionally the relationship and interaction between the
professional and client is asymmetrical. Due to the changes in our modern
society, such as the information and communication revolution, this hierarchical
relationship and vertical interaction are starting to come under pressure. Many
clients and informal carers no longer automatically accept the expertise and
authority of their medical doctor, and ask for ‘second opinions’. Parents are
more critical about the school and the qualities of teachers. The work of
professionals often concerns sensitive topics and they sometimes have to make
far-reaching, ethical decisions. For example, medical doctors and nurses who are
confronted with claims of parents for an abortion or with the wish of family
members to conduct passive euthanasia. The effects of professional work are
problematic to assess and measure. The societal effects of teaching are, for
example, only ‘visible’ in the longer run and even then contested. The long-
standing discussion on the impact of the mental-health care sector on the general
welfare of the population is another illustration of how hard professional work
is to measure.
Strategic processes concern the public, social and physical infrastructure and
the demarcation of public space. Strategic processes are technically complex,
and they involve decisions that have long-term effects on many levels (inter-
national, national, regional and local). Given the far-reaching implications,
many interests and values (quality of life, environmental, economical and
aesthetic values, etc.) are at stake, and many stakeholders try to influence these
processes. As such they are highly ‘complex’. In the public administration litera-
ture they are qualified as ‘politically sensitive’. The interaction with citizens and
others is, ideally, two-sided, because the demarcation of space has to be nego-
tiated with citizens.
Abma and Noordegraaf:Public Managers amidst Ambiguity
293
04Noordegraaf (bc/d) 8/7/03 10:50 am Page 293
Routine Production: Legitimized Algorithms
Industrial and professional processes are characterized by a routine production
process that follows a more or less stable algorithm (an ‘if X then Y’ argumen-
tation). Under normal, stable conditions industrial processes are characterized
by a low degree of ambiguity; confusion and possible contradictions are dealt
with elsewhere, for example, in policy circles where algorithms are formulated.
Managers of industrial processes can translate these legitimized algorithms into
operational terms and organize the production process in terms of strict
procedures. The algorithms serve as constraints for practitioners (Wagenaar,
1997). The rules determine what is and what is not possible in a situation, and
they indicate how a civil servant should act (and to a lesser degree also how a
client should behave), and what are good and workable decisions and what are
unacceptable decisions. The constraints can take the form of:
formal rules and guidelines (of juridical, organizational or procedural
nature);
implicit habits, expectations or norms; and
cognitive artefacts (symbols and myths) (Scott, 1995).
The relatively low degree of ambiguity, however, does not alter the fact that
there still can be a fair amount of residual ambiguity when the seemingly uni-
interpretable algorithms are applied. Practitioners encounter concrete people
with concrete questions and problems. The stories of clients are full of details,
and it is not obvious which details are relevant and which are not relevant for
the current problem. In some situations, algorithms may conflict or they may
become unfeasible due to practical limitations (e.g. a civil servant may not be
able to visit a client at home even though a home visit is required). In positive
terms the residual ambiguity creates ‘policy freedom’ for street-level bureaucrats
(Lipsky, 1980). Practitioners who are meeting clients have to interpret and apply
the general rules in particular situations and individual cases, and this gives them
the opportunity to distinguish between cases and to honour the particulars of a
situation, and hence of the people and lives involved. Given the individualized
and emancipated nature of contemporary society, we expect that in the near
future those street-level bureaucrats will be confronted with more contradictory
claims (e.g. Vinzant and Crothers, 1998). Civil servants may find that the once-
suitable algorithms are no longer appropriate given the situations they
encounter. This creates a vacuum until new rules are developed. As we will see,
the nature of the residual ambiguity is related to questions regarding appropri-
ate indicators to decide whether or not a claim is justified and how to weight
certain extraordinary circumstances.
Professional processes are complex, but the ‘production’ is repetitive and stan-
dardized (the skills are standardized, not the outcomes) (Mintzberg, 1973).
Medical doctors may, for example, face unexpected clinical phenomena and
atypical cases, but as soon as they have formulated the diagnosis, the therapy –
in the form of ambulatory consultations or clinical operations – will follow
‘standard operating procedures’. While professional processes are complex and
potentially ambiguous, a profession reduces a great deal of the ambiguity.
Evaluation 9(3)
294
04Noordegraaf (bc/d) 8/7/03 10:50 am Page 294
Professionals specify their work, its domain in terms of product-market combi-
nations, how they conduct their work (in the healthcare sector it becomes more
and more common to develop protocols for certain illnesses), and how they make
difficult – often recurring – decisions. That explains why highly educated
professionals derive their knowledge and skills from professional networks
outside the organization in which they work.
However, there are always residual ambiguities that professionals confront
even when they apply general guidelines and protocols. Rein (1983) has charac-
terized the nature of ambiguity of professional work. He observes that
professional worries typically relate to questions about categorization (which
patients ‘fit’ in which programme/therapy?) and the large number of cases
(workload always exceeds the capacity of the relevant professional to pay atten-
tion to individual cases and to their wish to innovate practice). Professional
theories may reduce ambiguity, but these have their limitations; the abstract
nature of a theory may serve as a generalization, but it cannot explain the vari-
ation within a group of people (i.e. what makes them different from each other).
When professional theories no longer have an experiential value ‘there is still
orderly action . . . the day-to-day routine goes on. But this complex structure of
actions is partly unshielded and unjustified in an ideological and emotional sense’
(Rein, 1983: 152). Ultimately the professional is personally responsible.
No matter how large the social establishment of a profession, how moral or ideal its
purposes, how certain its basic knowledge and command of theory, ultimately the
practice of that profession comes down to one person of that profession in real time
living with other people. (Rein, 1983: 140)
Non-Routine Production: Absence of Legitimized Algorithms
At first sight, enforcing processes may seem routine. Consider sending people to
prison. Yet, behind the screen policy, conflicts linger. Another source of ambi-
guity relates to the complicity of causal links. Due to the number of players and
the influence of external developments, it is hard to attribute effects to individual
players. Improvement of safety, for example, can be related to the number of
police on the streets, but it also relates to other factors, such as urbanization,
economic circumstances, and the other players in the juridical system. This
implies that enforcing processes are characterized by conditioned ambiguity.
Ambiguity becomes controllable through the application of ‘standard operating
procedures’, but despite their intentions, almost always lead to confusion.
It is obvious that strategic processes are not routine. The production process
centres on the weighting of values, and inherently involves conflict and is ‘soft’
– contested since outputs are extremely difficult to measure other than in terms
of meetings, documents and laws. Furthermore the process is capricious, goals
are not clear at the start, because it is the process itself that generates these goals.
Incidents and crisis further fragment the process. The media and journalists also
play an influential role, because they describe and analyse the debates on the
radio, television and newspapers. As such they influence public opinion, and this
motivates key players to adjust their behaviour towards the media.
In short, strategic processes are extremely ambiguous; there is confusion and
Abma and Noordegraaf:Public Managers amidst Ambiguity
295
04Noordegraaf (bc/d) 8/7/03 10:50 am Page 295
many contradictory and incommensurable claims, requests and opinions that
cannot be prioritized by ‘objective facts’. For example, take the case of the expan-
sion of Schiphol airport (see Box 1 below) and the migration of refugees; in these
cases there are no ‘optimal’ solutions. Strategic policy processes are first and
foremost meant to reduce this ambiguity and to develop shared rules and
interpretations. These can then be translated into algorithms, which can subse-
quently be applied by operational organizations. But those algorithms are in
themselves unstable. The involvement of new actors or incidents may always lead
to a reinterpretation of the existing rules and interpretations.
To summarize, we have distinguished four public management settings with
varying levels and features of ambiguity. Industrial processes, marked by one-
sided interactions with the consumer and routine production, have the lowest
level of ambiguity. The two-sided interactions that characterize professional and
Evaluation 9(3)
296
Box 1. The Schiphol Dialogue
For several years the future of Dutch aviation has been the subject of discussion and
research. In the summer of 1995 it became clear that Schiphol – the main international
airport in Holland – would exceed agreed passenger limits much sooner than expected.
In order to open up the discussion on the issue of the benefits and necessity of expand-
ing air travel and transport, a dialogue was organized to examine this issue. After a series
of informal roundtable discussions, a broad societal process and a more formal process
took place, the latter involving representatives of 80 stakeholder organizations (includ-
ing regional and local government, the aviation and related businesses as well as environ-
mental and citizen-interest groups). Although an independent commission had evaluated
the extent to which the dialogue met a set of criteria suggested by the Government
Scientific Council, the project leader wished to begin another evaluation in order to find
out how participants had experienced the dialogue. Below is a summary of this evalu-
ation and its main findings (November 1997–June 1998).
The evaluators analysed various documents (policy and annual reports, scientific
studies, newspaper articles and opinion programmes) and conducted interviews with a
selected group of approximately 20 stakeholders. The evaluation gave a detailed insight
into the process. Participants valued the dialogue, because the dialogue stimulated a
substantial conversation among groups that were otherwise only communicating via the
media. The dialogue enhanced the mutual understanding among parties and illuminated
the various dimensions of ‘the’ problem. Gradually the focus shifted from aviation infra-
structure (how to accommodate growth) to the aviation sector and mobility in general.
According to the participants the dialogue also had an enormous impact in relational
terms: the development of new relationships, new interaction patterns and deconstruc-
tion of old stereotypes and conflicts between those in favour and those against growth.
Controversies that emerged in the evaluation related to the substantial issues discussed
(no serious attention was paid to the necessity of growth and alternatives to economic
growth according to environmental interest groups) and the conceptualization of dialogue
(instrumental versus substantial).
Source: Abma (2001c)
04Noordegraaf (bc/d) 8/7/03 10:50 am Page 296
strategic processes create high levels of ambiguity, because the product is the
outcome of a negotiation process in which clients act as co-producers. Their
expectations and interpretations influence the definition of the product and as a
result the product changes depending on the situation at hand. The non-routine
production that characterizes enforcing and strategic processes relates to value-
laden topics and power issues, and the absence of legitimized algorithms. This
generates intense levels of ambiguity. In the next section we will highlight
methodological issues relating to the key themes across the four categories.
Implications for Evaluative Practices
Our departure point is that various evaluation approaches can and will be used
across the various management settings identified, but that the balance of
approaches in different contexts will vary. If the level of ambiguity is relatively
low, such as in the case of industrial processes, a strong emphasis on perform-
ance measurement seems to be appropriate, but does not preclude the use of
supplementary qualitative approaches. The ‘appropriateness’ of a certain set of
evaluation approaches is grounded in a theoretical line of reasoning about the
nature of management settings.
4
Data: Measuring Outcomes
Performance measurement is clearly suitable in the case of industrial processes.
Industrial products are tangible and outcomes can be known and measured. To
a certain extent performance measurement is also suitable when dealing with
enforcement processes. Daily operations and activities in police stations and
prisons are measurable. We know, for example, the kind of criminal acts that the
police may encounter (robbery, assaults and offences against morality, juridical
assistance and so on). The effects are to a certain degree also known: the inci-
dence of thefts, assaults and so on can be identified, and we may ask citizens
whether or not they feel safe. In the case of war, we may monitor the process and
make up a report of the gains and losses. Parts of the enforcing processes are,
however, surrounded with ambiguity, as noticed above, and require additional
evaluation strategies to represent adequately what is happening and how this is
valued. In the case of professional and strategic processes the use of perform-
ance measurement has a more limited value. In the case of healthcare settings,
one may use epidemiological indicators such as morbidity (incidence and preva-
lence of illness among a population) and mortality rates, but the question arises
whether these are the right indicators for the process that one is evaluating.
Performance measurement in those instances is not necessarily impossible or
difficult to assess and measure, but less appropriate, given the multiple inter-
vening factors and range of values and interests at stake. Performance measure-
ment cannot take this complexity and ambiguity into account. Therefore it is not
uncommon that the application of indicators results in outcomes that are not
recognized by the participants. One should not underestimate the emotional
consequences of this lack of congruence.
Performance measurement requires the specification and operationalization of
Abma and Noordegraaf:Public Managers amidst Ambiguity
297
04Noordegraaf (bc/d) 8/7/03 10:50 am Page 297
the relevant independent (the product or treatment) and dependent variables
(the intended goals or objectives) into measurable indicators. Quantitative
methods – in the form of surveys and statistical analysis – help to measure the
factual outcomes and these should then be compared with the expected outcomes
or intended policy goals. Where standards (norms) are not available, evaluators
may also use historical analyses to compare the performance of a certain organiz-
ation or department over time. Within the private sector ‘benchmarks’ are
popular. This is a method to compare the performance of one company with the
‘best practice’ in the particular sector. Benchmarks may also be appropriate in
the evaluation of industrial processes in the public sector (Green and Anders-
son, 2002). The rankings based on these benchmarks may be generally known,
and inform the general public. Ideally they may enable customers to make delib-
erate choices.
Performance indicators, eventually in combination with historical comparisons
and benchmarks, are appropriate to evaluate industrial and enforcing processes
within the public sector. But in settings where it is less clear that the processes
being evaluated are of these types, like the distribution of benefits, we urge evalu-
ators to reconsider the appropriateness of these instruments. The residual ambi-
guity practitioners confront can be so intense that the methods just mentioned
are simply too crude to represent the plurality and complexity encountered.
Imagine, for example, prisoners having dissenting views in an evaluation of
prisons. In that case it would be inappropriate not to represent these views given
their knowledge of the setting and that their interests are at stake. The short-
comings of performance indicators, as pointed out by Jennifer Greene (1999),
become obvious in these circumstances: the practice cannot be meaningfully
defined or reduced to simple endpoints, because there are so many diverse and
sometimes contradictory endpoints or even an absence of meaning. Normatively
the use of performance indicators is then also questionable; whose values and
criteria are represented, and whose values and voices are subsequently neglected
or ignored?
We propose that in the case of contradictions or the absence of clear norms, it
should not be the evaluator who defines norms and criteria. Lack of consensus or
the absence of norms requires that those engaged in the evaluated setting set up
a democratic and dialogic process to develop norms and algorithms. Besides
policy makers in the higher policy circles, we suggest that practitioners and
citizens should also participate in these dialogic processes, because they are
experts when it comes to the various claims of individuals and because their
interests are at stake. The argument for the inclusion of these stakeholder groups
is drawn from a knowledge component (expertise) and a value component
(Greene, 1997). In order to enable public managers to talk sensibly about daily
practices and to develop realistic algorithms and norms evaluators may present
stories that represent the lived experiences of citizens and practitioners. Stories
can capture the complexity and ambiguity of the daily practice (Wagenaar, 1997;
Abma, 1999). Vicariously public managers may experience what it is like to act
in ambiguous situations. Stories give them a context to develop realistic algo-
rithms and hence evaluation criteria.
Evaluation 9(3)
298
04Noordegraaf (bc/d) 8/7/03 10:50 am Page 298
Description: Context and Process
The problem of attributing outcomes to aspects of service provision is most
obvious if legitimized algorithms are missing, such as in the case of enforcing and
strategic processes. It is then hard to find out why certain objectives and stan-
dards are not met. We distinguish between two types of failures: ‘theory failure’
and ‘program failure’ (Weiss, 1992). ‘Theory failure’ refers to the algorithms and
operational guidelines that practitioners follow. The algorithms are often based
on implicit assumptions and common understandings of policy makers, which
may be unrealistic or wrong. Theory failure can be detected by a comparison of
the algorithms in use with social scientific theories and insights. ‘Program failure’
refers to the implementation of the algorithms and the interpretation of general
rules by civil servants. Process evaluations and qualitative methods – interviews
with practitioners and participative observations – help evaluators to identify the
mechanisms that lead to ‘program failure’.
Given the attribution problem, evaluators are forced to develop sophisticated
hypotheses that include many intervening variables in order to be able to explain.
The construction of these sophisticated models requires that evaluators can rely
upon existing social scientific theories (Chen and Rossi, 1981, 1989). Instrumental
case studies (e.g. Yin, 1984) may also be helpful, because they acknowledge the
complexity of the practice and its context. When theories are missing – which is
often the case – evaluators may follow a more inductive design and qualitative
methods to explore and gain insight into the complexity of the situation. Intrin-
sic case study designs will prove to be helpful (Stake, 2000). In an intrinsic case
study the case is considered interesting in itself and not simply because it
represents a particular phenomenon. The aim is to unravel the unique charac-
teristics of the case and to do this from the perspectives of the participants. The
approach is, therefore, different from that of an instrumental case study in which
the case is treated as a means to develop or refine a theory (Yin, 1984). While
Yin (1984) proposes to study the case from the etic perspective of an outsider,
Robert Stake (2000) prefers to understand the case from the emic perspective of
the insider.
Descriptions and analyses of fundamental conflicts and dilemmas experienced
may increase public managers’ understanding, and serve as a rich source to inter-
pret and judge the data that come out of the performance measurement. These
contextual descriptions and analyses may also help practitioners to see that the
dilemmas they face are not an individual problem, but relate to the larger social
context and the debates and confusions that surround their practice. In our eyes
judgement of the situation is not a matter for the evaluator qua expert, but is a
collaborative act for the evaluator and participants (managers and practitioners).
Together they may develop ideas to improve practice.
Dialogue: Values and Power Issues
Given the complexity of professional and strategic processes, values and power
issues limit the appropriateness of performance indicators. In the case of
professional and strategic practices, and given the nature and intensity of ambi-
guity, it is more appropriate to use an evaluation approach that acknowledges
Abma and Noordegraaf:Public Managers amidst Ambiguity
299
04Noordegraaf (bc/d) 8/7/03 10:50 am Page 299
ambiguity as a departure point for a reflexive dialogue. Such an approach is sensi-
tive to the way people interpret and evaluate their practice and how their narra-
tives are developed, changed and transmitted in conversations. The political
scope of a responsive approach to evaluation is pluralist. Criteria and standards
are not solely derived from the goals, intentions and conceptions of policy
makers, but also from the values and meanings embedded in the narratives of
other actors with an interest in the evaluated practice. Methodologically this
implies that the ‘design’ is not preordained, but gradually emerges (Stake, 1975).
The evaluation is not considered a technical and analytical procedure, but a
‘dialogical’ process in which various people are approached as partners (as
opposed to information givers). Furthermore, the evaluation has a participative
or collaborative character. It is not carried out by an evaluator qua expert, but
rather in close collaboration with those who have an interest in the practice being
evaluated (Guba and Lincoln, 1989). They are involved in the evaluation process
and have a say in all the critically important stages of the evaluation as well as
in the formulation of issues and the interpretation of findings. Although this
involvement may stimulate the acceptance and utilization of findings (Greene,
1988), the main reason for stakeholder participation is political: to acknowledge
plurality and the agency of people and to make sure that no voice is marginal-
ized or excluded (Greene, 1997).
In the evaluation studies of professional processes the perspective of the client
deserves extra attention given the asymmetrical relationship and interaction
between professionals and clients (Koch, 1994, 2000; Wadsworth, 2001). Evalu-
ators should deliberately pay attention to those weak voices in order to create a
power balance and fair ‘negotiation’. The voices of clients (patients, family care
givers, school children, students and parents) are easily dismissed or ignored by
professionals, and commonly lay-people are not approached as serious partners
from whom one may learn. Evaluators should therefore actively bring in those
‘other’ voices, directly or indirectly in the form of personal stories, and invite
professionals to suspend their own judgement and to listen to those stories.
Furthermore evaluators should be aware that professionals are not used to looking
critically at each other’s work. They tend to follow a strategy of self-defensive
avoidance and non-intervention (‘if you are not critical of my work, I am not
critical of your work’) (Moen and Abma, 1992). It is therefore important to
create a safe environment where participants feel comfortable to talk openly
about their experiences. Having exchanged experiences it may become less
threatening to discuss more delicate subjects. Given the fact that professionals
are almost always overloaded with work (and some medical doctors may ‘lose’
income), time investments should be realistic. If possible, activities should be
scheduled as part of daily activities and meetings, and participants should always
receive something in ‘return’ (at least a transcript or report, but preferably a
sense of community and enhanced insights). Finally, evaluators should be aware
that professionals will only be interested in joining an evaluation process if the
issues discussed are meaningful to them and if they relate to their own concerns.
The example in Box 2 illustrates what kind of research activities a responsive
evaluation includes, and what kind of findings it may generate.
Evaluation 9(3)
300
04Noordegraaf (bc/d) 8/7/03 10:50 am Page 300
Given the intensity and nature of ambiguity of strategic processes, a respon-
sive approach – as proposed for the evaluation of professional processes – is the
most appropriate method. We will not repeat the characteristics of this approach,
but focus on the particular kinds of problems evaluators may face in these
settings.
One of the most challenging aspects of the evaluation of strategic processes is
related to the politically sensitive nature of strategic processes. The interests at
stake are often so high that the key players have the tendency to behave strate-
gically; they see the evaluation as a vehicle to bring their point of view to the
Abma and Noordegraaf:Public Managers amidst Ambiguity
301
Box 2. Evaluating Palliative Care Responsively
Palliative care is a relatively new concept that is used in connection with the integral care
provided to those who are unable to recover from their illness. The specific meaning of
the concept has not been clearly defined. What follows is a summary of a responsive
evaluation of a palliative care project in a Dutch healthcare authority (February
1997–April 1998).
The project consisted of three phases. The preparatory phase involved the collection
of a broad set of meanings:a relatively small number of interviews were carried out with
managers, professionals, clients and informal caregivers and participative observations
were made in order to gain an understanding of different view-points on the current
practice of palliative care. The field notes and interview transcripts were analysed and
presented in the form of an intermediary evaluation report. This report had the char-
acter of a ‘working document’ and acted as a vehicle in the second phase of the project.
The findings in the report were discussed in a working conference with a consultation
team (composed of an executive manager of the local hospital, a spokesperson of the
regional cancer institute and several nurses and medical doctors) and a series of story
workshops among health professionals. The goal of the second phase was to expand the
dialogue on palliative care and to engage as many participants as possible in a dialogue
on the implications and meaning of the findings for the practice of palliative care. The
third phase aimed to integrate the findings. These would then be shared with partici-
pants and others who might be interested. The conversations were transcribed, analysed
and presented in a final evaluation report. A draft version provided the input for a second
working conference with the consultation team. The final evaluation report was dissem-
inated to a wide audience of health professionals and organizations both inside and
outside the authority in order to promote an ongoing dialogue on palliative care and to
develop ways to improve the practice.
One of the most important impacts of the evaluation was that it opened up a multi-
plicity of interpretations and controversies surrounding the meaning of palliative care.
While this has not given rise to complete clarity or consensus about the meaning of
palliative care, it has nevertheless brought voices into the discourse that are often not
heard, including those of patients and informal caregivers. Health professionals became
more aware of constraining organizational conditions and possible ways of changing these
conditions. They also reflected critically on neglected themes, such as the importance of
physical expressions of affection and of love and the conflictual relationships with informal
caregivers.
Source: Abma (2001a)
04Noordegraaf (bc/d) 8/7/03 10:50 am Page 301
fore and to influence the decision-making process. This kind of behaviour is
counterproductive in the case of a dialogical evaluation; instead of a discussion
taking place in which parties exchange standpoints and arguments, a dialogue
aims to enable a conversation in which people with names and faces share experi-
ences and inquire collaboratively into the nature of ‘the problem’. For a genuine
dialogue and honest response it is important to create a safe environment and to
build trust (Abma, 2001b).
Another dilemma evaluators may face is related to the institutional setting.
Evaluation studies have more impact and are taken more seriously if they are
sanctioned by key players in higher policy circles. They might, however, not
always be willing to do so (e.g. because the approach is unknown to them or
because they do not want to share power with other stakeholders). Lack of
support from the top is in itself not a reason to cancel a responsive evaluation
study (it might in fact be the reason to conduct such an evaluation), but evalu-
ators should consider its effects. They might, for example, seek support among
key players ‘outside’ the realm of government, and use other networks and the
media to communicate their findings.
Finally we want to draw attention to the role of the evaluator who is operat-
ing in a politically laden context. Evaluation studies, of course, always take place
in a political context, but when considering strategic processes, political inten-
sity is greater. This may affect evaluators in many different ways: they may have
strong substantive opinions about the matter itself, and naturally sympathize
with one of the parties. In the case of asymmetrical power relationships, the
evaluator is particularly vulnerable. In order to create a power balance the
evaluator will deliberately try to bring in ‘marginalized voices’; this may,
however, be read by the dominant parties as over-identification, and result in
mistrust and non-co-operative behaviour. To prevent such a situation it is of
great importance to keep the process in mind, and to be reflexively aware of
one’s own convictions and loyalties (Abma, 2001b). Working in teams with
evaluators with differing opinions on the matter at hand is also a good antidote
to one-sided advocacy. See Box 1 for an example of this.
Conclusions
We started this article assuming that public managers work amidst ambiguity, and
that the nature and intensity of ambiguity vary depending on the type of process
and setting. Having distinguished and identified four different kinds of process,
we have argued that evaluative practices should be adjusted to the typical nature
and intensity of the evaluated process. In laying out this argument we have taken
a balanced view of performance measurement, and our contingency approach
differs from the positions taken in the field which can be described as a polariza-
tion between those for and those against performance measurement. The short-
comings Greene (1999) notices are evident when it concerns professional and
strategic processes, and in these cases responsive approaches that foster dialogue
are indeed more suitable. In settings characterized by a low degree of ambiguity,
such as enforcing and industrial processes, performance measurements (data) in
Evaluation 9(3)
302
04Noordegraaf (bc/d) 8/7/03 10:50 am Page 302
combination with process evaluations and case studies (contextual descriptions)
do have value (Blalock, 1999; Davies, 1999).
For the implementation of this contingency we expect that the following
barriers need to be overcome.
The trend of new public management, and the implicit belief in the ability
to control and rationalize society by means of scientific knowledge, results
in an uncritical embrace of performance indicators.
A lack of knowledge about the nature of public management settings
among evaluators and the face-validity of rational models may easily result
(inappropriately) in performance indicators being chosen.
A lack of knowledge among public managers about various evaluation
approaches may prevent them from searching for evaluation designs that
acknowledge the nature of the setting.
A fear of losing one’s identity and role may prevent managers and evalu-
ators from experimenting with approaches other than goal-based and
performance measurements. These latter approaches enable public
managers to present themselves as neutral decision makers, and emphasize
that evaluators are experts.
Public management settings are complex: in one organization we may find
more than one type of process, and it can be hard to categorize the
processes to be evaluated. In order to handle this complexity it may be very
tempting to reduce every process to an industrial process, and to use
performance indicators.
There are also several trends that may support our contingency approach.
In the field of public administration we note a transition from purely
rational models to institutional and constructionist approaches that
acknowledge the shortcomings of rational approaches of policy and
decision making.
In the field of evaluation the shortcomings of performance measurement
are becoming more known and there are several authors arguing for
additional process evaluations or alternative approaches.
Practitioners in public management are experiencing the shortcomings of
the use of performance-driven models and the underlying economic
discourse (rigidity, internal orientations etc.).
Balanced evaluation acknowledges the apparent fuzziness of public manage-
ment, and the fact that ambiguity can only be resolved following its acknow-
ledgement.
Notes
1. An earlier version of this article was presented at the 4th European Evaluation Society
Conference, Lausanne, 2000: ‘Evaluation for Public Managers’.
2. All of these ideas are neither new, nor original. Many ideas come from the private
sector – the emphasis on customer demand, on performance contracts and indicators,
Abma and Noordegraaf:Public Managers amidst Ambiguity
303
04Noordegraaf (bc/d) 8/7/03 10:50 am Page 303
on ‘lean and mean’ structures – they can be found in the business literature. The so-
called ‘sectoral structures’ resemble ‘multi divisional structure’ (Mintzberg, 1973), in
which the strategic apex sets out strategic lines and autonomous business units are
responsible for the day-to-day running of their production lines. The new public
management is a relative phenomenon; it has become popular, not because of its inno-
vativeness or originality, but because of its timing, ‘fit’ and PR.
3. In this article the term ‘algorithm’ does not only refer to a numerical order. An algo-
rithm (versus a heuristic) is understood as a rule for action that guarantees a certain
outcome.
4. As one of the reviewers rightly pointed out, normative justifications also interfere here.
The properties of each of the public management settings are not just given, but are
an outcome of interpretation and therefore are inevitably coloured by normative attri-
butions, which are ideologically and culturally loaded.
References
Abma, T. A. (1999) ‘Introduction’, in T. A. Abma (ed.) Telling Tales, On Narrative and
Evaluation, Advances in Program Evaluation 6, pp. 1–28. Stamford, CT: JAI Press.
Abma, T. A. (2001a) ‘Evaluating Palliative Care: Facilitating Reflexive Dialogues about
an Ambiguous Concept’, Medicine, Health Care and Philosophy 4(3): 261–76.
Abma, T. A., ed. (2001b) ‘Special Issue: Dialogue in Evaluation’, Evaluation 7(2): 155–280.
Abma, T. A. (2001c) ‘Narratieve infrastructuur en fixaties inbeleidsdialogen: de Schiphol-
discussie als casus’, B en M 20(2): 66–79.
Blalock, A. B. (1999) ‘Evaluation Research and the Performance Management Movement’,
Evaluation 5(2): 117–49.
Chen, H. T. and P. H. Rossi (1981) ‘The Multi-Goal Theory Driven Approach to Evalu-
ation, A Model Linking Basic and Applied Social Science’, in H. E. Freeman and M.
A. Solomon (eds) Evaluation Studies Review Annual. Beverly Hills, CA: Sage.
Chen, H. T. and P. H. Rossi (1989) ‘Issues in the Theory-driven Perspective’, Evaluation
and Program Planning 12: 299–306.
Clarke, J. and J. Newman (1997) The Managerial State. London: Sage.
Cohen, M. D. and J. G. March (1974) Leadership and Ambiguity. New York: McGraw-
Hill.
Davies, I. C. (1999) ‘Evaluation and Performance Management in Government’, Evalu-
ation 5(2): 150–9.
Forester, J. (1993) Critical Theory, Public Policy and Planning Practice: Toward Critical
Pragmatism. New York: State University of New York Press.
Green, P. and U. Andersson (2002) ‘Evaluating the Uses and Abuses of Good Examples
in Environmental Politics’, Paper presented at the 5th European Evaluation Society
conference, Seville, October 10–12.
Greene, J. C. (1988) ‘Stakeholder Participation and Utilization Program Evaluation’,
Evaluation Review 12(2): 91–116.
Greene, J. C. (1997) ‘Participatory Evolution’, in L. Mabry (ed.) Evaluation and the Post-
modern Dilemma, Advances in Program Evaluation 3, pp. 171–90. Stamford, CT: JAI
Press.
Greene, J. C. (1999) ‘The Inequality of Performance Measurements’, Evaluation 5(2):
160–72.
Grit, K. (2000) Economisering als probleem. Assen/Maastricht: Van Gorcum.
Guba, E. and Y. S. Lincoln (1989) Fourth Generation Evaluation. Beverly Hills, CA: Sage.
Hajer, M. (1993) ‘Discourse Coalitions and the Institutionalization of Practice: The Case
Evaluation 9(3)
304
04Noordegraaf (bc/d) 8/7/03 10:50 am Page 304
of the Acid Rain in Great Britain’, in F. Fisher and J. Forester (eds) The Argumenta-
tive Turn in Policy Analysis and Planning. London: UCL Press.
Hofstede, G. (1981) ‘Management Control of Public and Not-for-Profit Activities’,
Accounting, Organizations and Society 6(3): 193–211.
Hood, C. A. (1991) ‘Public Management for All Seasons?’, Public Administration 69(1):
3–19.
House, E. R. (1993) Professional Evaluation. Newbury Park, CA: Sage.
Kaufman, H. (1980) The Administrative Behavior of Federal Bureau Chiefs. Washington,
DC: Brookings Institution.
Kickert, W. J. M., ed. (1997) Public Management and Administrative Reform in Western
Europe. Cheltenham: Edward Elgar Publishing.
Koch, T. (1994) ‘Beyond Measurement: Fourth Generation Evaluation in Nursing’,
Journal of Advanced Nursing 20: 1148–55.
Koch, T. (2000) ‘ Having a Say”: Negotiation in Fourth Generation Evaluation’, Journal
of Advanced Nursing 31(1): 117–25.
Kotter, J. P. (1982) The General Managers. New York: Free Press.
Krosse, H. (1995) ‘De gemeentelijke praktijk en het Tilburgs model’, in M. Noordegraaf,
A. B. Ringeling and F. J. M. Zwetsloot (eds) De ambtenaar als publiek ondernemer,
pp. 141–8. Bussum: Coutinho.
Leeuw, F. L. (2001) ‘Evaluatie-onderzoek, auditing en controle in Nederland anno 2000’,
in T. Abma and R. in ‘t Veld (eds) Handboek beleidswetenschap [Handbook Policy
Sciences], pp. 70–82. Amsterdam: Boom.
Lips, A. M. B. (1996) Autonomie in kwaliteit, Ambiguiteit in bestuurlijke communicatie
over de ontwikkeling van kwaliteitszorg in het hoger onderwijs. Delft: Eburon.
Lipsky, M. (1980) Street-Level-Bureaucracy. New York: Russell Sage Foundation.
March, J. G. (1994) A Primer on Decision-Making. New York: Free Press.
March, J. and J. Olsen (1979) Ambiguity and Choice in Organizations. Bergen: Universi-
tyforlaget.
March, J. and J. Olsen (1989) Rediscovering Institutions. New York: Free Press.
Meyer, M. W. and V. Gupta (1994) ‘The Performance Paradox’, Research in Organiz-
ational Behaviour 16: 309–69.
Mintzberg, H. (1973) The Nature of Managerial Work. New York: Free Press.
Mintzberg, H. (1996) ‘Managing Government, Governing Management’, Harvard
Business Review 74: 75–83.
Moen, J. and T. A. Abma (1992) ‘Het ziekenhuis als een transactienetwerk van
maatschappen’, in P. de Jong, A. F. A. Korsten and J. H. van der Made (eds) Zieken-
huizen: besluitvorming en management, pp. 18–27. Assen/Maastricht: Van Gorcum.
Moore, M. (1995) Creating Public Value. Cambridge, MA: Harvard University Press.
Noordegraaf, M. (1999) ‘De overheid is geen bedrijf! Over de scheidslijn tussen publiek
en privaat’, Socialisme & Democratie 3: 179–203.
Noordegraaf, M. (2000) ‘Attention! Work and Behavior of Public Managers amidst Ambi-
guity’, dissertation. Delft: Eburon.
Noordegraaf, M. and R. Stewart (2000) ‘Managerial Behavior in Private and Public
Sectors: Distinctiveness, Disputes and Directions’, Journal of Management Studies
37(3): 427–44.
Pollitt, C. (1993) Managerialism and the Public Services. Oxford: Blackwell.
Pollitt, C. and G. Bouckaert (2000) Public Management Reform. Oxford: Oxford University
Press.
Power, M. (1999) The Audit Society. Oxford: Oxford University Press.
Rein, M. (1983) From Policy to Practice. New York: Sharpe.
Abma and Noordegraaf:Public Managers amidst Ambiguity
305
04Noordegraaf (bc/d) 8/7/03 10:50 am Page 305
Sanderson, I. (2002) ‘Does it Work? Is it “Appropriate”? Instrumental and Practical
Rationality in Evidence-Based Policymaking’, Paper presented at the 5th European
Evaluation Society Conference, Seville, October 10–12.
Scott, R. (1995) Institutions and Organizations. Thousand Oaks, CA: Sage.
Stake, R. E. (1975) ‘To Evaluate an Arts Program’, in R. E. Stake (ed.) Evaluating the
Arts in Education: A Responsive Approach, pp. 13–31. Colombus, OH: Merrill.
Stake, R. E. (2000) ‘Case Studies’, in N. Denzin and Y. S. Lincoln (eds) Handbook of
Qualitative Research, 2nd edn, pp. 435–55. Thousand Oaks, CA: Sage.
van Gunsteren, H. R. (1992) Eigentijds Burgerschap [Contemporary Citizenship], WRR
report. The Hague: SDU.
van Twist, M. J. W. and C. J. A. M. Termeer (1991) ‘Introduction to Configuration
Approach: A Process Theory for Societal Steering’, in R. in ‘t Veld, L. Schaap, C. J. A.
M. Termeer and M. J. W. van Twist (eds) Autopoiesis and Configuration Theory,
pp. 19–30. Dordrecht: Kluwer.
Vinzant, J. C. and L. Crothers (1998) Street-level Leadership: Discretion and Legitimacy
in Frontline Public Service. Washington, DC: Georgetown University Press.
Vugt, G. W. M. and P. W. Tops (1998) Zoeken naar een modern bestuur [Searching for
Modern Administration]. Alphen aan den Rijn: Samsom.
Wadsworth, Y. (2001) ‘Becoming Responsive – and Some Consequences for Evaluation
as Dialogue Across Distance’, in J. C. Greene and T. A. Abma (eds) Responsive Evalu-
ation, New Directions for Evaluation 92, pp. 45–58. San Francisco, CA: Jossey Bass.
Wagenaar, H. (1997) ‘Beleid als fictie’, Beleid en Maatschappij 24(1): 1–43.
Weick, K. E. (1979) The Social Psychology of Organizing. London: Random House.
Weick, K. E. (1995) Sensemaking in Organizations. Thousand Oaks, CA: Sage.
Weiss, C. H. (1992) Evaluation Research: Methods of Assessing Program Effectiveness.
Englewood Cliffs, NJ: Prentice-Hall
Wilson, J. Q. (1989) Bureaucracy. What Government Agencies Do and Why They Do It.
New York: Free Press.
Yin, R. (1984) Case Study Research: Design and Methods. Beverly Hills, CA: Sage.
TINEKE A. ABMA is senior researcher at the department of Healthcare Ethics
and Philosophy, University of Maastricht. Please addresss correspondence to:
Department of Healthcare Ethics and Philosophy, University of Maastricht,
Postbox 616, 6200 MD Maastricht, The Netherlands.
[email:t.abma@ zw.unimaas.nl]
MIRKO NOORDEGRAAF is Senior Researcher at the Utrecht School of
Governance (USG). Please send correspondence to:University of Utrecht,
Bijlhouwerstraat 6, 3511 ZC Utrecht, The Netherlands.
[email:m.noordegraaf@usg.uu.nl]
Evaluation 9(3)
306
04Noordegraaf (bc/d) 8/7/03 10:50 am Page 306
... On the basis of the definition of ambiguity of the administrative activities proposed by Abma and Noordegraaf (2003), based on the level of routine of the administrative action and on the importance of the interactions with subjects that are external to the production process, this study has two objectives: first, to check whether, on the basis of the data collected, the type of activity measured is relevant in determining the ability of the measurement system to frame performance; second, on the basis of the empirical analysis, to formulate some considerations that are useful to raise awareness of the implementation of PM in the activities of the public administrations. In other words, this article intends to explore the connection between the characteristics of administration and PM in Italian ministries in order to draw useful considerations in relation to the use of PM systems in public administrations. ...
... Finally, transition activities are in an intermediate situation between these two cases and refer to "many new and unknown elements of standards that are subject to discussion". Abma and Noordegraaf (2003) present a similar reasoning starting from the consideration of the ambiguity that characterises the activities (productive process) carried out by public administrations. ...
... On the basis of what has been illustrated in Section 2, the expectation is that, in the presence of a drive for the introduction and development of PM systems in the public administration, with the same factors favouring the development of such systems being equal, some activities are measurable in terms of performance while others are measurable to a lesser extent. In this regard, on the basis of the classification of Abma and Noordegraaf (2003), a distinction is made between highly ambiguous and less ambiguous activities. Highly ambiguous activities are those characterised by two-side interactions between producers and consumers, where customers and citizens are coproducers and actively influence the service and the nature of the routine production process. ...
Full-text available
Article
Following the spread of performance measurement (PM) in public organisations, a vast literature has developed on its functioning and difficulties. This paper intends to investigate the relationship between administration activities, which can by their nature have different characteristics, and PM systems. The application of PM in the Italian ministries through the analysis of 1,511 objectives/indicators shows the relevance of administrative activities in influencing the ability to measure performance and suggests that neglecting this link is detrimental to the evaluation system and undermines its ability to provide a reliable representation of the contribution of public organisation in relation to performance. © 2018, © 2018 The Author(s). This open access article is distributed under a Creative Commons Attribution (CC-BY) 4.0 license.
... Osborne et al., 2015), professionals are expected to negotiate both the process and its outcomes with client co-producers as part of their work routines. Moreover, strategic processes, such as neighborhood development activities, include negotiations among many different stakeholders, such as citizens, communities and private sector actors (Abma & Noordegraaf, 2003;Kleinhans, 2017). In a manner typical of human interactions, these negotiations include much complexity. ...
... Traditionally, professionals are considered to solve such complex situations by leaning on their professional expertise (Lipsky, 1980;Abma & Noordegraaf, 2003;Krogstrup & Brix, 2019). While professionals certainly have opportunities to use their expertise in such situations, the knowledge hierarchy between professional and the client is changing from a top-down, one-directional relationship to a collaborative relationship based on user empowerment and interdependence (Ewert & Evers, 2012;Bovaird, 2007;Moynihan & Thomas, 2013). ...
Chapter
This chapter builds a conceptual model for how inter-organizational relationships can be built to enable the creation of learning across administrative and organizational boundaries. The conceptual model is discussed in relation to the body of knowledge concerning co-production and the new roles required of organizational members and frontline staff when services cut across these boundaries. The argument is that it is becoming increasingly important for professional co-producers and their organizations to identify, analyze, and improve the opportunity space for co-production when this opportunity space unfolds beyond one organization.
... Tem-se assistido a uma crescente preocupação sobre como é que os resultados da avaliação podem contribuir como evidência para a tomada de decisões (Weiss, 1979), ao mesmo tempo que inúmeros trabalhos de investigação tendem a sugerir que, frequentemente, as avaliações não são utilizadas (Henry e Mark, 2003;Hofstetter e Alkin, 2003) e que essa utilização pode ser apenas simbólica, nomeadamente para legitimar um programa e a sua teoria (Alkin e Taut, 2002). Mesmo quando as avaliações têm grande qualidade, não há garantia de que os seus resultados sejam utilizados (Dahler-Larsen, 1998;Abma e Noordegraaf, 2003). ...
Full-text available
Thesis
This thesis analyses the institutionalization process of evaluation in the Portuguese development cooperation, between 1994 (the year in which the evaluation was integrated in the cooperation agency) and 2012 (the year in which IPAD merged with the Camões Institute). The key research question is: how the institutionalization of evaluation took place in the Portuguese development cooperation? To answer this question, a literature review and a desk analysis have been conducted. A survey and several semi-structured interviews with Portuguese cooperation actors have also been carried out. Based on the policy transfer theory, it starts from the hypothesis of an incomplete transfer, driven by external actors. Although there is a broad research on evaluation use, there is a research gap regarding the Portuguese reality. There is also no research based on policy transfer theory. This research seeks to contribute to fill these gaps, developing a model that identifies the factors that influence the institutionalization of evaluation in the Portuguese development cooperation. The results show that, despite the internal and, above all, the external determinants, the nature of the policy, the organizational/institutional context and the evaluation system adopted had an influence on the institutionalisation of the evaluation, in particular its use in the decision-making process. The results of this research are a contribution to the understanding of the institutionalization of evaluation in development cooperation organizations and a basis for future research. The conclusions can provide evidence to the cooperation professionals, guide the evaluation practice and promote its use in the decision-making process.
... This in turn implies that professionals, are in need of different skills. After all, these professionals are to work alongside co-producers daily as well as represent and/or report to different stakeholders, such as citizens, communities and private sector actors (Abma & Noordegraaf, 2003;Tuurnas, 2016b). Within this field of community development, in which co-producers were in essence vulnerable citizens, these street-level professionals have become experts in undervalued skills such as being an empathic communicators, lending an ear or simply being a trustworthy partner in order to successfully co-produce with this citizen group (Currie et al., 2009;Needham & Mangan, 2014;O'Leary et al., 2012)The co-production process can thus be seen as an iterative process as the professional is needed to lower the threshold to the co-production project while the citizens are needed to keep the co-production project running. ...
... Each issue has different meaning for different stakeholders; our search was focused on generating these meanings and starting a dialogue about these meanings to enhance the personal and mutual understanding of participants. Our research was therefore deliberately focused on the polyvocality and the process of sensemaking itself (Abma & Noordegraaf, 2003). To understand the complexity of psychiatric emergency care, we needed to include various perspectives and remain open to its meanings. ...
Full-text available
Article
Descriptive studies on the experiences of service users in psychiatric emergency wards are increasing. However, the experience of users throughout the whole psychiatric emergency procedure, the ‘patient journey’ from the moment of mental health crisis leading to admission in a psychiatric inpatient unit, has rarely been studied. This article aims to make a further contribution to this body of knowledge by not only describing the service-users’ journey but also shows whether and how that perspective is received by other stakeholders like healthcare professionals, family-members, police, representatives of the municipality and ambulance staff and representatives of the municipality, to jointly make action-plans for improvement. A responsive evaluation including interviews, focus groups and dialogue session with a transformative aim was conducted in a mixed team of researchers with and without experiential knowledge. Service users and other stakeholders agree on the main issues: the importance of contact; signaling; expropriation and earning freedom. However, proposed actions differed between users and professionals. Service users proposed relational actions to offer good care. Professionals recommended actions in response to issues of fragmentation and discontinuity. We conclude that responsive evaluation can help to create a more informed and kaleidoscopic view of the complexity of psychiatric emergency care. Therefore, we recommend that various actions need to be considered to meet the needs of service users better covering all phases of Tronto’s care ethical model: caring about, taking care, caring giving and care receiving.
... Evaluators have argued that educational accountability systems typically represent a narrow set of decontextualized outcomes and are not meant to be responsive to dynamic schooling contexts (Greene, 1999;Ryan, 2002). Finally, Abma and Noordegraaf (2003) argue that certain public sectors may benefi t from performance measurement systems, or accountability systems, more than others. Performance measurement may be less suited for management when services depend on interactions with participants and are not routine tasks, which is characteristic of school administrators. ...
... A similar trend has been spotted in several public sector domains in Sweden, as well as in other countries (e.g., Hagerman, 2014;Levin & Fullan, 2008;Rapp, 2010;Wolmesjö, 2008). New public management (NPM) ideals may be one reason for the increase of administrative tasks linked to accountability and the continuously scrutinizing of processes, improvements, performance, results, costs, and customer satisfaction (Abma & Noordegraaf, 2003;Pollitt & Bouckaert, 2000) at the same time as the direct administrative support often has decreased (Arman et al., 2009;Höckertin, 2007). Although a high workload can constitute a positive challenge in a job (Crawford et al., 2010), work overload has been associated with ill-health (e.g., Schaufeli & Bakker, 2004). ...
Full-text available
Article
Managers’ psychosocial working conditions are important for managerial sustainability in the public sector. The job demands-resources (JD-R) model is a widely applied and well-recognized framework for measuring psychosocial working conditions. However, there is still a need for methodological contributions including more objective as well as qualitative ways to assess these conditions. In this study, job demands and job resources as well as the balance between them was qualitatively and externally assessed for first-line human service managers using a work content analysis method. Conditions and actions were focused upon with an external perspective. Special attention was paid to concrete examples and consequences of work characteristics with predefined criteria and cut-off points to guide the assessments. The results reveal an imbalance for human service managers between high levels of job demands and the lack of job resources available to meet these demands. Work overload, conflicting and unclear goals and tasks, emotional demands, restricted control, and lack of supervisory and organizational support generally characterized the managerial assignment. The analysis provided concrete explanations of the current work strain in this group of employees, thereby giving both short-term and long-term possibilities for improvement of managerial work and sustainability.
Full-text available
Article
This study undertakes an in-depth and rigorous exploration and explanation of the sources and implications of ambiguity in performance measurement systems and performance management practices (PMM) systems of complex multi-stakeholder organisations. In doing so, it contributes to the literature on performance measurement theories. The study is undertaken in the form of a fine-grained, inductive case study with the Child and Adolescent Mental Health Services (CAMHS) of the National Health Service of England. Data are obtained from multiple exploratory interviews with CAMHS stakeholders. The study surfaces factors that create ambiguities in the PMM practices of complex multi-stakeholder organisations, which interact in complex ways, raising questions over the value of PMM systems and practices. The originality of the paper is threefold. First, it opens a new area of debate in relation to performance measurement in complex multi-stakeholder organisations. Second, the findings demonstrate the complex interrelationships between the sources and manifestations of ambiguity advance the knowledge of PMM systems and implications in such organisations. Third, the findings reveal that the nature of complex multi-stakeholder organisations suppresses open, participative and inclusive social controls.
Book
This wide-ranging edited volume provides a state of the art account of theory and research on modern street-level bureaucracy, gathering internationally acclaimed scholars to address the varying roles of public officials who fulfill their tasks while interacting with the public. These roles include the delivery of benefits and services, the regulation of social and economic behavior, and the expression and maintenance of public values. Questions about the extent of discretionary autonomy and the feasibility of hierarchical control are discussed in depth, with suggestions made for the further development of research in this field. Hence the book fills an important gap in the literature on public policy delivery, making it a valuable text for students and researchers of public policy, public administration and public management.
Article
A source of one of the more serious problems of evaluation research is the typical finding that evaluated programs have little or no effectiveness. Recently, claims have been made that the problem of no effect may be attributed to the inadequacy of current evaluation methodology. In this paper, the reasons for no effect in current program evaluation are analyzed theoretically. In order to cope with this problem, an alternative evaluation method, the “multi-goal, theory-driven” approach, is proposed. The principles, procedures, and justification of this new evaluation approach are discussed. Finally, the advantages of applying the multi-goal, theory-driven approach to program evaluation are illustrated.