The Public Value of Citizens’ Initiatives: Evidence from a Dutch Municipality

Abstract

Active citizens who take initiative are generally regarded as desirable. However, the precise reasons why citizens’ initiatives are considered valuable and what their value consists of remain unclear, vague, and often unanswered. In this study, we used Q methodology to explore how civil servants, local politicians, and societal actors in a Dutch municipality view the public value of citizens’ initiatives. The analysis reveals four distinct views of the value of citizens’ initiatives: a view that values intangible results, a view that values a hands-on mentality, a view that values acting out of a sense of purpose, and a view that values citizens organizing and acting out of their own interests. Theoretically, we distinguish between material, immaterial, and process-oriented interpretations of values, and empirically this distinction shows that across the four value views, the process-oriented values are the most disagreed upon. Finally, we find common ground between the value views that we label “selfish collectivism.” This is the view that appreciates citizens’ initiatives for solving problems for the sake of the community, not for their altruism, but because they are self-serving. The strong differences in value views suggest that there is a risk that subsequent policy language and instruments based on these views could lead to conflict between the actors involved.

Full text

The Public Value of Citizens’Initiatives:
Evidence from a Dutch Municipality
Steven Blok
1
, A. W. van Buuren
1
, and H. J. M. Fenger
1
Abstract
Active citizens who take initiative are generally regarded as desirable. However, the precise reasons why citizens’initiatives are
considered valuable and what their value consists of remain unclear, vague, and often unanswered. In this study, we used Q
methodology to explore how civil servants, local politicians, and societal actors in a Dutch municipality view the public value
of citizens’initiatives. The analysis reveals four distinct views of the value of citizens’initiatives: a view that values intangible
results, a view that values a hands-on mentality, a view that values acting out of a sense of purpose, and a view that values
citizens organizing and acting out of their own interests. Theoretically, we distinguish between material, immaterial, and pro-
cess-oriented interpretations of values, and empirically this distinction shows that across the four value views, the process-
oriented values are the most disagreed upon. Finally, we find common ground between the value views that we label “selfish
collectivism.”This is the view that appreciates citizens’initiatives for solving problems for the sake of the community, not for
their altruism, but because they are self-serving. The strong differences in value views suggest that there is a risk that subse-
quent policy language and instruments based on these views could lead to conflict between the actors involved.
Keywords
citizens’initiatives, civil society, self-organization, public value, q methodology
Expectations of Citizens’Initiatives
Citizens are asked to participate in a wide range of policy pro-
cesses (Healey, 2015; Uitermark, 2015; Verhoeven &
Tonkens, 2013; van de Wijdeven, 2012) and self-organizing
citizens’initiatives are counted on to produce public services
in a wide variety of domains. Citizens’initiatives or
community-based initiatives also contribute to public life
and generally have positive effects on society (Hurenkamp
et al., 2006, p. 10 & p. 20; Boonstra & Boelens, 2011,
p. 117; Healey, 2015, p. 116).
Citizens’initiatives are viewed by local governments as
promising vehicles to deliver solutions to a broad range of
public problems. These initiatives are often expected to
reduce costs for implementing policy programs, sometimes
because they are “believed to empower and educate citizens
and reduce the reliance of individuals and social organiza-
tions on state bureaucracies”(Bakker et al., 2012., p. 396,
see also Edwards, 2009 about civil society expectations).
Local governments want to include citizens in both policy-
making and its implementation, or at times, want citizens to
take care of themselves as self-sufficient and self-organizing
beings (Hurenkamp & Tonkens, 2011, p. 27). Active citizen-
ship is approached by governments to provide a solution to
different kinds of societal puzzles. These range from provid-
ing libraries, playgrounds, and kindergartens, to correcting a
deficiency of social cohesion, consumerism and asocial
behavior, social exclusion, or to bridging the increasing
gap between government and society (Tonkens, 2006, p. 5;
Uitermark, 2015, p. 2311).
The “promise”of citizens’initiatives is sometimes
labeled public value (Bryson et al., 2013, p. 25; Brandsen
et al., 2017, p. 682; Duijn & Popering-Verkerk, 2018).
However, what this value entails is often fuzzy (e.g., van
de Wijdeven, 2012). Examples of describing these values
are varied:
•citizens produce “publically valued goods and services in
non-state and non-market ways”(Healey, 2015, p. 116),
•citizens’initiatives contribute to various societal values
(Mattijssen et al., 2015, p. 85),
•citizens’engagement is necessary for better results and
because of moral necessity (Hurenkamp and Tonkens,
2011, p. 27), or
•citizens’initiatives have a favorable effect on the local
community (Hurenkamp et al., 2006, p. 57).
1
Department of Public Administration and Sociology, Erasmus University
Rotterdam, Rotterdam, the Netherlands
Corresponding Author:
Steven Blok, Department of Public Administration and Sociology, Erasmus
University Rotterdam, Postbus 1738, 3000 DR Rotterdam, the Netherlands.
Email: blok@essb.eur.nl
Article
American Review of Public Administration
1–17
© The Author(s) 2023
Article reuse guidelines:
sagepub.com/journals-permissions
DOI: 10.1177/02750740231175162
journals.sagepub.com/home/arp

The specific ideas about the public value that citizens’initia-
tives produce are oftentimes vague while the expectations by
local governments are high. We expect differences because
initiators or participants themselves may have other views
on why citizens’initiatives are more valuable than those of
policy-makers or politicians (van Straalen et al., 2017). If
this expectation is true and there is indeed a mismatch in
what is perceived as the value of a citizen initiative, such
ambiguity can help in understanding why the interaction
between governments and citizen initiatives is often problem-
atic. Governments may expect too much of citizens’initia-
tives, even demanding too much of them, resulting in
disappointment by both initiators and governments. Or gov-
ernments may create a suboptimal institutional context in
terms of rules, enabling conditions and policy instruments
which are not in agreement with the motives and ambitions
of the initiative itself (Streeck & Schmitter, 1985, p. 122).
Some overlap or the complementarity between perceptions
and ideas of public value by citizen initiatives is likely to
be important for successful coproduction. Therefore, we
propose to investigate the following question: To what
extent do ideas of the public value(s) of citizens’initiatives
differ between local politicians, civil servants and societal
actors?
With this study, we analyze whether there is such a differ-
ence and, if so, what that possible difference could look like.
In doing so, we provide some clarifications within the public
value debate (Alford & Hughes, 2008; Hartley et al., 2017;
Prebble, 2018). Responding to Stoker’s (2006) argument
that public value has a highly contextual character, this
research creates a specific contextual setting to determine
what is valued by the public: the value of citizens’initiatives
within a local context. In other words, our research adds to
the understanding of the question of what “a public
values”instead of what “the public values”(Benington,
2009, p. 233; Hartley et al., 2017, p. 672). If there is a percep-
tion gap between the initiators of citizens’initiatives and
local governments of what public values are, unraveling
this gap may help to revisit the current ‘ways of working’
between these two. Consequently, clarifying the current
working relationship, including the usual incentives to
support (Blok et al., 2022) and the institutional arrangements
between citizens’initiatives and local governments, may
promote productive interaction.
Theoretical Literature: Public Value and
Categorizing Outcomes
Two types of literature are discussed: the literature that helps
to understand the value that citizens’initiatives could lead to
and the literature that helps to further distinguish different
kinds of valuable outcomes. We start with the former by pro-
viding three lenses that provide a perspective on why citi-
zens’initiatives could be valuable.
Public value is a broadly coined concept that is also used to
capture some of the trends surrounding self-organization (e.g.,
Mattijssen et al., 2015, who speak of “creating value with the
public”; Duijn & Popering-Verkerk, 2018). Public value is pri-
marily known through Moore’s book Creating Public Value
(1995). But public value is a complicated concept for at least
two reasons. The firstreasonisthatMoore’s book “has the
term ‘public value’in its title, but the book is really more
about quality public management and presents no stable
concept of public value”(Jørgensen & Bozeman, 2007,
p. 357). Therefore, the absence of a sound definition of
public value, while the concept is massively used
1
, is a pro-
found problem (ibid., but see also Alford & O’Flynn, 2009;
Bozeman, 2002; Overeem & Tholen, 2011). For example,
the concept is used to make a point about innovation in the
public sector, but, although fulfilling an important part of the
authors’arguments, “what the value actually is”is often
taken for granted (e.g., Hartley, 2005). Rhodes and Wanna
state (2007, pp. 408 & 419): “Moore is unclear whether he
offers a new theoretical framework, a concept, a heuristic
device, or an operational tool of management.”The second
reason is summarized by Stoker (2006, p. 50): “The concept
of public value does appear to have a decidedly context-
dependent character”which makes it difficult to use to define
valuable outcomes in general (see also Alford & O’Flynn,
2009, p. 176). Anything can be valuable given “the right”
context.
The public value paradigm, framework or theory offers, so
far, little if any help in defining what makes citizens’initia-
tives valuable. The premise on which the public value dis-
course is based—value—could be of use. As Alford and
O’Flynn (2009, p. 175) note about value: “that property of
a thing because of which it is esteemed, desirable or useful;
worth, merit or importance.”Similar definitions arise from
Meynhardt (2009, p. 197).
In sum, someone must find value in something that citi-
zens’initiatives produce. The question remains: who is to
do the valuing? Alford and Hughes (2008) argue that
public value is valuable to whoever consumes it.
Self-organizing citizens often produce valued goods or ser-
vices that they use themselves as well. In that sense, from
the public value perspective, citizens’initiatives are valuable
because they do something that initiators or participants
themselves see as desirable (and because they use it them-
selves). Moreover, as argued in the introduction, public
authorities value citizens because they produce desirable
goods and services. There are multiple actors who may be
suited to do the valuing (see also Hartley et al., 2017, p. 674).
The second lens is about how value is created.
Self-organized citizens’initiatives are a form of collective
action: (Bakker et al., 2012; Blok et al., 2020). When citizens
successfully organize and produce “something,”they over-
come the collective action problem (Ostrom et al., 1994).
Succeeding in collective action can be seen as valuable
because it signals the ability to cooperate. Thus, citizens’
2American Review of Public Administration 0(0)

initiatives are seen as valuable because they are a form of col-
lective action or because they show evidence that its prob-
lems, as described by Olson (1965), can be overcome.
The third and final lens is value expressed by concepts such
as civil society, social cohesion, and social capital. These con-
cepts are prominent cornerstones in Putnam’swork(2001,
p. 22): bonding social capital, as he labels it, is “good for under-
girding specific reciprocity and mobilizing solidarity.”In line
with this, bonding social capital is associated with trust and
citizen participation, a combination that enables the production
of socially desirable outcomes (ibid., p. 288). Social capital,
Putnam states in the end of Bowling Alone, makes us smarter,
healthier, safer, richer, and better able to govern a just and
stable democracy, (ibid., p. 290). Social capital “is presumably
what produces a dense civil society”(Fukuyama, 2000, p. 7; see
also Walzer, 1991). Civil society encompasses all social rela-
tions between citizens (Bryant, 1993) and has a dense
network of civil association (Foley & Edwards, 1996, p. 38).
As such, it is also seen as necessary for fruitful and spontaneous
self-organization (Bryant, 1993, p. 399) and a healthy public
sphere (Diamond, 1994; Edwards, 2009). Castells (2008,
p. 76) argues that civil society is the place where various
views are expressed and that a good relationship between the
state and civil society is the cornerstone of democracy (see
also Keane & Merkel, 2019). Self-organizing citizens’initia-
tives are valuable because they can be seen as an expression
of “the civil society.”Within this civil society, citizens deliber-
ate about common affairs and act and organize independently of
the state.
However, making distinctions between these last two
lenses might lead to some confusion. Citizens’initiatives
are manifestations of cooperative behavior (lens two) and,
therefore, naturally connected to concepts such as social
cohesion and civil society (lens three). As such, we feel it
is necessary to elaborate more on these distinctions.
Following Edwards (2009) position on this matter, distinc-
tions should be made between civil society or collective
action as a means to get somewhere on the one hand and
as a vision of “the good society”on the other. In this research,
we view civil society (that one wants to create) and social
capital (which is something one possesses) as valuable ends.
These ends could be achieved through collective action by
citizen’s initiatives, but this is not necessarily the case. Civil
society can also be created by healthy states, well-functioning
markets, and non-associated citizens. Simultaneously, a vivid
associational life full of initiatives is hardly sufficient in itself
to make a society civil (Edwards, 2009, pp. 44–48).
Next, for the second type of literature, we identified three
fruitful starting points to better understand the public value(s)
of citizens’initiatives: first, distinguishing between internal
and external; second, conceptualizing different levels of out-
comes; and third, defining public value in terms of material,
non-material, and process outcomes.
Igalla and colleagues (2019) make the distinction between
internal and external outcomes. External outcomes are
observable outside the initiative; they are the produced
goods. They state that citizens’initiatives “provide and main-
tain an alternative form of traditional governmental public
services, facilities, and/or goods themselves, such as water
distribution, education and training, and residential care”
(ibid., p. 7). In terms of external outcomes, the authors find
that, according to the literature, most initiatives contribute
to social improvements of the community, economic
improvements of the community, physical improvements of
the community, and environmental improvements. In terms
of internal outcomes, they refer to organizational aspects
that the initiatives realize for the initiative itself, such as dura-
bility, legitimacy, and organizational outcomes (ibid., p. 10).
Another distinction in outcomes is made by van de
Wijdeven (2012, p. 257). He differentiates between the out-
comes in terms of the questions of “what does it yield,”
“what does it bring about”and “what does it bring forth”?
Van de Wijdeven answers these questions sequentially: the
first with “the clean park,”then, with “sociality and self-
confidence,”and finally, with “citizenship.”His distinction
is primarily aimed at how the impact of outcomes is experi-
enced by those who are involved.
Van der Torre et al. (2019) studied public value for “The
Association of Netherlands Municipalities.”As a starting
point, they used three categories of values (ibid., p. 5) that
are also useful for this research: material values (of which
lower taxes or higher employment opportunities are exam-
ples); immaterial values (of which happiness or a sense of
community are examples); and process values (of which
co-creating a space for citizens, making decisions transpar-
ently, or anchoring in democratic processes are examples).
Ultimately public value, social capital, and collective
action are well-connecting concepts (Benington, 2009). The
public value of citizens’initiatives is related to its external
goods, also to its internal goods, and to the process of how
citizens’initiatives do things. Our framework captures
these distinctions by the previous authors in combination
with the three lenses on valuable outcomes that we have
just discussed. In this way, we can distinguish between
these values:
•The material values: refer to the external goods of citi-
zens’initiatives (Igalla et al., 2019) or the “what does it
yield-question”by van de Wijdeven (2012). Material
values are ends and may also encompass the more instru-
mental benefits of self-organizing citizens’initiatives (see
e.g., Eriksson, 2012, p. 688).
•The immaterial values: are ends that refer to the outcomes
that are good for citizens and society (Tonkens, 2006; van
Dam et al., 2015). This relates to the ‘what does it brings
about-question’(van de Wijdeven, 2012). Self-reliance,
civil society, and mutual relating concepts, such as social-
ity (Walzer, 1991) or social capital (Putnam, 2001), are in
this category.
Blok et al. 3

•The process values: are somewhat difficult to separate
definitively from the immaterial values. We follow our
framework by pointing to collective action, deliberation
(Bryant, 1993, p. 356; Bohman, 2000, pp. 55–56), or inter-
active problem-solving as examples of process values. The
process values are related to the valuable aspects of how
citizens’initiatives do things (and what results from the
“act of doing,”such as acquiring a skill or the ability to
work better together)
2
and in that sense process values
are categorically more like means.
Next, we discuss the ground for expecting differences in the
perspectives of public value between different actors sur-
rounding citizens’initiatives.
The Tensive Relationship Between the
Government and Initiatives
In many countries collectives of citizens who engage with
public problems are becoming increasingly important
(Baum, 2001; Bailey, 2012; Healey, 2015; Henriksen et al.,
2018). There are multiple reasons why governments are
partial to citizens’initiatives. Firstly, most ideas refer to
“the right behavior”of citizens (Boonstra & Boelens, 2011,
p. 101). For example, citizens should be self-reliant
because it is good for them and good for society (Tonkens,
2006, p. 9; van Dam et al., 2015, pp. 165 & 175).
Arguments often start with stating that citizens rely too
much on the government and, furthermore, that they
behave like consumers (Kleistra, 2019; van Rooij et al.,
2019, p. 429). The Dutch government stated in 2013 that
“the goal is to activate as many citizens as possible to take
on local activities”(Ministry of the Interior and Kingdom
Relations, 2013). Secondly, active citizenship is necessary
for a well-functioning democracy associated with natural
communal activities and self-organization and is therefore a
crucial building block for civil society (Eriksson, 2012,
p. 687). Within the Dutch context of this research, the gov-
ernment has set up multiple efforts to increase the number
of citizens participating in society or to support, facilitate,
and even entice citizens to participate and organize them-
selves (Ministry of the Interior and Kingdom Relations,
2018; van Dam et al., 2015, p. 164).
Both of these views held by the government illustrate that
there are high expectations about citizens’initiatives. These
high expectations by political parties and civil servants
have also been found in previous studies (Den Ouden
et al., 2019; de Haan et al., 2018, p. 313; who, for the
record, also found some lower expectations of civil servants).
In addition, social funds and housing corporations have laid
their eye on citizens’initiatives (Denters et al., 2013). To
illustrate the point of having unreasonably high expectations,
we quote Edwards (2009, p. 16) about associations in the
civil society, which we believe also applies to citizens’
initiatives:
“There is equal danger in expecting too much from associational
life, as if it were a ‘magic bullet’for resolving the intractable
social, economic and political problems (…). It seems, [they]
are expected to organize social services, govern local communi-
ties, solve the unemployment problem, save the environment,
and still have time left over for rebuilding the moral life of
nations.”
These expectations make governments eager to help (Den
Ouden et al., 2019, p. 51). However, with the best intentions,
governmental involvement can have adverse effects (Vollan,
2008, p. 563). Brandsen and colleagues (2017, p. 676) warn
that local governments may take over citizens’initiatives,
resulting in a manufactured civil society instead of a spontane-
ous one. In such a situation, the character and value for which
citizens’initiatives are valued are crowded out. Moreover,
while intended as supporting, governmental attitudes are some-
times described as controlling (Streeck & Schmitter, 1985,
p. 122; Benington, 2009, p. 242), and citizens’initiatives some-
times encounter this controlling tendency (Boonstra & Boelens,
2011, p. 107; van Straalen et al., 2017).
In conclusion, governments have high expectations about
citizens’initiatives and the public value they create. But these
expectations may be too high, and, moreover, we seriously
doubt that all expectations are shared by those involved in cit-
izens’initiatives. This mismatch of expectations may conse-
quently result in wrong or suboptimal policies and
inappropriate instruments aimed at inviting or supporting cit-
izens’initiatives. As such, we find it crucial to understand
more precisely what the perspectives of all the actors are
and how they differ.
Research Design
We used Q methodology (Brown, 1993; McKeown &
Thomas, 1988; Stephenson, 1953; Watts & Stenner, 2005),
which has quantitative features, such as factor analysis to
order and to couple certain qualitative points of view.
“Fundamentally, Q methodology provides a foundation for
the systematic study of subjectivity”(Brown, 1993, p. 93).
The purpose of this study is to discover how different
actors, or different ‘publics’if you will (Benington, 2009),
view the public value(s) of citizens’initiatives. In a sense
we are exploring the “contests, debates and arguments
between and among different [actors] about what constitutes
public value”(Hartley et al., 2017, p. 674). And because
values “express subjectivity”(Meynhardt, 2009, p. 199), a
systematic study of subjectivity fits the purpose because the
participants in this study decide what they think is valuable
about citizens’initiatives (Watts & Stenner, 2005, p. 69).
To successfully execute the Q study, the following proce-
dure was applied: (1) making a Q sample of statements by
sampling a topic such as the public value of citizens’initia-
tives; (2) selecting the P set, i.e., the participants or respon-
dents; (3) having the respondents sort the sample of
4American Review of Public Administration 0(0)

statements, their Q sort; followed by (4) analyzing the data
through factor analysis, and finally, (5) interpreting the anal-
ysis (Brown, 1993; McKeown & Thomas, 2013; Watts &
Stenner, 2005). The study was done in a medium-sized
municipality (approximately 57.000 inhabitants in 2021) in
the Netherlands. The municipality was merged in 2019 and
contains 21 villages. Given the village-like character and
the possibly close-knit community there, there may be a rel-
atively strong tendency to value things such as social
cohesion.
The Q Sample: Public Value Statements
The primary concern of constructing the Q sample is making
it generally representative (Watts and Stenner, 2005, p. 75).
Collecting the samples can be done either in an unstructured
or in a structured way. And as structured samples are drawn
based on theoretical arguments (McKeown & Thomas, 2013,
p. 23), or, more specifically, the theoretical framework, we
chose to do the latter. A total of 179 statements was
reduced to 36 statements (12 for each value). First, we cate-
gorized the statements on face value, which led to roughly 25
categories of statements for each type of value. Next we
further compressed the statements by finding overlaps
between the categories until a total of 12 statements for
each type of value remained. One of the authors scraped
the Dutch ‘Coalition Agreements’by city councils of 2018
on citizens’initiatives (and similar labels). This led to an
inventory of 31 municipalities profiling themselves with
high ambitions for citizens’initiatives (all but one explicitly
and actively wanted to support citizens’initiatives). None of
the municipalities explicitly mentioned citizens’initiatives
before the merger. In the new coalition agreement, the munic-
ipality is quite vocal about citizens’initiatives. Examples are
that they “really want to give space,”that they see “citizens’
initiatives as leading in how the municipality deals with its
community”and that they want to “stimulate, encourage
and facilitate”citizens’initiatives. The inventory of coalition
agreements was then used to check whether the 36 statement
also represented these coalition agreements. Finally, we dis-
cussed the statements with a panel of eight experts ranging
from science and practice, to local and national government.
We created a Q sample from the following sources (see
appendix 1 for an overview):
•Dutch commissioned research from practice: for example,
a ministry commissioned a series of five research projects
on citizens’initiatives and another ministry commissioned
a research report on the possibilities for the Right to
Challenge;
•Dutch scientific research that is often used by researchers
or practitioners;
•International scientific research that is often used by
researchers or practitioners;
•Other semi-public research or pieces about citizens’initia-
tives, such as documents by Courts of Audits
[Rekenkamercommissie], the Ombudsman, or the Dutch
information center for democracy, “ProDemos.”
It was already remarkable how most of the documents were
aimed at stimulating citizens’initiatives or describing indi-
cators that increase the success of citizens’initiatives
without explaining what it is that is valuable or successful
in terms of citizens’initiatives.
3
Many of the documents
also have a rather instrumental vision of citizens’initia-
tives. For example, Movisie (2011) presents an overview
of interventions to stimulate active citizenship, and a com-
mittee of a local Court of Audit stresses the need for citi-
zens’initiators to work with other actors to realize their
goals (instead of working well together internally, i.e., col-
lective action).
The P set: Selecting Participants
The selection of actors who will do the “valuing”has partly
been theoretically presorted (McKeown & Thomas, 2013,
p. 31) because we expect various social actors to have differ-
ent views. Therefore, we included local politicians, civil ser-
vants, and societal actors, including initiators. Together with
the municipality, we selected a diverse set of participants
within these groups, for example, local politicians from dif-
ferent political parties and civil servants ranging from the
city manager to street-level bureaucrats. As for citizens, we
included initiators and non-initiators and they were all
active and societally-involved respondents (from schools,
the library or the ‘village council’) who could be approached
via the municipality. With this step in the Q methodology, we
tried to refrain from including only enthusiasts (Fiorina 1999,
p. 402).
With this P set, we figuratively “created well-informed
publics”—again, not the public—to express ideas about the
public value of citizens’initiatives (Benington, 2009,
p. 232; Meynhardt, 2009, p. 206).
The Q Sort and Interview: Ordering Statements and
Expressing Ideas
Our sample of 22 respondents ranked the 36 statements
according to a forced-choice format (appendix 2). The state-
ments were ranked on a normal distribution from –4to+4,
which is appropriate for Q samples smaller than N=40
(McKeown & Thomas, 2013, p. 29). Respondents ranked
the statements from ‘least in accordance’to “most in accor-
dance.”This range indicates that a score (or position on the
format) is always to some extent in accordance.
Due to the covid pandemic, the interviews took place in
the summer of 2021 via MS Teams and Miro (an online
whiteboard) and the statements were posted on digital
post-its. The Q interviews typically lasted 50 −70 min. A
Blok et al. 5

few questions were asked before the sort: e.g., “what is your
background?”and “how do you define citizens’initiatives?”
And a few after the sort: “why did you put these statements at
the extremes?”,“did you miss specific statements?”;“do you
think citizens’initiatives can do the majority of public tasks
better than governments?”;“do you think citizens’initiatives
should always be financially supported?”; and “what are the
downsides with and problems for citizens’initiatives?”The
answers to these questions and the additional explanations
to the rankings were written down by the researchers,
leading to 22 research notes of approximately 1,000 words
a piece.
Analysis and Results
We analyzed the Q sorts with KADE (Banasick, 2019), an
open-source application for the analysis of Q study data.
First, the correlations of all respondents were calculated,
resulting in a .22 correlation between all the sorts, indicating
that the respondents were a rather heterogenous group.
Second, the Principal Components Analysis led to 8 factors
of which 7 had an Eigenvalue higher than 1. We reviewed
the analysis and noticed that, with 5 factors or higher, only
single respondents represented those factors. Therefore, we
continued with 4 factors having an explained variance of
57% (the "composites" of these 4 factors are in appendix 3).
Third, factors were rotated using Varimax.
Table 1 shows the various respondents (categorized by
types) and how they “loaded”on the factors. Respondents
can load negatively (–1 to 0) or positively (0 to +1) on the
factors. Next, we interpreted each of the factors based on
the characterizing statements, i.e., those statements with a
score of ±3, 4 or the distinguishing statements (these are
“significantly”different for one factor compared to the
others), and the explanations of the respondents.
Factor 1: Idealistic Immaterialists
Factor 1 has 8 significant loadings: 6 civil servants and 2
local politicians. Factor 1 opines that citizens’initiatives
are intrinsically valuable. That citizens’initiatives contribute
to the involvement in the community, liveability, and solid-
arity/community spirit are expressions of that intrinsic
value. Respondents who associate strongly with this factor
value citizens’initiatives because they are viewed as a
form of reassurance that people are still willing to do some-
thing for their community and the public cause.
Factor 1 is also rather idealistic in the sense that it values
the immaterial rather than the material. All the statements that
the participants did not agree with are material values;
Table 1. Correlation Coefficient Between Respondents and Factors.
Resp. # Respondent type Factor 1 Factor 2 Factor 3 Factor 4
9 Civil servant (project office) 0,7415* −0,0284 0,3009 0,2046
12 Civil servant (housing) 0,7051* 0,0365 −0,114 0,0975
5 Politician (liberal) 0,6928* 0,2847 0,0857 0,1553
3 Civil servant (communications) 0,6886* 0,2853 0,1197 −0,033
6 Civil servant (social domain) 0,6634* −0,0365 0,1544 0,2226
18 Civil servant (clerk) 0,6589* −0,1262 −0,4021 0,3353
4 Civil servant (legal) 0,6516* 0,2092 −0,0663 0,1095
21 Politician (local party) 0,524* 0,3809 −0,3476 0,1315
17 Politician (local party) 0,0194 0,6737* −0,2058 0,2065
15 Societal actor (village council and business association) −0,0112 0,6514* 0,0511 −0,1869
22 Politician (Christian democrat) 0,1393 0,5222* 0,079 0,1369
1 Societal actor (teacher and board member school) 0,0146 0,5134* 0,3495 0,3306
16 Civil servant (field service) 0,4284 0,4692* 0,004 0,467
10 Societal actor (village council) 0,3361 0,4079* 0,065 −0,0114
14 Societal actor (Fortress Governor and initiator) −0,1117 −0,1015 0,8176* 0,0973
2 Societal actor (initiator and member foundation) 0,2617 0,4191 0,5367* −0,3943
11 Civil servant (city management) 0,3117 0,2235 0,5307* 0,1087
20 Societal actor (director of a societal organization) −0,0514 0,1817 0,0972 0,797*
19 Civil servant (strategic) 0,3809 0,1387 −0,1122 0,7317*
13 Civil servant (communications) 0,3194 0,224 −0,2984 0,7014*
7 Societal actor (volunteer) 0,213 −0,0851 0,2229 0,5992*
8 Societal actor (librarian) 0,5181 −0,227 0,2094 0,5198*
*Are significant factor loadings.
At the p < .05 is one that exceeds 1.96(1/√36) =±0,33. Significance the p < .01 exceeds 2.58(1/√36) =±0,43 (see Watts and Stenner, 2005, pp. 87–88). All
significant factor loadings also exceed Humphrey’s rule (Brown, 1980, p. 223): multiplying the highest factor loadings should exceed twice the standard error
(SE) =.32 (The SE in this study is 1√[number of statements], 1√36 =.16).
6American Review of Public Administration 0(0)

furthermore, their idealism is visible in the statement quotes.
Citizens’initiatives should—in principle—not be valued for
the savings they might realize or as replacements for the gov-
ernment’s role in major maintenance projects. In other words,
the real value of citizens’initiatives is not as instruments to
realize output. Another respondent also highlights how
“feeling involved”is what matters for citizens’initiatives,
and anything else is secondary (Table 2).
Factor 2: Pragmatic Partners
Factor 2 has 6 significant loadings: 3 societal actors, 2 local
politicians, and 1 civil servant. This factor is called pragmatic
because it conveys a certain kind of hands-on-mentality. Not
the kind that leads to all sorts of material output, but the kind
that leads to liveability and a better relationship between the
municipality and the community. The statement ‘CI’s solve
problems that they themselves find important’also ranked
+3 on this factor.
The combination of taking action when necessary as initiators
is central to this factor and also central for nurturing collabora-
tion/partnership between the municipality and the community.
The interaction is necessary to get acquainted and to strengthen
cooperation/partnership between governments and initiators. In
addition to Table 3, other quotes illustrate this position:
‘The relationship really improves by talking to each other. (…)
And it is good to listen to initiatives because then you notice
what is happening locally.’(16)
This factor is also characterized by some sort of sobriety. One
should not overexaggerate the value of citizens’initiatives.
They are not training grounds for better behavior, personal
development, or mutual relationships. They also do not con-
tribute to “the economy,”nor do they create value for the
entire society. Pragmatic partners want no nonsense and
view citizens’initiatives as a way to force governments and
initiatives to partner up.
Table 2. Statements with Highest/Lowest Scores for Factor 1.
Rank Sig Type Statement/quote (by respondent)
Agree +4 ** P 25 CI’s are valuable in themselves.
+3* I 15CI’s contribute to community involvement.
+3 ** I 17 CI’s contribute to liveability in the neighbourhood.
Disagree −4 M 10 CI’s create savings because they take over tasks from the government.
−3M4CI’s contribute to the economy.
−3M6CI’s ensure ‘major maintenance’in the neighbourhood.
Notable −2* P29CI’s contribute to the self-reliance of the participants.
Quotes “People are –still –willing to do something, which is why citizens’initiatives are so valuable.”(6)
“The value of citizens’initiatives lies in those side effects: how one feels. Those are positive.”(9)
“It’s not about money in my view. Money should not, in principle, be ‘the reason.’” (3)
“You shouldn’t do it for savings because it demotivates. Then you are immediately watching the money again
and steering accordingly.”(5)
Distinguishing statement at * p <.05, ** p < .01. Types in the table are Pfor process values, Ifor immaterial values and Mfor material value. Citizens’iniatiatives
are abbreviated as CI’s in the tables.
Table 3. Statements with Highest/Lowest Scores for Factor 2.
Rank Sig Type Statement/quote (by respondent)
Agree +4 ** P 27 CI’s contribute to the ability to work together.
+3 ** I 17 CI’s contribute to liveability in the neighbourhood.
+3 ** I 14 CI’s contribute to a better relationship between the government and residents.
Disagree −4∼P32CI’s contribute to altruistic/selfless behaviour.
−3 ** I 13 CI’s create societal value for the entire society.
−3 I 21 CI’s contribute to personal development.
Quotes “Collaboration between the government, residents, entrepreneurs and village councils is very important and it
will get much better.”(15)
“That’s what it’s about. You often see situations of ‘it is so bad if something happens to it or not’and then
people take action and it is solved.”(10)
“It doesn’t have to yield anything. Citizen initiatives often cost a lot. (…) Finally, many things are difficult or
undesirable to calculate in economic terms, such as quality of life!”(1)
Blok et al. 7

Factor 3: Personal Purpose
Factor 3 has 3 significant loadings: 2 societal actors and 1
civil servant. This third perspective is characterized by a
mix of immaterial and material values. Factor 3 encom-
passes elements of both meaning and fun because taking
initiative provides both. Furthermore, they find meaning
in engaging with important subjects such as ecology and
sustainability.
But factor 3 is also quite unique by disagreeing strongly
with three process-type values. Citizens’initiatives do not
contribute to democracy, inclusivity, or altruistic behaviour.
Respondent 11 summarized this view:
“When people are involved in citizens’initiatives, it really con-
tributes to their sense of purpose. That is not selfless. They do it
because they find it important themselves or because they want
to achieve something. Or because it’s fun. And that’sfine. But
that’s not altruism and it doesn’t have to be.“(11)
This factor perceives citizens’initiatives a little bit in the
“bonding-social-capital-approach”as it sees initiatives as ori-
ented on themselves. Citizens’initiatives are tightly knit—there-
fore, they are also quite exclusive toward outsiders—and it is
important for the in-group to search for the important things
in life and to engage with them (14) (Table 4).
Factor 4: Democratic Developers
Factor 4 has 5 significant loadings: 3 societal actors and 2
civil servants. This final factor sees citizens’initiatives as
boosters for democracy. But there’s a catch: citizens’initia-
tives are not democratic per se, but they strengthen democ-
racy by advocating for and acting on their interest. In that
sense, they increase the democratic influence of participants
and increase the quality of policy decisions. It is expected,
but nevertheless, legitimate, that citizens’initiatives act—
perhaps egoistically—out of their own interest. No person
can be truly altruistic and, moreover, democracy needs the
articulation of interests and action of collectives. It is
valuable and important that citizens’initiatives take responsi-
bility and ‘solve the problems that they themselves find
important.’
Factor4alsovaluescitizens’initiatives because they are
viewed as self-determined and constructive. Protest is seen
both as “being against something”or as encouraging
others to do something rather than taking action oneself.
Citizens’initiatives should also—just as the idealistic imma-
terialists believe—not be viewed too instrumentally. They
are not to be seen as producers of public goods, a savings
opportunity, or as replacements for the municipal field
service (Table 5).
Similarities and differences
In the introduction, we argued that it is extremely important
that a specific agreement or complementarity between the
perceptions and ideas of public value as produced by
citizen initiatives is to result in successful coproduction. To
understand this (lack of) overlap and complementarity, we
make a few comparisons.
In table 6, the correlations between factors are presented
(1 is high, 0 is low). The correlations show that especially
the factor of personal purpose seems to have the most dis-
agreements with the other factors. There also seems to be a
considerable correlation between the idealistic immaterialists
and the democratic developers, indicating that a consensus
about the value of citizens’initiatives could be reached
easier between these two factors. For example, both factors
show disagreement on the statements that citizens’initiatives
contribute to sustainability or ensure “major maintenance”in
the neighborhood. Both factors also show relatively strong
agreement with statements that citizens’initiatives solve
problems they themselves find important or that they increase
control of and influence over the participants.
There are three statements about citizens’initiatives that
lead to the most disagreements (in appendix 4 we discover
some more differences):
Table 4. Statements with Highest/Lowest Scores for Factor 3.
Rank Sig Type Statement/quote (by respondent)
Agree +4 ** I 20 CI’s contribute to the meaning of the participants.
+3** I 7CI’s contribute to sustainability.
+3** M 3CI’s ensure ecological value and nature.
Disagree −4* P33CI’s strengthen democracy
−3 ** P 31 CI’s contribute to an inclusive society.
−3 P 32 CI’s contribute to altruistic/selfless behaviour
Quotes of interviews “We all think it’s very important, we enjoy doing it, and everyone gets a lot out of it.”(2)
“Democracy, that’s way too big. Municipal councils and citizens’initiatives are sometimes at
odds with each other.”(11)
“There is a split between ‘old’groups and newcomers here (…). So, it is quite exclusive.“(14)
8American Review of Public Administration 0(0)

•They strengthen democracy
•They are valuable in themselves
•They contribute to meaning-making for participants
These disagreements are salient because political or govern-
mental discourse on citizens’initiatives is often related to
democracy (see van Dam et al., 2015).
In general, this Q study shows that the conflicting statements
are generally process statements, and the most agreeable state-
ments are generally material and immaterial statements. It
seems that the material value of citizens’initiatives—within
this study context—are the least disputed.
4
In the end, all
views combine multiple types with what they agree or disagree
on. The idealistic immaterialists, however, strongly agree on
immaterial statements (that’s no surprise), and this factor also
relates quite well to the second lens (citizens’initiatives as
expressions of the civil society and “what does it bring
forth?”question). The personal purpose type, on the other
hand, strongly disagrees with the process-type statements.
The statement “citizens’initiatives solve problems they them-
selves find important”was the most agreed upon with scores of
+2and+3 by all factors. And the statement “Citizens’initiatives
contribute to altruistic/selfless behaviour”was disagreed upon
by all factors with scores of either −3or−4. Two other state-
ments that the factors generally agree on is that citizens’initia-
tives do not necessarily create safety nor innovation. But
generally, this “selfish collectivism”(self-selection of problems
and non-altruïsm) is common ground across all the value views.
Finally, while it is not at the heart of Q studies to compare
the respondents (rather, it is about their views), we were still
interested to—albeit prudently—compare the social actors.
Figure 1 shows how our actors, on average, relate to the
four value views. Societal actors share the view of “personal
purpose”the most and “idealistic immaterialists”the least.
Civil servants mostly share the view of “idealistic immateri-
alists”and least share the view of “personal purpose.”Local
politicians share the view of “pragmatic partners”the most
and of “personal purpose”the least. This comparison is a
first glimpse, and again such a comparison is not at the
heart of Q studies, so this result should be considered
cautiously.
Our results show that there are some shared aspects across
the views, for example, selfish collectivism, that could be
common ground for fruitful coproduction. At the same time,
as Figure 1 illustrates, there are not only differences between
views, but also meaningful disagreements between views
and, likely, between social actors. To illustrate this point
imagine a city council that shares a vision and sets budgets
in line with the pragmatic partners, and civil servants who
(having some discretionary room) write/employ policy lan-
guage and act in line with the immaterial idealists. While the
initiators and participants themselves need support to meet
Table 5. Statements with Highest/Lowest Scores for Factor 4.
Rank Sig Type Statement/quote (by respondent)
Agree +4 ** P 33 CI’s strengthen democracy.
+3 P 28 CI’s increase the control/influence of the participants.
+3M5CI’s contribute to the quality of policy decisions.
Disagree −4 ** P 34 CI’s ensure protest about what is wrong.
−3 M 10 CI’s create savings because they take over tasks from the government.
−3M1CI’s ensuring collective/public goods for all.
Notable −2** M 2CI’s contribute to the practical management and refurbishment of things.
Quotes of
interviews
“Especially because I believe that taking initiative and taking action is a social decision. It is about
determining in your own living environment.”(19)
“Control, certainly. [Local] Politics is absolutely sensitive to citizens’initiatives. And she should be!”
(8)
“You don’t achieve much with protest in itself. You mainly encourage others to do something while
you are not doing it yourself.”(7)
“I think that basically people also come into citizens’initiatives out of an interest that they also have
themselves. (…) Altruism is wonderful, but at its core it must/needs to arise from altruistic
motives. (19)
Table 6. Correlation Coefficient Between Respondents and Factors.
Idealistic immaterialists Pragmatic partners Personal purpose Democratic developers
Idealistic immaterialists 1 0,3674 0,053 0,4968
Pragmatic Partners 0,3674 1 0,1129 0,3628
Personal purpose 0,053 0,1129 1 0,0013
Democratic developers 0,4968 0,3628 0,0013 1
Blok et al. 9

their needs in line with personal purpose.Suchascenario
would lead to strain, frustration, and would reduce motivation.
Conclusion and Discussion
In the introduction and theoretical sections, we problematized
the possible misalignment of values between governments
and citizens’initiatives. The vast majority of information
that we analyzed to help us collect statements for the Q inter-
views, more or less took the value of citizens’initiatives for
granted. The majority of the research or reports was con-
cerned with increasing the conditions for the success of citi-
zens’initiatives. It was all about pushing the buttons or
changing the levers in order to “systematically increase out-
comes”regardless of what the outcomes are (MacIntyre,
2011, p. 90). But exactly what the value of citizen’s initia-
tives is, was often taken for granted, unspecified, and/or
accepted as the one thing that they “just create.”The value
perspective was not questioned. Citizens’initiatives are
approached as “initiatives-as-they-happen-to-be”and facili-
tated, upgraded, and governed. But if the values are examined
more closely, there’s a whole world behind them, and our
research was able to show that this world contains a plurality
of values.
By studying different types of social actors within a
medium-sized Dutch municipality using Q methodology,
we found four different views connected to the values of cit-
izens’initiatives. These four views all have different config-
urations of how they define what is valuable about citizens’
initiatives. But there are also similarities. All four views
value citizens’initiatives because of their self-determination.
Various actors value citizens taking action and doing some-
thing instead of waiting for officials to step in. This relates
to other findings (Edelenbos et al., 2018, p. 61) where it is
concluded that citizens’initiatives take action because of
some trigger. Another interesting similarity among the
views is that citizens’initiatives are not necessarily valuable
because they “contribute to selfless behavior.”This view
subscribes to the idea that it is logical and acceptable that cit-
izens’initiatives are (partly) driven by individual or group
interests. We labelled the common ground between value
views “selfish collectivism.”
Figure 1. Average correlations of actors and value views.
10 American Review of Public Administration 0(0)

One of the theoretical lenses that helped us to examine our
research question was about distinguishing types of values.
We found that process-type statements created the most dis-
agreements. Statements expressing the idea that citizens’ini-
tiatives are valuable in themselves or strengthen democracy
were especially prone to grave disagreements. In contrast,
the material values are, relatively, the least prone to disagree-
ments (because most disagreed with these statements).
As for how the different actors viewed the value of citi-
zens’initiatives, we did a more in-depth study towards how
the different views of the values concerning citizens’initia-
tives take shape. We did, however, also take a look at how
the various types of actors/respondents connect to the four
views. While this type of study is not definitive in claiming
that these differences are significant—these findings should
be interpreted cautiously—it does suggest that there may be
meaningful differences between actors in value views about
citizens’initiatives that may hinder fruitful collaboration.
This is especially the case when it comes to creating the
right “invited space”or funding schemes for citizens’
initiatives.
Next, we set out a few implications based on our study
and findings. Given that governments are fond of citizens’
initiatives (Boonstra & Boelens, 2011; Bakker et al.,
2012; Brandsen et al., 2017; Hurenkamp & Tonkens,
2011) and that they likely have specificvalue views about
citizens’initiatives or specific goals (such as strengthening
democracy), tensions could definitely arise. As citizens’ini-
tiatives may be shaped by the perceived expectations and
goals of governments (van Dam et al., 2015, p. 174), they
could be subjected, controlled, and governed in a way that
does not align with their value views about themselves.
“This governmental control of organizations can be
achieved through at least four mechanisms: (…) [such as]
the use of symbolic policy language”(Brandsen et al.,
2017, p. 684). We have already demonstrated how a mis-
match between a council’s language, policy language, and
needs of initiators can exist. And there is still the risk that
citizens’initiatives can be seen as magic bullets (Edwards,
2009). We therefore argue that it is crucial to first discover
the relevant value patterns within a certain context before
creating various arrangements.
In this study, we set a specificcontext(amedium
sized Dutch municipality) in which we let various
respondents express their value-views of citizens’initia-
tives. This approach which fits exceptionally well the
studying of public value(s) (Meynhardt, 2009, p. 199),
even in municipalities with different characteristics. We
started with the argument that what people value about
citizens’initiatives is often somewhat fuzzy, unclear, or
that it is just “a heap of good things.”We believe that
with this study, we made the discourse about the value
of citizens’initiatives clearer and more comprehensible
and that we have discovered distinctive value patterns.
Q methodology is indeed, as suggested by Hartley and
colleagues (2017), suited to trace and map values. Our
final contribution is to show that it indeed matters to
explore the value views of citizens’initiatives and be
specific about them. As “public value can only exist in
a coproduction between government, citizens, associa-
tions, entrepreneurs and firms”(Brandsen et al., 2017,
p. 682) it seems wise in these interdependent coproduc-
tions to understand each other’s value views before
action is taken.
This study has limitations and we discuss the most
important ones. First, this study is mainly about citizens’
initiatives with some government involvement. There are
citizens’initiatives that operate almost completely indepen-
dently of governments (although they must abide by the
law). These initiatives are naturally less affected by high
or specific expectations from governments. Second, we
did not quantitively examine significant differences in
views between groups. In order to further inquire the possi-
ble tensions and problems we described, such a study
would be very helpful. Third, studying values is a difficult
exercise. With our distinctions between type of goods and
type of values, we made the exercise somewhat less diffi-
cult. But these distinctions are not perfect. We acknowledge
that some statements could have fitted in other types of
value-views, depending on how they are interpretated.
Another weakness is that Q study does well when the state-
ments are unambiguous and when not open to too many
different interpretations from participants. We tried to
design such statements, but straightforward statements can
also flatten a complex discourse, such as the one about
values. Nevertheless, there is little, or even no room, for
conditional viewpoints about a statement because it needs
to be ranked. Working with these statements also leaves
less room for the sedimentation in values: the potential
future spinoff in values or the changes in values over
time. Finally, as the Q methodology is not yet mainstream
(but becoming more en vogue), some would also criticize
this method; for example, there is the criticism that any
sample of statements is prone to selection bias by the
researchers. From a Q-point-of-view, we argue that it is
not so much about “the total meaning”of the Q sample,
but how meaning is given by respondents to “the Q
sample”; that is the heart of the method (Brown, 1993,
p. 101).
Of course, there are multiple promising options for further
research. First, it would be useful to do this Q study in other
municipalities, regions, or provinces to see how meaning is
given to the value of citizens’initiatives elsewhere. It is
also promising to zoom in on various actor groups: What
kind of value views characterize different groups such as
civil servants or local politicians, and which value differences
exist within these groups? In this study we more a less treated
these actor groups as homogeneous, not acknowledging nor
inquiring about the differences within these groups. Finally,
a more quantitative approach—perhaps based on the
Blok et al. 11

statements or the four views in this study or future studies –to explore the various value views of citizens’initiatives of a rep-
resentative sample of the whole population would be insightful. Such an approach would greatly increase our knowledge about
how various actors perceive the value of self-organization and about the possible problems and tensions set out in this piece.
Appendices
Appendix 1: Used literature and checked coalition agreements for the concourse
Author Type
Bailey, N. (2012). The role, organisation and contribution of community enterprise to urban
regeneration policy in the UK. Progress in Planning, 77(1), 1-35.
International Scientific
Igalla, M., Edelenbos, J., & van Meerkerk, I. (2019). Citizens in action, what do they
accomplish? A systematic literature review of citizen initiatives, their main characteristics,
outcomes, and factors. Voluntas: International Journal of Voluntary and Nonprofit
Organizations, 30(5), 1176-1194.
International Scientific
Van Dam, R., Duineveld, M., & During, R. (2015). Delineating active citizenship: The
subjectification of citizens’initiatives. Journal of Environmental Policy & Planning, 17(2),
163-179.
International Scientific
Bakker, J., Denters, B., Oude Vrielink, M., & Klok, P. J. (2012). Citizens’initiatives: How local
governments fill their facilitative role. Local Government Studies, 38(4), 395-414.
International Scientific
Healey, P. (2015). Citizen-generated local development initiative: recent English experience.
International Journal of Urban Sciences, 19(2), 109-118.
International Scientific
de Haan, E., Meier, S., Haartsen, T., & Strijker, D. (2018). Defining ‘Success’of Local Citizens’
Initiatives in Maintaining Public Services in Rural Areas: A Professional’s Perspective.
Sociologia Ruralis, 58(2), 312-330.
International Scientific
Verhoeven, I., & Tonkens, E. (2013). Talking active citizenship: Framing welfare state reform in
England and the Netherlands. Social Policy and Society, 12(3), 415.
International Scientific
Bakker, J., Denters, S. A., & Klok, P. J. (2011). Welke burger telt mee (r) in de
doe-democratie?. B en M: tijdschrift voor beleid, politiek en maatschappij, 38(4), 402-418.
Dutch Scientific
Vrielink, M. O., & Verhoeven, I. (2011). Burgerinitiatieven en de bescheiden overheid. Beleid
en Maatschappij, 38(4), 377-387.
Dutch Scientific
Van de Wijdeven, T. M. F., De Graaf, L. J., & Hendriks, F. (2013). Actief burgerschap: lijnen in
de literatuur.
Dutch Scientific
Igalla, M., & Van Meerkerk, I. (2015). De Duurzaamheid van Burgerinitiatieven. Een
Empirische Verkenning (The Sustainability of Citizen Initiatives. An Empirical Exploration).
Een Empirische Verkenning (The Sustainability of Citizen Initiatives. An Empirical
Exploration), 25-53.
Dutch Scientific
Salverda, I., & Van Dam, R. (2008). Burgers en landschap deel 1: Voorbeelden van
burgerparticipatie en maatschappelijk initiatief. Wageningen: Alterra Wageningen UR.
Practical/commisioned Research
Van Dam, R., During, R., & Salverda, I. (2008). Burgers en Landschap deel 2: Trends en
theorieën over betrokkenheid van burgers. Wageningen: Alterra Wageningen UR.
Practical/commisioned Research
Dam, R. V., Salverda, I., & During, R. (2010). Burgers en Landschap deel 3: Strategieën van
burgerinitiatieven. Wageningen: Alterra Wageningen UR.
Practical/commisioned Research
Duineveld, M., van Dam, R. I., During, R, & van der Zande, A. N. (2010). Burgers en
Landschap deel 4: The importance of being nimby; Een essay over burgerverzet en
erfgoed. Wageningen: Alterra Wageningen UR.
Practical/commisioned Research
Van Dam, R. I., Salverda, I. E., & During,R. (2011).). Burgers en Landschap deel 5: Effecten van
burgerinitiatieven en de rol van de rijksoverheid. Wageningen: Alterra Wageningen UR.
Practical/commisioned Research
Denters, S. A. H., Tonkens, E. H., Verhoeven, I., & Bakker, J. H. M. (2013). Burgers maken hun
buurt. Platform31.
Practical/commisioned Research
ProDemos (2015). Doe-democratie. Tips voor raadsleden, wethouders, ambtenaren en
burgers. Available via: https://www.prodemos.nl/wp-content/uploads/2017/12/ProDemos-
Doe-democratie_brochure.pdf
Practical Literature
Ombudsman (2018). Burgerinitiatief: waar een wil is…. Onderzoek naar de rol van Practical Literature
(continued)
12 American Review of Public Administration 0(0)

(continued)
Author Type
overheidsinstanties bij burgerinitiatieven. Available via: https://www.nationaleombudsman.
nl/system/files/bijlage/Burgerinitiatieven%20waar%20een%20wil%20is%20rapport%
202018-020.pdf
Movisie (2011). Actief burgerschap. Een overzicht aan interventies. Available via: https://
www.movisie.nl/sites/movisie.nl/files/publication-attachment/Verkenning%20Actief%
20Burgerschap%20%5BMOV-177889-0.4%5D.pdf
Practical Literature
Ouden W. den, Boogaard G. & Driessen E.M.M.A. (2019), Right to Challenge. Een studie naar
de mogelijkheden voor aan algemene regeling voor het ‘Right to Challenge’en andere
burgerinitiatieven in Nederland [eindrapport]. Leiden: Universiteit Leiden.
Practical/commisioned Research
Rekenkamercommissie Wassenaar, Voorschoten, Oegstgeest, Leidschendam-Voorburg
(2018). De burger neemt het initiatief. Via: https://kennisopenbaarbestuur.nl/media/
255725/rapport_onderzoek_burgerinitiatieven.pdf
Practical Research
Ministerie van BZK (2020). Lokale democratie: tentoonstelling over de ontwikkeling van de
lokale democratie na 1945. Nota Doe-democratie. Available via: https://
kennisopenbaarbestuur.nl/het-geheugen-van-bzk/lokale-democratie/tijdlijn-lokale-
democratie-4/2010-heden/2013-nota-doe-democratie/
Practical literature
Ministerie van BZK (2013). De Doe-Democratie. Kabinetsnota ter stimuleren van een vitale
samenleving. Available via: https://www.rijksoverheid.nl/documenten/publicaties/2013/07/
09/kabinetsnota-doe-democratie
Practical literature
Buurtwijs. (2019). Wat maakt burgerinitiatieven duurzaam?. Available via: https://www.
buurtwijs.nl/content/wat-maakt-burgerinitiatieven-duurzaam
Practical literature
Universiteit van de Humanistiek & Movisie. (2018). Niet meer zomaar te stoppen: Over
duurzaamheid van burgerinitiatieven Available via: https://www.movisie.nl/sites/movisie.nl/
files/2019-01/Niet-meer-zomaar-te-stoppen-duurzaamheid-burgerinitiatieven.pdf
Practical research
van Delden, P. (2020). Sociaal offensief tussen lockdowns door. Sociale vraagstukken.
Available via: https://www.socialevraagstukken.nl/sociaal-offensief-tussen-lockdowns-
door/
Practical Literature
Visser, V. (2019). Lager opgeleiden voelen zich te min om te participeren in
burgerinitiatieven. Sociale vraagstukken. Available via: https://www.socialevraagstukken.nl/
lager-opgeleiden-voelen-zich-te-min-om-te-participeren-in-burgerinitiatieven/
Practical Literature
de Moor, T. (2020). Meer ruimte voor de commons’’ Sociale vraagstukken. Available via:
https://www.socialevraagstukken.nl/meer-ruimte-voor-de-commons/
Practical Literature
Salverda, I., van Dam, R. & Pleijtje, M. (2017). Pionieren: De impact van innovatieve
maatschappelijke initiatieven op een natuur-inclusieve samenleving. Wageningen
University Research. Available via: https://edepot.wur.nl/425310
Dutch Scientific
Vullings, L.A.E., A.E. Buijs, J.L.M. Donders, D.A. Kamphorst, H. Kramer & S. de Vries (2018).
Monitoring Green Citizens’Initiatives; Methodology, indicators and results of a pilot
project and baseline assessment. Wageningen, Statutory Research Tasks Unit for Nature &
the Environment, WUR. WOt-technical report 125
Dutch Scientific
Coalition agreements scanned of municipalities:
Eijsden-Margraten Leidschendam-Voorburg Hollands Kroon Soest
Achtkarspelen Leudal Zaanstad Voorschoten
Amersfoort Maasgouw Zuidplas Weesp
Asten Weststellingwerf Hilversum Utrecht
Bunschoten Enkhuizen De Wolden Westerveld
Doesburg Goes Houten Woudenberg
Horst aan de Maas Gooise Meren Oegstgeest Zwartewaterland
Leiderdorp Bernheze Rijswijk
Blok et al. 13

Appendix 2: Q sort format
Appendix 3: Composites: views, statements and positions
36
35
32 33 34
29 30 31
24 25 26 27 28
17 18 19 20 21 22 23
10 11 12 13 14 15 16
1 23456789
Least in accordance Most in accordance
−4−3−2−10+1+2+3+4
# Statement 1234
1CI’s ensuring collective/public goods for all. 0 −1−3−3
2CI’s contribute to the practical management and refurbishment of things. 0 0 1 −2
3CI’s ensure ecological value and nature. −203−1
4CI’s contribute to the economy −3−3−1−1
5CI’s contribute to the quality of policy decisions. 0 1 −23
6CI’s ensure ‘major maintenance’in the neighbourhood. −311−2
7CI’s contribute to sustainability. −2−13−2
8CI’create mainly nice extras in the neighborhood. 1 1 1 −2
9CI’s solve problems they find important themselves. 2 3 2 2
10 CI’s create savings because they take over tasks from the government. −400−3
11 CI’s contribute to the public cause. 2 −1−10
12 CI’s contribute to policy implementation. −10−10
13 CI’s create societal value for the entire society. 1 −302
14 CI’s contribute to a better relationship between the government and residents. 1 3 0 1
15 CI’s contribute to community involvement. 3 2 0 1
16 CI’s create a sense of responsibility among its participants. 1 2 0 2
17 CI’s contribute to liveability in the neighbourhood. 3 3 0 1
18 CI’s lead to (more) innovation. 01−10
19 CI’s ensure more mutual relationships between the participants. 0 −231
20 CI’s contribute to the meaning of the participants. −1−140
21 CI’s contribute to personal development. −2−31−1
22 CI’s contribute to safety in the neighbourhood. −10−1−1
23 CI’s lead to a sense of community/belonging among its participants. 3 0 1 1
24 CI’s make sure that society –aside from the government and the market –has more influence on the public space. 1 1 2 3
25 CI’s are valuable in themselves. 4 −12−1
26 CI’s increase diversity of contacts and identities. 0 2 −20
27 CI’s contribute to the ability to work together. 0 4 0 1
28 CI’s increase the control/influence of the participants. 2 2 −13
29 CI’s contribute to the self-reliance of the participants. −2002
30 CI’s increases societal skills among its participants. −1120
31 CI’s contribute to an inclusive society. 0 −1−3−1
32 CI’s contribute to altruistic/selfless behaviour −3−4−3−3
33 33 CI’s strengthen democracy 1 −2−44
34 CI’s ensure protest about what is wrong. −1−21−4
35 CI’s contribute to the development of (public) opinions among its participants. −1−2−20
36 CI’s lead to a change of how governments work. 2 0 −20
14 American Review of Public Administration 0(0)

Appendix 4: Differences in most disagreed statements across factors
Acknowledgments
The authors thank the anonymous reviewers for their constructive
comments on the submitted manuscript.
Declaration of Conflicting Interests
The author(s) declared no potential conflicts of interest with respect
to the research, authorship, and/or publication of this article.
Funding
The author(s) received no financial support for the research, author-
ship, and/or publication of this article.
ORCID iD
Steven Blok https://orcid.org/0000-0003-1535-1877
Notes
1. Just consider the rough 5.000 citations of Moore’s one book. Most
cited articles (Scopus or WebOfScience) are, among others, by
Stoker (2006), Bozeman (2007), Jørgensen & Bozeman (2007)
or work by Alford & O’Flynn (2009). We use their work here
as well.
2. And this type also connects to the last category by van de
Wijdeven (2012), but more to his example—civility—than to
the question “what does it bring forth”because that question
allows for a broad range of answers (such as a clean park too).
3. De Haan et al. (2018, p. 313) also found this in their literature
as well: “The conditions and indicators of success are also well
described in the literature (…), but these studies do not
examine how success is conceptualised.”
4. When we calculate the total z score variances of the three types
of statements, the material types have the lowest (indicating
that these statements are often placed around the same posi-
tions on the Q sort) and the process types the highest (indicat-
ing the opposite).
References
Alford, J., & Hughes, O. (2008). Public value pragmatism as the
next phase of public management. The American Review of
Public Administration,38(2), 130–148. https://doi.org/10.
1177/0275074008314203
Alford, J., & O’Flynn, J. (2009). Making sense of public value:
Concepts, critiques and emergent meanings. Intl Journal of
Public Administration,32(3-4), 171–191. https://doi.org/10.
1080/01900690902732731
Bailey, N. (2012). The role, organisation and contribution of com-
munity enterprise to urban regeneration policy in the UK.
Progress in Planning,77(1), 1–35. https://doi.org/10.1016/j.
progress.2011.11.001
Bakker, J., Denters, B., Oude Vrielink, M., & Klok, P. J. (2012).
Citizens’initiatives: How local governments fill their facilita-
tive role. Local Government Studies,38(4), 395–414. https://
doi.org/10.1080/03003930.2012.698240
Banasick, S. (2019). KADE: A desktop application for Q methodol-
ogy. Journal of Open Source Software,4(36), 1360. https://doi.
org/10.21105/joss.01360
Baum, H. S. (2001). How should we evaluate community initia-
tives? Journal of the American Planning Association,67(2),
147–158. https://doi.org/10.1080/01944360108976225
Benington, J. (2009). Creating the public in order to create public
value? Intl Journal of Public Administration,32(3-4), 232–
249. https://doi.org/10.1080/01900690902749578
Blok, S. N., Fenger, H. J. M., & van Buuren, M. W. (2022).
Stimulating civic behavior? The paradoxes of incentivising
self-organization. Local Government Studies,1–20. https://
doi.org/10.1080/03003930.2022.2087061
Blok, S. N., van Buuren, M. W., & Fenger, H. J. M. (2020).
Exclusivity of citizens’initiatives: Fuel for collective action?
Journal of Civil Society,16(3), 243–259. https://doi.org/10.
1080/17448689.2020.1794168
Bohman, J. (2000). Public deliberation: Pluralism, complexity, and
democracy. MIT press.
Statement: Citizens’initiatives…Type
Idealistic
immaterialists
Pragmatic
partners
Personal
Purpose
Democratic
developers
Difference beween
agreement &
disagreement
strengthen democracy P 0,37 −1.89 2,26
−1,10 1,31 2,33
−1,89 1,31 3,2
are valuable in themselves P 2,4 −0,54 2.93
2,4 −0,58 2,9
contribute to meaning making for participants I −0,6 2,14 2,73
−0,5 2,14 2,64
contribute to the ability to work together*P 0,05 2,36 2,32
create societal value for the entire society I −1,59 1,06 2,64
ensure protest about what is wrong P 0,84 −2,21 3,05
*Even though there is quite a difference in scores, they are still both positive.
Blok et al. 15

Boonstra, B., & Boelens, L. (2011). Self-organization in urban
development: Towards a new perspective on spatial planning.
Urban Research & Practice,4(2), 99–122. https://doi.org/10.
1080/17535069.2011.579767
Bozeman, B. (2002). Public-value failure: When efficient markets
may not do. Public Administration Review,62(2), 145–161.
https://doi.org/10.1111/0033-3352.00165
Bozeman, B. (2007). Public values and public interest:
Counterbalancing economic individualism. Georgetown
University Press.
Brandsen, T., Trommel, W., & Verschuere, B. (2017). The state and
the reconstruction of civil society. International Review of
Administrative Sciences,83(4), 676–693. https://doi.org/10.
1177/0020852315592467
Brown, S. R.. (1980). Political Subjectivity: Applications of Q
Methdology in Political Science. New Haven: Yale University
Press.
Brown, S. R. (1993). A primer on Q methodology. Operant
Subjectivity,16(3/4), 91–138. https://doi.org/10.22488/okstate.93.
100504
Bryant, C. G. (1993). Social self-organisation, civility and sociol-
ogy: A comment on Kumar’s’Civil Society’.British Journal
of Sociology,44(3), 397–401. https://doi.org/10.2307/591809
Bryson, J. M., Quick, K. S., Slotterback, C. S., & Crosby, B. C.
(2013). Designing public participation processes. Public
Administration Review,73(1), 23–34. https://doi.org/10.1111/
j.1540-6210.2012.02678.x
Castells, M. (2008). The new public sphere: Global civil society,
communication networks, and global governance. The
aNNalS of the American Academy of Political and Social
Science,616(1), 78–93. https://doi.org/10.1177/00027162
07311877
De Haan, E., Meier, S., Haartsen, T., & Strijker, D. (2018). Defining
‘Success’of local citizens’initiatives in maintaining public ser-
vices in rural areas: A professional’s perspective. Sociologia
Ruralis,58(2), 312–330. https://doi.org/10.1111/soru.12173
Den Ouden, W., Boogaard, G., & Driessen, E. M. M. A. (2019).
Right to Challenge. Een studie naar de mogelijkheden voor
aan algemene regeling voor het ‘Right to Challenge’en
andere burgerinitiatieven in Nederland [Right to Challenge.
A study into the possibilities for a general arrangement for
the ‘Right to Challenge’and other citizens’initiatives in the
Netherlands].
Denters, S. A. H., Tonkens, E. H., Verhoeven, I., & Bakker, J. H. M.
(2013). Burgers maken hun buurt [Citizens make their neigh-
bourhood].Platform31.
Diamond, L. (1994). Rethinking civil society: Toward democratic
consolidation. Journal of Democracy,5(3), 4–17. https://doi.
org/10.1353/jod.1994.0041
Duijn, M., & Popering-Verkerk, V. (2018). Integrated public value
creation through community initiatives—evidence from Dutch
water management. Social Sciences,7(12), 261. https://doi.
org/10.3390/socsci7120261
Edelenbos, J., van Meerkerk, I., & Schenk, T. (2018). The evolution
of community self-organization in interaction with government
institutions: Cross-case insights from three countries. The
American Review of Public Administration,48(1), 52–66.
https://doi.org/10.1177/0275074016651142
Edwards, M. (2009). Civil Society. Polity.
Eriksson, K. (2012). Self-service society: Participative politics and
new forms of governance. Public Administration,90(3), 685–
698. https://doi.org/10.1111/j.1467-9299.2011.01996.x
Fiorina, M. P. (1999). Extreme voices: A dark side of civic engagement.
In T. Skocpol & M. P. Fiorina (Eds.), Civic engagement in
American democracy (pp. 395–425). Brookings Institution Press.
Foley, M. W., & Edwards, B. (1996). The paradox of civil society.
Journal of Democracy,7(3), 38–52. https://doi.org/10.1353/
jod.1996.0048
Fukuyama, M. F. (2000). Social capital and civil society.
International Monetary Fund.
Hartley, J. (2005). Innovation in governance and public services: Past
and present. Public Money and Management,25(1), 27–34.
https://doi.org/10.1111/j.1467-9302.2005.00447.x
Hartley, J., Alford, J., Knies, E., & Douglas, S. (2017). Towards an
empirical research agenda for public value theory. Public
Management Review,19(5), 670–685. https://doi.org/10.1080/
14719037.2016.1192166
Healey, P. (2015). Citizen-generated local development initiative:
Recent English experience. International Journal of Urban
Sciences,19(2), 109–118. https://doi.org/10.1080/12265934.
2014.989892
Henriksen, L. S., Strømsnes, K., & Svedberg, L. (2018). Civic
engagement in Scandinavia. Volunteering, informal help and
giving in Denmark, Norway and Sweden. Springer.
Hurenkamp, M., & Tonkens, E. (2011). De onbeholpen samenlev-
ing: burgerschap aan het begin van de 21e eeuw [The inept
society: citizenship at the Dawn of the 21st century].
Amsterdam University Press.
Hurenkamp, M., Tonkens, E., & Duyvendak, J. W. (2006). Wat
burgers bezielt: een onderzoek naar burgerinitiatieven [What
inspires citizens: a study of citizens’initiatives]. Universiteit
van Amsterdam/NICIS Kenniscentrum Grote steden.
Igalla, M., Edelenbos, J., & van Meerkerk, I. (2019). Citizens in
action, what do they accomplish? A systematic literature
review of citizen initiatives, their main characteristics, out-
comes, and factors. VOLUNTAS: International Journal of
Voluntary and Nonprofit Organizations,30(5), 1176–1194.
https://doi.org/10.1007/s11266-019-00129-0
Jørgensen, T. B., & Bozeman, B. (2007). Public values: An inven-
tory. Administration & Society,39(3), 354–381. https://doi.org/
10.1177/0095399707300703
Keane, J., & Merkel, W. (2019). Civil society. In W. Merkel,
R. Kollmorgen, & H. J. Wagener (Eds.), The handbook of polit-
ical, social, and economic transformation (pp. 390–398). Oxford
University Press.
Kleistra, Y. (2019). Over Rousseau, goede burgers en de participa-
tiesamenleving [About Rousseau, good citizens and the partic-
ipation society]. Beleid en Maatschappij,4(46), 410–428.
https://doi.org/10.5553/BenM/138900692019046004002
MacIntyre, A. (2011). After Virtue: A Study in Moral Theory.
Bloomsbury.
Mattijssen, T. J. M., Buijs, A. E., Elands, B. H. M., & Dam, R. I. v.
(2015). De betekenis van groene burgerinitiatieven; analyse
van kenmerken en effecten van 264 initiatieven in Nederland
[The significance of green citizens’initiatives; analysis of char-
acteristics and effects of 264 initiatives in the Netherlands].
Wettelijke Onderzoekstaken Natuur & Milieu, Wageningen
UR. WOt-rapport 127.
16 American Review of Public Administration 0(0)

McKeown, B. F., & Thomas, D. B. (1988/2013). Q methodology
(vol. 66). Sage Publications.
Meynhardt, T. (2009). Public value inside: What is public value cre-
ation? Intl Journal of Public Administration,32(3-4), 192–219.
https://doi.org/10.1080/01900690902732632
Ministry of the Interior and Kingdom Relations (2018). Kamerbrief plan
van aanpak voor versterking lokale democratie en bestuur [Letter to
parliament on an action plan for strengthening local democracy and
governance]. Kamerstukken 2017/18 VII 34 775, nr. 69.
Ministry of the Interior and Kingdom Relations [Ministerie van
Binnenlandse Zaken en Koninkrijkrelaties] (2013). De
Doe-Democratie. Kabinetsnota ter stimuleren van een vitale
samenleving [The Do Democracy. Cabinet memorandum to
stimulate a vital society]. Den Haag.
Moore, M. H. (1995). Creating public value: Strategic management
in government. Harvard University Press.
Movisie. (2011). Actief burgerschap. Een overzicht aan interventies
[Active citizenship. An overview of interventions]. Movisie.
Olson, M. (1965). The Logic of Collective Action: Public Goods and
the Theory of Groups.
Ostrom, E., Gardner, R., & Walker, J. (1994). Rules, games, and
common-pool resources. University of Michigan Press.
Overeem, P., & Tholen, B. (2011). After managerialism:
MacIntyre’s lessons for the study of public administration.
Administration & Society,43(7), 722–748. https://doi.org/10.
1177/0095399711413728
Prebble, M. (2018). Is “we”singular? The nature of public value.
The American Review of Public Administration,48(2), 103–
118. https://doi.org/10.1177/0275074016671427
Putnam, R. D. (2001). Bowling alone: The collapse and revival of
American community. Simon and Schuster.
Rhodes, R. A., & Wanna, J. (2007). The limits to public value, or
rescuing responsible government from the platonic guardians.
Australian Journal of Public Administration,66(4), 406–421.
https://doi.org/10.1111/j.1467-8500.2007.00553.x
Stephenson, W. (1953). The study of behavior; Q-technique and its
methodology. University of Chicago Press.
Stoker, G. (2006). Public value management: A new narrative for net-
worked governance? The American Review of Public
Administration,36(1), 41–57. https://doi.org/10.1177/027507400
5282583
Streeck, W., & Schmitter, P. C. (1985). Community, market, state-
and associations? The prospective contribution of interest gov-
ernance to social order. European Sociological Review, 1(2),
119–138. https://doi.org/10.1093/oxfordjournals.esr.a036381
Tonkens, E. (2006). De bal bij de burger [The ball is in the citizen’s
court]. Oratie, Vossiuspers Amsterdam.
Uitermark, J. (2015). Longing for Wikitopia: The study and politics
of self-organisation. Urban Studies,52(13), 2301–2312. https://
doi.org/10.1177/0042098015577334
van Dam, R., Duineveld, M., & During, R. (2015). Delineating
active citizenship: The subjectification of citizens’initiatives.
Journal of Environmental Policy & Planning,17(2), 163–
179. https://doi.org/10.1080/1523908X.2014.918502
van der Torre, L., Douglas, S., & ‘t Hart, P. (2019). Werken aan publieke
waarde: Leren van en voor gemeenten [Working on public value:
Learning from and for local governments]. Vereniging Nederlandse
Gemeenten. https://doi.org/10.13140/RG.2.2.17258.36806
van de Wijdeven, T. (2012). Doe-democratie: Over actief burger-
schap in stadswijken [Do democracy: About active citizenship
in urban districts]. Eburon Uitgeverij BV.
van Rooij, H., van Wessel, M., & Aarts, N. (2019). Over zelfred-
zame burgers gesproken [Talking about self-reliant citizens].
Beleid en Maatschappij,4(46), 429–452. https://doi.org/10.
5553/BenM/138900692019046004003
van Straalen, F. M., Witte, P., & Buitelaar, E. (2017). Self-organisation
in oosterwold, almere: Challenges with public goods and exter-
nalities. Tijdschrift Voor Economische en Sociale Geografie,
108(4), 503–511. https://doi.org/10.1111/tesg.12267
Verhoeven, I., & Tonkens, E. (2013). Talking active citizenship:
Framing welfare state reform in England and The
Netherlands. Social Policy and Society,12(3), 415–426.
https://doi.org/10.1017/S1474746413000158
Vollan, B. (2008). Socio-ecological explanations for crowding-out
effects from economic field experiments in Southern Africa.
Ecological Economics,67(4), 560–573. https://doi.org/10.
1016/j.ecolecon.2008.01.015
Walzer, M. (1991). The civil society argument. Statsvetenskaplig
Tidskrift,94(1), 1–11.
Watts, S., & Stenner, P. (2005). Doing Q methodology: Theory,
method and interpretation. Qualitative Research in Psychology,
2(1), 67–91. https://doi.org/10.1191/1478088705qp022oa
Author Biography
S. N. Blok is a PhD candidate at the Department Public
Administration & Sociology at Erasmus University Rotterdam. His
research interests are the collective action problems, crowding
effects and value perceptions of self-organizing collectives of citizens.
H. J. M. Fenger is Full professor Institutional Policy Analysis at the
Department of Public Administration & Sociology at Erasmus
University Rotterdam. His research focusses on analyzing and
explaining changes in the policies, governance, administration,
and implementations of European welfare states
M. W. van Buuren is Full Professor of Public Administration at
Erasmus University Rotterdam and Director of the Department
of Public Administration & Sociology. His research interests
are on issues of (co-)design for policy and governance, invita-
tional governance and self-organization, collaborative gover-
nance and co-creation, policy innovation and institutional
change.
Blok et al. 17