The Integrity of Deliberative Procedures - A Research Agenda for Measuring Deliberative Quality with a Systemic Framework

Abstract

Assessing the integrity of democratic procedures is essential in order to evaluate democratic quality. In this paper, we argue that deliberative procedures' quality should be measured in an analogous way. In order to do so, existing measurement approaches (such as the DQI) have to be upscaled to the macro level of political systems. We outline two exemplary challenges this undertaking has to deal with: Firstly, deliberative practices occurring throughout democratic political systems are heterogeneous with regards to their communicative style and their purposes. Secondly (and relatedly), they are occurring in a diversity of loci (such as democratic innovations, parliaments, courts, or social media). We suggest strategies to address each of these challenges by applying concepts of the systemic approach to deliberation. In the concluding section, we exemplify the benefits of these strategies for a valid measurement of deliberative quality at the macro level.

Full text

1
The Integrity of Deliberative Procedures –
A Research Agenda for Measuring Deliberative Quality with a Systemic
Framework
− Draft! Please do not cite without permission –
Dr. Dannica Fleuß
Postdoctoral Researcher, Helmut-Schmidt-University, Hamburg,
email: dannica.fleuss@hsu-hh.de
Prof. Dr. Gary S. Schaal,
Full Professor of Political Theory, Helmut-Schmidt-University, Hamburg
email: gschaal@hsu-hh.de
Abstract
Assessing the integrity of democratic procedures is essential in order to evaluate democratic
quality. In this paper, we argue that deliberative procedures’ quality should be measured in an
analogous way. In order to do so, existing measurement approaches (such as the DQI) have to be
upscaled to the macro level of political systems. We outline two exemplary challenges this
undertaking has to deal with: Firstly, deliberative practices occurring throughout democratic
political systems are heterogeneous with regards to their communicative style and their purposes.
Secondly (and relatedly), they are occurring in a diversity of loci (such as democratic innovations,
parliaments, courts, or social media). We suggest strategies to address each of these challenges
by applying concepts of the systemic approach to deliberation. In the concluding section, we
exemplify the benefits of these strategies for a valid measurement of deliberative quality at the
macro level.
Last update: April 2019
Paper to be presented at the IPSA World Congress 2018, Brisbane, July 21st -25th
Panel RC06.02: Direct and Deliberative Democracy. How to Measure Integrity?

2
1.
Introductory: Why Deliberative Procedures’ Integrity Matters and How to
Assess them Empirically
Assessing the ‘integrity’ of democratic procedures is essential in order to evaluate
democratic quality. In 2013, the Electoral Integrity Project (EIP), famously advanced by
Pippa Norris, offered an approach for the multi-dimensional evaluation of one procedure
fundamental to all democracies, i.e.: elections (Norris, Frank, & Martínez i Coma, 2013;
Norris, Frank, & Martínez i Coma, 2014). This paper’s point of departure is the premise
that electoral procedures are not the only procedures that should be analyzed with a more
fine-grained, multi-dimensional approach.
Even though the importance of deliberative procedures might be evaluated differently
dependent on scholars’ theoretical point of view,
1
we argue that all measurements of
democracy have to take into account deliberative procedures’ quality (cf. Fleuß, Helbig,
& Schaal, 2018). This claim is based on deliberative procedures’ role in both theoretical
and empirical research on democratic quality: From a theoretical perspective, the
deliberative paradigm has become the most important normative approach to
democratic legitimacy (Dryzek, 2015; Elstub, 2015). From an empirical perspective,
democratic deliberation plays an important functional role in contemporary (liberal)
democracies: In ‘conventional’ representative institutions such as courts or parliaments
as well as in the broader public sphere, deliberation is essential for democratic decision-
making. Even measurements of democratic quality presupposing a liberal concept of
democracy (such as the Democracy Barometer) consider deliberation in the public sphere
to be highly relevant for democratic quality (Bühlmann, Merkel, Müller, & Weßels, 2012).
In addition, there is an increasing amount of ‘non-conventional’ means of participation
(such as deliberative mini-publics) that aim at introducing deliberative procedures in
liberal democracies in order to improve citizen support vis-à-vis democratic decision-
making (on these so-called “democratic innovations”: Geissel & Newton, 2012; Smith,
2005).
Nevertheless, a comprehensive framework for evaluating deliberative procedures’
integrity at the macro level (of nation states) is missing so far. Therefore, we will argue
that deliberative procedures in contemporary democratic systems have to be empirically
assessed in a similar way and outline cornerstones for an analogue project to be
conducted by deliberative scholars. Accordingly, our paper addresses the following
overarching question: How can a valid measurement of deliberative quality at the macro
level of political systems be conducted? We answer this question by arguing (a) that the
systemic approach to deliberation provides empirical researchers with the conceptual
resources necessary and (b) that actually conducting this measurement would be possible
1
Liberal theories of democracy focus on the aggregation of ‘fixed’ citizen preferences that are articulated
by voting, though they often attribute a legitimizing function to deliberation in the public sphere or the
parliament. At the extreme, democratic Theories such as Joseph Schumpeter’s identify the essential politics
in competition, not in consensus-oriented reason-giving (cf. Niesen, 2014; cf. Thompson, 2008, p. 498).

3
only as a collaborative project of scholars on deliberative democracy (for which the EIP in
many respects can serve as a model).
The paper is structured in the following way: Subsequent to outlining our understanding
of the fundamental goals of the Electoral Integrity Project (chapter 2), we will argue that
the systemic approach has the conceptual resources to meet the deliberation-specific
challenges posed by such an enterprise (chapter 3). In chapter 4, we will provide a more
detailed account of our measurement approach by suggesting strategies to cope with two
fundamental problems of ‘upscaling’ existing measurements of deliberation to the macro
level. In the concluding section (chapter 5), we will illustrate the benefits of these
strategies with the help of exemplary empirical studies applying DQI-criteria to different
‘types’ of deliberative settings.
2.
Context: The Differentiated Measurement of Democratic Quality and the
Electoral Integrity Project
The overarching aim of the EIP is to assess the “overall quality of elections” (Norris et al.,
2013, p. 124). Although all measurement approaches to democratic quality take “fair and
free elections” to be the necessary condition for democratic rule, the third wave of
democratization has called for a more differentiated assessment of democratic
performance (Geissel, Kneuer, & Lauth, 2016; Giebler, Ruth, & Tanneberg, 2018; Lauth,
Pickel, & Welzel, 2000). This change of political reality challenges scholars to develop
measurement approaches that are able to distinguish systematically between political
systems that are performing better or worse in different dimensions of “democratic
quality” (Morlino, 2004):
“With a few notable exceptions, regimes without the fig leaf of elections have dropped
out of fashion. […]. The spread of direct elections worldwide has been accompanied by
growing concern about the unevenness of their quality […].” (Norris, 2014, p. 5 f.)
In order to provide a valid measurement of democratic quality, political scientists
accordingly “[…] need to be able to distinguish autocracies that permit superficial
competition and manipulated elections from states that are more clearly moving toward
democratic contests that would allow alternation between governing and opposition
parties.” (Norris et al., 2013, p. 125) To achieve this overarching aim, it is necessary to
develop “a more systematic and comprehensive source of independent evidence that
[…enables a] rigorous assessment of elections worldwide”.
Norris et al.’s project starts from a multidimensional conceptualization of “electoral
integrity” that allows for gradual differentiation (Norris et al., 2013, p. 126). The concept
of electoral integrity is derived from “international commitments and global norms
surrounding elections” as outlined in international obligations (endorsed by the European
Union, the Organization of American States, the OSCE, or the Electoral Institute for
Sustainable Democracy in Africa, etc.) (Davies-Roberts & Carroll, 2014, p. 20; Norris, 2014,
p. 9). The core concept consists of four sequential dimensions representing different

4
phases of the “electoral cycle” (pre-election, campaign, election day, and post-election)
(Norris et al., 2013, p. 127). 49 indicators are allocated to these dimensions and are coded
by (political science) experts on “a standardized and consistent five-point agree – disagree
scale.” (Norris, Frank, & Martínez i Coma, 2014, p. 790). The EIP thereby supplements
previous empirical assessments of elections by combining a “thick”, multidimensional
concept of democratic elections with the suitability for large-N comparisons (Norris,
Frank, & Martínez i Coma, 2014, p. 791).
2
Norris et al. thereby aim at providing a “gold standard” for the measurement of electoral
procedures’ quality that is more fine-grained and differentiated than the measurement
provided by established indices for democratic quality such as Freedom House and Polity,
(cf. Abramowitz, 2018; Marshall, Jaggers, & Gurr, 2002). One fundamental challenge the
EIP has to cope with is to develop a framework that is suitable for large-N comparative
analyses while doing justice to the concept’s complexity.
3
Associated with this challenge
is the need to instruct multiple coders from all over the world and guarantee the
necessary reliability. In the following chapter, we will outline in how far the project of
measuring deliberative procedures’ quality at the macro level is confronted with partly
analogous challenges. In the next step, we will suggest an approach to address them by
applying a systemic framework.
3.
Goals and Two Major Challenges of Measuring Deliberative Procedures’
Quality at the Macro Level
In this paper, we propose to assess the quality of a certain type of democratic procedures
at the macro level of nation states and aim at an analytical and empirical framework
suitable for large-N comparative research. This overarching aim is comparable to the EIP’s
and might lead to similar feasibility problems that we will address below. However,
measuring the quality of deliberative procedures at the macro level (i.e. upscale the
measurement of individual deliberations) has its own specific requirements and brings
about new deviances. In this paper, we will address two exemplary challenges of
‘upscaling’ the measurement of deliberative quality.
First challenge:
Real-world deliberative practices occur in different loci of a polity, fulfill different
functions, and actors involved in these processes apply different styles of reason-giving.
This heterogeneity challenges a valid empirical assessment: applying one and the same
standard for “good deliberation” to all of these instances – to deliberations in federal
2
Norris et al. thereby see themselves as avoiding certain flaws of measurements of democratic
performance (such as Freedom House or Polity IV) as well as the qualitative reports of news media and
monitoring agencies (Norris et al. 2014, p. 791).
3
“In all countries, evidence from the World Value Survey suggests that public perceptions of electoral
malpractices erode trust and confidence in elected authorities, discourage voter turnout and generate
protests, and even undermine regime stability […]” (Norris, Frank, & Martínez i Coma, 2014; Norris, Frank,
& Martínez i Coma, 2014, p. 789).

5
courts, in designed mini-publics, in informal political debates in social media, etc. – might
be misleading.
Second challenge:
Macro level deliberative quality is not simply an accumulation of individual deliberations’
qualities. Rather, it is plausible to assume that deliberations in different loci can (and
should) influence each other mutually. From the perspective of deliberative theory, it is of
outstanding importance that deliberations in the civil society, the media, or in democratic
innovations are actually influencing deliberations among actors with the competency to
make collectively binding decisions. Both measurement and aggregation procedure have
to reflect these interactive relationships and potential complementarities (Møller &
Skaaning, 2012).
We argue that theories and concepts that evolved in the course of the so-called systemic
turn have the conceptual resources to address both challenges. Systemic research on
deliberation combines the insights of normative reflections that provided the starting
point for deliberative democracy in the 1980s and 1990s (Cohen, 1989, 1999; Habermas,
1990, 1996) with the empirical insights gathered in subsequent phases of the research
paradigm’s evolution (Elstub, Ercan, & Mendonça, 2016). For these periods, empirical and
theoretical research focused on deliberative procedures at the meso level (exemplary:
Fishkin, 1995; Steenbergen, Bächtiger, Spörndli, & Steiner, 2003). Systemic approaches,
by contrast, seek to analyze the complex interplay of these individual instances of
deliberation in different spheres of a democratic society.
4
The focus of analyses inspired
by systemic thinking is the “deliberative quality” of a polity as a whole. In this paper, we
will label this deliberative quality of the system as a whole “systemic deliberative quality”.
The core assumptions of the systemic approach to deliberation that will be relevant for
the framework outlined in this paper can be summarized in the following way:
a) Loci of deliberation: Deliberative systems are constituted by individual loci of deliberation
that are located in either the public or the empowered space.
b) Public and empowered space: Both spaces are primarily distinguished by the power to
make collectively binding decisions.
c) Transmissions: In order to constitute a system, the individual loci have to be connected,
i.e. the results of their deliberations are to be transmitted to other loci. These
transmissions can (and should) in principle go both ways, i.e. from the public to the
empowered space and from decision-making institutions as a “feedback to the wider
public” (Caluwaerts & Reuchamps, 2014).
5
4
Although the Varieties of Democracy Project’s “deliberative component index” incorporates some of these
insights in measuring democratic deliberation, the problem of upscaling is not adequately addressed.
5
The analysis of Caluwaerts & Reuchamps (2014) focuses on the interactions between democratic
innovations and the wider public. However, the general premise – that transmissions are supposed to go
“both ways” holds for a systemic analysis independently of the concrete loci analyzed.

6
d) Direction of Transmissions: Transmissions from loci in the public space to decision-making
loci are of outstanding importance for the deliberative quality of the system as a whole
(Dryzek & Niemeyer, 2010, p. 11).
Based on this framework, “concepts developed and refined in the study of small-scale
forums […] can [in principle…] be applied to large-scale systems [such as nation states]”
(Dryzek & Niemeyer, 2010, p. 9). Upscaling previous assessments of small-scale forums
brings about numerous theoretical, conceptual, and methodological challenges that we
reflected on in previous work by outlining “Four Parameters for Measuring Macro
Deliberation” (Fleuß et al., 2018). In the next chapter of this paper, we will address two
exemplary challenges and propose strategies to address them.
4.
Upscaling the Measurement of Deliberation: Strategies to Address Two
Fundamental Challenges
4.1.
The Concept of Systemic Deliberative Quality
The original meaning of ‘deliberation’ is ‘taking and giving of reasons’. It has to be
acknowledged that (purely rational, impartial, etc.) reason-giving as characterized by
Cohen (1989) or Habermas (1990; 1996) is rarely taking place in real-world politics.
Therefore, more recent approaches tend to include diverse styles of communication in
the concept of ‘deliberation’ (Chambers, 2003). In theorizing and empirically assessing
democratic deliberation, both procedural and output-oriented criteria are applied:
Deliberation is supposed to be both fair, inclusive, free of coercion and leading to
epistemically beneficial policy outputs (Cooke, 2000). However, high performance
according to procedural standards for deliberation and high performance according to
epistemic standards do not necessarily coincide.
6
As this article deals with measuring the procedural dimension of democratic deliberation,
we do not have to engage with this debate. We will rather focus on the established criteria
measuring the procedural aspects such as inclusiveness, non-coercion, and rationality
(Bohman, 1998; Estlund, 2008; Habermas, 1996; Peter, 2008). We expect individual
instances of deliberation in different spheres of a political system to perform differently:
Deliberations in federal courts are a prime example for procedures that tend to perform
very well according to the standards of deliberative theory. Informal political debates in
the public debates or social media, e.g., tend to deviate from them. This complication of
measuring deliberation at the macro level will not be reflected in the conceptualization,
but will be relevant in the aggregation rule applied (chapter 4.2).
For a valid conceptualization and measurement of systemic deliberative quality, it is not
sufficient to assess the procedural quality of each (type of) deliberative procedure
6
In Fleuß et al. (2018), we therefore asked researchers to make a reflected decision for applying one of
them.

7
separately. Rather, the interconnectedness of procedures in the system as a whole has to
be evaluated, i.e. the transmissions between different loci of deliberation (Boswell,
Hendriks, & Ercan, 2016; Dryzek & Niemeyer, 2010; Dryzek, 2000; Habermas, 1996).
Therefore, the concept of ‘systemic deliberative quality’ has two dimensions: Systemic
deliberative quality is constituted by (a) the quality of individual deliberations occurring
throughout the political system and society; (b) the “flow of communication” between
these individual loci. The general hypothesis is that a high deliberative quality at the
macro level demands a high degree of transmissions, predominantly from the public to
the empowered space.
This amounts to the following concept of macro level deliberative quality (henceforth:
“systemic deliberative quality”):
a) For a polity to have “systemic deliberative quality”, there needs to be deliberation
in individual loci of the system. It is not a necessary condition that these individual
instances of deliberation show a specific quality (that might be specified by a
certain DQI-score).
b) A polity has a “systemic deliberative quality” (of whatever degree) only if there is
a pronounced “flow of communication” between loci of deliberation, especially
from the public to the empowered space (i.e.: deliberative topics are ‘traveling’
from one locus to another). [necessary condition]
4.2.
Strategy 1: Measurement of Individual Instances of Deliberative
Procedures
One challenge of measuring systemic deliberative quality results from the fact that
deliberative democracy is essentially a “talk-based approach”. This implies that (often not
recorded or transcribed) talk on political matters has to be systematically evaluated. In
order to offer a valid assessment of the reason-giving taking place in the heterogeneous
loci of a political system, we suggest to categorize loci according to the formality and
decision-making capacity of the respective deliberative procedures (cf. Conover &
Searing, 2005). Building on Dryzek’s approach (Dryzek, 2000), we allocate loci of
deliberation to two categories, i.e. the empowered and the public space:
7
1. Empowered space: Highly formal deliberations predominantly occur within
institutions with formal decision-making capacity such as national courts or
parliaments. These deliberations tend to be most compliant with high standards
of rational reason-giving.
2. Public space: Less formal deliberations predominantly occur within the broad
realm of the public space. In general, we assume that loci in the public space, such
7
In our previous approach, we made a more fine-grained distinction and added an intermediate third
category: “semi-formal deliberations”.

8
as debates in the media, the civil society, or even everyday political talk will
deviate from standards of rational reason-giving to a greater degree.
8
Although we assume that each of these arenas has its specifics and might ask for an
adapted standard for “good deliberation”, this would increase the theoretical and
empirical complexity of the measurement approach extraordinarily. For feasibility
reasons, we therefore suggest a simplified strategy: All instances of deliberative
procedures are measured with the same basic DQI-standards and -methodology
(Steenbergen et al., 2003). From a conceptual perspective, the criteria applied by the DQI
to evaluate individual instances of deliberation match the research interest of this paper:
Steenbergen et al. define “high quality of deliberation” in strongly procedural terms, i.e.
they exclude the evaluation of the (epistemic) quality of deliberations’ outputs (2003, p.
22).
Accordingly, the indicators used to calculate the DQI-score
9
mirror the procedural
dimension of deliberative quality:
10
Indicators
Possible indicator values (only
integers)
1. Participation [P], referring to the “speaker’s
ability to participate freely in a debate”
0-1
2. Level of Justification [LJ], referring to the
rationality of the deliberation in terms of the
logical structure of the arguments given.
0-3
3. Content of Justification [CJ], referring to the
common good-orientation of the arguments
given.
0-2
4. Respect for other groups [RG]
0-2
5. Respect for counterarguments [RC] –
0-3
6. Constructive Politics [CP], referring to the
consensus-orientation displayed in the
speech actors contributing to the
deliberation.
0-2
8
An explicit measurement of discourse quality in the broader public sphere, especially in informal “everyday
political talk” (Mansbridge, 1999), is complicated by matters of data availability: Usually, this type of
deliberation is not recorded or transcribed. Due to their artificiality, field experiments and studies on
deliberative quality in small forums provide useful insights for both measuring and explaining deliberative
quality, while their implications for the broader public sphere should be treated with caution. An exception
is online deliberation, though it raises additional methodological problems for existing measurement
instruments (Bello Hutt, 2018).
9
The score for the discourse quality of an individual instance of deliberation is then constructed by an
additive procedure. The authors mention that alternatively, a factor index can be used to calculate the DQI-
score for an instance of deliberation.
10
Steenbergen et al. (2003) exclude one coding category for methodological reasons from the DQI-index
(“Demands”).

9
In terms of aggregation procedures, this strategy suggests to calculate an Individual
Deliberations Score (IDS) on the basis of the DQI scores reached in individual deliberative
loci. As we consider individual instances of deliberation (at least in principle) to be
mutually compensatory, we propose an additive aggregation procedure for calculating a
polity’s IDS (Møller & Skaaning, 2012).
The classic Habermas-criteria applied in the DQI in themselves will not do justice to the
plurality and heterogeneity of deliberations occurring throughout a complex political
system: Different maxima and minima are to be expected for different types of loci (with
formal and informal deliberations).
11
In order to compensate for the potentially lower
performance in terms of DQI-scores of loci in the public space, we suggest to weight the
DQI-scores dependent on the category the respective locus was allocated to (public
space/empowered space). The weight attributed to the sum of DQI-scores in the public
space (“a”) is to be higher than the weight attributed to the accumulated DQI-scores in
the empowered space (“b”).
12
From a pragmatic perspective, conducting this measurement for a polity as a whole (or
even for all democratic polities) would be a tremendous effort. The “pooling of expert
knowledge” suggested by the EIP, however, seems to be a feasible way to systematize
and collect the expert evaluations of specialists on deliberation worldwide. The
methodology applied for organizing expert judgments could serve as an inspiration for a
similar project systematically assessing deliberative procedures’ integrity from a
comparative perspective (cf. Daxecker & Schneider, 2014; Martínez i Coma & Frank,
2014).
Summary Strategy 1: Deliberations in the loci of deliberation of a polity are measured by
DQI-standards and methodology. In the aggregation procedure, DQI-scores are weighted
in order to compensate for (misleadingly) low DQI-scores of informal deliberations in the
public space.
4.3.
Strategy 2: Measurement of Transmissions
At first sight, measuring deliberative procedural quality in a political system might amount
to calculating a DQI-score for the deliberations taking place in all (relevant) loci of
deliberation and then accumulating their values. This, however, would not do justice to
the conceptualization of “systemic deliberative quality” outlined in chapter 4.2: The
necessary condition for a system to display ‘systemic deliberative quality’ is that the loci
11
The communicative style, and accordingly the quality of deliberations as measured by DQI-standards,
tends to differ dependent on the type of locus it takes place in. This problem is aggravated by the fact that
from a normative perspective, deliberations among the general public are far more important for systemic
deliberative quality than deliberations among political professionals and experts (exemplary: Habermas
1996).
12
Determining the concrete value of these weights in order to reflect the theoretical considerations
presented above is, however, an intricate matter. One option would be to determine the value of these
weights on the basis of a comprehensive comparative empirical study that calculates maxima and minima
of DQI-scores for loci allocated to the public or the empowered space.

10
are connected, i.e. that the results of individual instances of deliberations are transmitted
to other loci – especially to loci in the empowered space. Assessing these interactions
between individual loci raises numerous problems that cannot exhaustively be addressed
in this paper.
13
In a nutshell, we suggest the following steps to assess the quantity of
transmissions in a deliberative system:
(1) Identify the potentially interacting loci of deliberation. What is transmitted from one
locus to another is in the following part of the paper labeled “topics”.
(2) Identify the topics that are deliberated on in loci in the public space and in loci in the
empowered space.
14
(3) Calculate the share of topics that are co-occurring in several loci of deliberation. As it
might not be feasible to address each and every possible pair of loci separately, we
suggest to calculate the share of topics that are deliberated on in both the public and
the private sphere.
(4) Determine the sequence of deliberations. Although step (3) might be a first
approximation for assessing the ‘flow of communication’ between loci, it is certainly
not sufficient: Firstly, deliberative topics might be co-occurring accidentally, i.e.
without one deliberation actually causing a deliberation on the same topic in another
locus. Secondly, the flow of communication directed from the public to the
empowered space is of extraordinary importance for high-quality deliberation.
15
Both problems cannot be solved, but mitigated by assessing the sequence of
deliberations on the topics taken into account, i.e. answering the question in which
locus a topic first appeared. We therefore suggest to calculate a “succeeding share”,
i.e. the share of topics in a system that originated (first appeared) in loci allocated to
the public space.
(5) Calculate the transmission score by combining quantity and direction of exchanges
between loci: We propose to use this “succeeding share” as a weight in calculating
the transmission score. This implies that the more deliberative topics originate in the
public space, the higher the transmission score will turn out.
16
Summary Strategy 2:
Firstly, the quantity of transmissions between different loci deliberation is
(approximately) measured by determining the share of topics co-occurring in
13
Some of the most serious methodological intricacies as well as a framework to address them will be
subject to a different paper. A first working paper on this approach was presented at the ECPR Joint Sessions
2018 (Fleuß & Helbig, 2018).
14
In the case of large amounts of text, text mining tools can be applied at this step.
15
We elaborate on this matter in chapter 5.2. The basic argument can be summarized in the following way:
If transmissions are only originating in decision-making loci, a high degree of transmissions might not be an
indicator for a high deliberative quality, but rather for the co-optation of loci in the public space.
16
This also implies that in case that all topics originate in the empowered space, i.e. the flow of
communication throughout the deliberative system is entirely dominated by formal decision-making
bodies, the transmission score will be zero.

11
different loci (share of co-occurring topics): The overlap of topics (OT) (0-100%) that
are topics of deliberations in public and empowered space is calculated.
Secondly, the share of overlapping topics that first occurred in the public space is
calculated, i.e. the succeeding share (SuS).
Thirdly, a transmission score (TS) is calculated that uses the share of deliberative
topics originating in the public space (succeeding share) as a weight: !
TS = OT × SuS
4.4.
How to Combine Strategy 1 and Strategy 2: Calculation of the overall
score for “Systemic Deliberative Quality”
Both strategies suggested so far should not be considered in isolation. Rather, we suggest
to combine them by multiplying the IDS with the degree of transmissions (TS). This
procedure seems to be the most fitting as transmissions between loci of deliberation are
conceptualized as a necessary condition for “systemic deliberative quality”.
Thus, we suggest an aggregation formula of the following form for calculating the systemic
deliberative quality (SDQ):
!"#$
%
&!$'$("!
)
*
*
*
+
,
- '
.
/01$23456$758294$67-4:!;$/01$23456$758294$67-4:"; <;$/01$23456$758294$67-4:#!
=>
?!
;
,
8$ '$
.
/01$23456$:@73A:B:C$67-4:!;$/01$23456$:@73A:B:C$67-4:";$D ; $/01$23456$:@73A:B:C$67-4:#"
=>
?"
$
E
F
F
F
G
With
H$ I J
5.
Exemplary Application: A Systemic Way to Address Two Fundamental
Challenges of Measuring Macro Deliberation
Measuring the quality of deliberation at the macro level poses serious challenges. We
argued that at least two of them can be addressed by applying a systemic framework. In
order to illustrate the benefits of the strategies we outlined in chapter 4, we will illustrate
their resources to cope with the two challenges highlighted in this paper: Firstly, the locus
of deliberation matters for an adequate appraisal and integration of individual
deliberations’ qualities in the evaluation of deliberative quality at the macro level (chapter
5.1). Secondly, the interactive, “systemic” effects between different forms of deliberation
(i.e., transmissions and their direction) have to be taken into account (chapter 5.2).

12
5.1.
DQI-scores in the Public Space and the Empowered Space and how to
deal with them
Pedrini (2014) analyzes differences in DQI-scores between a deliberative procedure in the
public space
17
and a Swiss parliamentary deliberation on the expulsion of foreigners from
Switzerland. In her analysis, she is able to confirm previous findings (Steiner, Bächtiger,
Spörndli, & Steenbergen, 2004) according to which political elites, e.g. in parliamentary
debates, tend to “score relatively highly on deliberative quality” (Pedrini, 2014, p. 266).
"Differences are particularly marked for sophisticated deliberation”, i.e. in the
argumentative complexity of the discourse (as measured by the Level of Justification- and
Content of Justification-indicators of the DQI). Referring to Steiner et al. (2004) and
Pedrini et al. (2013), she states that
“74% of the speakers in the best practice case [in the legislative realm, i.e. in the
empowered space] presented a complex argument at least once [while] this was true for
only about 30% of the speakers in Europolis [a deliberative poll, i.e. a locus of the public
space].” (Pedrini, 2014, p. 266)
The categorization of a locus as belonging to the public or the empowered space is neither
in all cases unambiguous
18
nor the only difference that matters for deliberative quality.
19
Nevertheless, case studies on deliberative procedures’ quality in the public and the
empowered space confirm that Habermas’s classic standards for “good deliberative
procedures” are more likely to be fulfilled in loci of the empowered space.
These differences in DQI-scores of loci in the public and the empowered space should be
reflected by a measurement of macro level deliberative quality. As DQI-scores can result
from different communicative styles (i.e., not all citizens will comply with Habermas’s
concept of rationality), we suggest not to classify “informal”, more “emotional” or less
“sophisticated” deliberation automatically as “worse deliberation”. Rather, in our
suggestion for measuring macro level deliberation, we compensate for this bias by
ascribing a higher weight to DQI-scores for the public space.
5.2.
Systemic Effects of Meso-Deliberations: Why transmissions must
considered for systemic deliberative quality
Qualities in individual loci are not “the whole story” about intact deliberative procedures
in a political system. This is impressively demonstrated by various analyses of deliberative
17
The data basis for Pedrini’s analysis is provided by an online field-experiment conducted by Bächtiger et
al. (2011).
18
As there are democratic innovations in which citizens deliberating have immediate decision-making
capacity, they cannot be allocated easily to one of both spheres. A prime example is the Citizens Assembly
on Electoral Reform in British Columbia (cf. Smith, 2009, p. 77 ff.).
19
Pedrini, 266: "Deliberative capacities are not only affected by the basic distinction between political and
civic sphere, but also by additional institutional varieties within these two basic contexts. With regard to
deliberation in the political sphere, the literature has identified a number of institutional arenas that
enhance deliberative capacities, especially publicity vs. non-publicity and first vs. second chambers."

13
mini-publics that display a high deliberative quality, but are either not connected to other
loci within the political system (cf. Hendriks, 2016) or “co-opted” by decision-making
institutions located in the empowered space (Caluwaerts & Reuchamps, 2016; Setälä,
2017):
“Deliberative democrats have also criticised practices of mini-publics because of
the possibility of their strategic use. Policy makers may organise mini-publics in
order to strengthen their own position in the eyes of the public, or to advance and
legitimise policies they pursue.” (Setälä, 2017, p. 6)
One illustrative case for co-optation are the Danish Consensus Conferences that were
established by a parliamentary institution, the Danish Board of Technology (ibid.). Even
though the results of the consensus conferences’ deliberations were integrated in the
decision-making process (i.e., they were ‘transmitted’ to a decision-making locus), they
are not considered to be unambiguously increasing the systemic deliberative quality: The
agenda-setting and the information for the citizens’ deliberations are explicitly performed
by loci from the empowered space. This at least poses the danger that political elites use
mini-publics in an instrumental fashion in order to legitimize their policies.
Our approach reflects “systemic effects” (Dryzek & Niemeyer, 2010, p. 155 ff.;
Mansbridge et al., 2012, p. 17) resulting from the interactive relationships between
different loci within the political system as a whole: Applying strategy 2 would result in
integrating an approximate measure of not only the quantity, but also the direction of
transmissions between loci. As this score for transmissions (TS) is integrated in the
multiplicative aggregation procedure for the calculation of the systemic deliberative
quality, the interactive relationships between deliberations in different loci are also
reflected in the overall score for a polity’s macro level deliberative quality.
6.
Summary and Discussion: Merits of a Systemic Framework for Measuring
Deliberative Integrity and Feasibility Issues
This paper started from the premise that electoral procedures are not the only democratic
procedures that have to be analyzed and evaluated with a more fine-grained and
differentiated approach. The Electoral Integrity Project developed a sophisticated multi-
dimensional measurement instrument for an evaluation of electoral procedures. We
argued that an analogous task has to be accomplished for assessing the quality of
democratic deliberation at the macro level (i.e. “systemic deliberative quality”).
‘Upscaling’ existing measurement approaches applied to meso level deliberations (such
as the DQI) to the macro level poses serious conceptual and methodological challenges.
The framework suggested in this article takes the basic premises of the systemic approach
to deliberation as a point of departure.
In the course of translating the systemic approaches premises into a measurement
approach, we tackled two major problems of measuring deliberative procedures’ quality

14
at the macro level: (1) We suggest to cope with the heterogeneity of styles of deliberation
occurring in different loci of the political system by attributing different weights to the
DQI-scores reached in different types of loci (in the empowered space vs. in the public
space) (chapter 4.2). (2) We respect the fact that the systemic deliberative quality is not
simply constituted by the quality of the individual deliberations occurring throughout the
polity, but also by the exchanges and interactions between them. We propose to
approximate the quantity and direction of these exchanges by measuring the
“transmissions” between loci of deliberation (chapter 4.3). We illustrate the merits of
integrating both steps in assessing deliberative quality at the macro level in chapter 5.
Nevertheless, future research will have to deal with several pragmatic and
methodological issues associated with these steps such as the methodology applied to
the measurement of transmissions and determining a non-arbitrary weight to DQI-scores
calculated for different spaces of a democratic society.
An evaluation of macro level deliberative quality would imply coding a vast number of
deliberative procedures according to DQI-criteria and therefore raises significant
feasibility issues. However, the Electoral Integrity Project’s approach to “pooling expert
knowledge” by systematizing expert evaluations on electoral procedures could inspire
scholars of deliberative democracy to address deliberative quality at the macro level as a
collaborative effort (cf. on the organization of the coding procedures in the EIP: Daxecker
& Schneider, 2014; Martínez i Coma & Frank, 2014).
7.
References
Abramowitz, M. J. (2018). Freedom in The World 2018: Democracy in Crisis. Freedom House, 1–
24. Retrieved from
https://freedomhouse.org/sites/default/files/FH_FITW_Report_2018_Final_SinglePage.pdf.
Bello Hutt, D. E. (2018, April 8). Measuring Popular and Judicial Deliberation: A Critical
Comparison. Retrieved from
https://papers.ssrn.com/sol3/papers.cfm?abstract_id=3158628.
Bohman, J. (1998). Survey article: The coming of age of deliberative democracy. Journal of
Political Philosophy, 6(4), 400–425.
Boswell, J., Hendriks, C. M., & Ercan, S. A. (2016). Message received? Examining transmission in
deliberative systems. Critical Policy Studies, 10(3), 263–283.
Bühlmann, M., Merkel, W., Müller, L., & Weßels, B. (2012). The democracy barometer: A new
instrument to measure the quality of democracy and its potential for comparative research.
European Political Science, 11(4), 519–536. https://doi.org/10.1057/eps.2011.46.
Caluwaerts, D., & Reuchamps, M. (2014). Strengthening democracy through bottom-up
deliberation : An assessment of the internal legitimacy of the G1000 project, 1–20.
https://doi.org/10.1057/ap.2014.2.
Caluwaerts, D., & Reuchamps, M. (2016). Generating Democratic Legitimacy through
Deliberative Innovations: The Role of Embeddedness and Disruptiveness. Representation,
52(1), 13–27. https://doi.org/10.1080/00344893.2016.1244111.

15
Chambers, S. (2003). Deliberative democratic theory. Annual Review of Political Science, 6(1),
307–326.
Cohen, J. (1989). Deliberation and democratic legitimacy. 1997, 67–92.
Cohen, J. (1999). Reflections on Habermas on Democracy. Ratio Juris, 12(4), 385–416. Retrieved
from https://philpapers.org/rec/undefined.
Conover, P. J., & Searing, D. D. (2005). Studying ‘everyday political talk’in the deliberative
system. Acta Politica, 40(3), 269–283.
Cooke, M. (2000). Five arguments for deliberative democracy. Political Studies, 48(5), 947–969.
https://doi.org/10.1111/1467-9248.00289.
Davies-Roberts, A., & Carroll, D. J. (2014). Assessing Elections. In Advancing Electoral Integrity
(pp. 18–33). New York: Oxford University Press, USA.
Daxecker, U., & Schneider, G. (2014). Election Monitoring: The Implications of Multiple Monitors
for Electoral Integrity. In Advancing Electoral Integrity (pp. 73–93). New York: Oxford
University Press, USA.
Dryzek, J. S. (2000). Deliberative democracy and beyond: Liberals, critics, contestations. Oxford
University Press on Demand.
Dryzek, J. S. (2015). Deliberative engagement: the forum in the system. Journal of Environmental
Studies and Sciences, 5(4), 750–754. https://doi.org/10.1007/s13412-015-0299-8.
Dryzek, J. S., & Niemeyer, S. (2010). Foundations and Frontiers of Deliberative Governance.
Oxford: Oxford University Press.
Elstub, S. (2015). A Genealogy of Deliberative Democracy. Democratic Theory, 2(1).
https://doi.org/10.3167/dt.2015.020107.
Elstub, S., Ercan, S., & Mendonça, R. F. (2016). Editorial introduction: The fourth generation of
deliberative democracy.
Estlund, D. M. (2008). Democratic Authority: A Philosophical Framework. Princeton University
Press. Retrieved from https://philpapers.org/rec/undefined.
Fishkin, J. (1995). Bringing Deliberation to Democracy: The British Experiment. The Good Society,
5(3), 45–49. Retrieved from http://www.jstor.org/stable/20710706.
Fleuß, D. & Helbig, K. (2018): Measuring the Embeddedness of Democratic Innovations and
Representative Democracy with a Systemic Framework. The degree of transmissions as an
indicator for the problem-solving potential of democratic innovations. Working-Paper
presented at ECPR Joint SessionsWorkshop WS05: Can Participatory Reforms Save
Representative Democracy? 10-14 April 2018, Nicosia.
Fleuß, D., Helbig, K., & Schaal, G. S. (2018). Four Parameters for Measuring Democratic
Deliberation : Theoretical and Methodological Challenges and How to Respond, 6(1), 1–11.
https://doi.org/10.17645/pag.v6i1.1199.
Geissel, B., Kneuer, M., & Lauth, H.-J. (2016). Measuring the quality of democracy: Introduction.
International Political Science Review, 37(5), 571–579.
https://doi.org/10.1177/0192512116669141.
Geissel, B., & Newton, K. (2012). Evaluating democratic innovations: Curing the democratic

16
malaise? Evaluating Democratic Innovations: Curing the Democratic Malaise?
https://doi.org/10.4324/9780203155196.
Giebler, H., Ruth, S. P., & Tanneberg, D. (2018). Why Choice Matters : Revisiting and Comparing
Measures of Democracy, 6(1), 1–10. https://doi.org/10.17645/pag.v6i1.1428.
Habermas, J. (1990). Moral Consciousness and Communicative Action. MIT press.
Habermas, J. (1996). Between facts and norms: Contributions to a discourse theory of law and
democracy. MIT Press.
Hendriks, C. M. (2016). Coupling citizens and elites in deliberative systems: The role of
institutional design. European Journal of Political Research, 55(1), 43–60.
Lauth, H.-J., Pickel, G., & Welzel, C. (2000). Demokratiemessung: Konzepte und Befunde im
internationalen Vergleich. Wiesbaden: Springer.
Mansbridge, J. (1999). Everyday talk in the deliberative system. In S. Macedo (Ed.), Deliberative
Politics: Essays on Democracy and Disagreement. Oxford University Press. pp. 1--211.
Mansbridge, J., Bohman, J., Chambers, S., Christiano, T., Fung, A., Parkinson, J., … Warren, M. E.
(2012). A systemic approach to deliberative democracy. Deliberative Systems: Deliberative
Democracy at the Large Scale, 1–26.
Marshall, M. G., Jaggers, K., & Gurr, T. R. (2002). Polity IV project. Center for International
Development and Conflict Management at the University of Maryland College Park.
Martínez i Coma, F., & Frank, R. W. (2014). Expert Judgments. In Advancing Electoral Integrity
(pp. 51–70). New York: Oxford University Press, USA.
Møller, J., & Skaaning, S.-E. (2012). Concept-measure inconsistency in contemporary studies of
democracy. Zeitschrift Für Vergleichende Politikwissenschaft, 6, 233–251.
https://doi.org/10.1007/s12286-012-0128-3.
Morlino, L. (2004). “Good” and “Bad” democracies: How to conduct research into the quality of
democracy. Journal of Communist Studies and Transition Politics.
https://doi.org/10.1080/13523270410001687082.
Niesen, P. (2014). Was heißt Deliberation? Eine theoriegeschichtliche Betrachtung - Deliberative
Kritik - Kritik der Deliberation: Festschrift für Rainer Schmalz-Bruns. In O. Flügel-Martinsen,
D. Gaus, T. Hitzel-Cassagnes, & F. Martinsen (Eds.) (pp. 49–71). Wiesbaden: Springer.
https://doi.org/10.1007/978-3-658-02860-2_3.
Norris, P. (2014). Introduction: Challenges of Electoral Integrity. In P. Norris, R. W. Frank, & F. M.
i Coma (Eds.), Advancing Electoral Integrity (pp. 3–17). New York: Oxford University Press,
USA.
Norris, P., Frank, R. W., & Martínez i Coma, F. (2013). Assessing the Quality of Elections. Journal
of Democracy, 24(4), 124–135. https://doi.org/10.1353/jod.2013.0063.
Norris, P., Frank, R. W., & Martínez i Coma, F. (2014). Advancing electoral integrity. Oxford
University Press.
Norris, P., Frank, R. W., & Martínez i Coma, F. (2014). Measuring electoral integrity around the
world: A new dataset. PS - Political Science and Politics, 47(4), 789–798.
https://doi.org/10.1017/S1049096514001061.

17
Pedrini, S. (2014). Deliberative capacity in the political and civic sphere. Swiss Political Science
Review, 20(2), 263–286. https://doi.org/10.1111/spsr.12074.
Pedrini, S., Bächtiger, A., & Steenbergen, M. R. (2013). Deliberative inclusion of minorities:
patterns of reciprocity among linguistic groups in Switzerland. European Political Science
Review, 5(3), 483–512.
Peter, F. (2008). Democratic Legitimacy. London: Routledge.
Setälä, M. (2017). Connecting deliberative mini-publics to representative decision making.
European Journal of Political Research, 56(4), 846–863. https://doi.org/10.1111/1475-
6765.12207.
Smith, G. (2005). Beyond the ballot: 57 democratic innovations from around the world. Power
Inquiry, XXXIII(2), 81–87. https://doi.org/10.1007/s13398-014-0173-7.2.
Smith, G. (2009). Democratic Innovations: Designing Institutions for Citizen Participation.
Cambridge University Press.
Steenbergen, M. R., Bächtiger, A., Spörndli, M., & Steiner, J. (2003). Measuring Political
Deliberation: A Discourse Quality Index. Comparative European Politics, 1(1), 21–48.
https://doi.org/10.1057/palgrave.cep.6110002.
Steiner, J., Bächtiger, A., Spörndli, M., & Steenbergen, M. R. (2004). Deliberative politics in
action: Analyzing parliamentary discourse. Cambridge University Press.
Thompson, D. F. (2008). Deliberative Democratic Theory and Empirical Political Science.
https://doi.org/10.1146/annurev.polisci.11.081306.070555.