Content uploaded by Jonathan Jackson
Author content
All content in this area was uploaded by Jonathan Jackson
Content may be subject to copyright.
Trust in Justice: Notes on the Development of European Social Indicators
Jonathan Jackson1, Methodology Institute and Mannheim Centre for Criminology, LSE
Ben Bradford, Methodology Institute and Mannheim Centre for Criminology, LSE
Mike Hough, Institute of Criminal Policy Research
Jouni Kuha, Department of Statistics and Methodology Institute, LSE
Sally Stares, Methodology Institute, LSE
Sally Widdop, Centre for Comparative Surveys, London City University
Rory Fitzgerald, Centre for Comparative Surveys, London City University
Maria Yordanova, Center for the Study of Democracy, Sofia, Bulgaria
Todor Galev, Center for the Study of Democracy, Sofia, Bulgaria
Please note: this is an extended, pre-publication version of a paper entitled ‘Developing European
Indicators of Trust in Justice’, which will be published in 2011 in the European Journal of Criminology
Abstract
A social indicators approach to trust in justice recognises that the police and criminal courts need public
support and institutional legitimacy if they are to operate effectively and fairly. We present in this paper
the conceptual and methodological tools to devise indicators of trust and legitimacy across Europe.
First, we outline the conceptual roadmap for a comparative European analysis of trust in justice.
Second, we describe the development process of a 45-item module in Round 5 of the European Social
Survey that fields the survey indicators. Third, we present the findings from the quantitative piloting of
the indicators in the UK and Bulgaria and document the final wording of the measures. Fourth, we
consider the policy implications of the procedural justice model of criminal justice that underpins the
current project: by demonstrating to citizens that they are both trustworthy and possess the authority to
govern, institutions can enhance public cooperation and compliance.
Key words: social indicators, trust in the police, social measurement, police legitimacy, compliance, measurement
invariance
1 Dr Jonathan Jackson, Methodology Institute, LSE, Houghton Street, London WC2A 2AE, United Kingdom, Tel:
0044 207 955 7652, Email: j.p.jackson@lse.ac.uk
1
1. Social indicators of trust in justice and the legitimacy of legal authorities
Public trust and institutional legitimacy help sustain and justify social and political institutions and
arrangements across Europe. Embodying a moral and practical connection between citizens and social
systems, concepts of trust and legitimacy remind us that individuals in a democratic society have the
right to live under a system that operates within the rule of law; that acts effectively and fairly within
commonly accepted norms; and that demonstrates to itself and to citizens its rightful possession of
power. For institutions to flourish, they should demonstrate to citizens that they are trustworthy and that
they possess the authority to govern.
In this paper we outline the conceptual and methodological roadmap for a comparative analysis
of trust in justice. In order to measure levels of trust, legitimacy, cooperation and compliance across the
European Union, we need valid and reliable indicators based upon a comprehensive conceptual scheme.
We present the theory and describe the methodological development process of a 45-item module in
Round 5 of the European Social Survey (ESS), which will provide data on public perceptions of the
police and courts in some 26 countries.
What are social indicators?
Economic indicators are widely used to trace economic development and predict future economic
performance. While the social, cultural or educational provision in a nation depends critically on its
economic condition, economic indicators do not tell us everything about a country’s overall social
condition. To fill this gap we need social indicators (Jowell & Eva, 2009). Combining national
information with transnational objectives, social indicators provide measurements of human well-being
and societal functioning, allowing us to monitor the broader system, identify change, and guide efforts
to improve policy and conditions in areas such as health (e.g. life expectancy rates), crime (e.g.
recorded crime figures) and education (e.g. school enrolment rates).
What constitutes human well-being is a normative and political question, but once consensus is
reached, social indicators can help policy-makers understand the shifting circumstances of life in
different countries. When taking the measure of a nation it is particularly important to assess how
citizens view the way in which their societies operate. As Jowell & Eva (ibid: 318) ask: ‘Do they, for
instance see their societies as generally fair or unfair? Do their country’s institutions inspire trust or
suspicion? Is their system of criminal justice seen to be even-handed or biased? Do their
neighbourhoods feel safe or dangerous?’ This paper outlines indicators of public trust in the police and
criminal courts and public perceptions of the legitimacy of these legal authorities. Developed as part of
a European Commission 7th Framework Programme funded project (www.eurojustis.eu; see also Hough
et al., 2010, and Jackson et al., in press), our guiding premise is that European Member States need to
pay closer attention to issues of confidence, legitimacy and security if they are to achieve balanced and
effective crime policies. An emphasis on public trust and institutional legitimacy can be contrasted with
more short-term and ‘populist’ policies that exploit public feelings for political gain at the expense of
ensuring that the justice system commands legitimacy and that citizens feel safe and secure. Social
indicators of trust in justice are vital for better formulation of the problems facing criminal justice
agencies, as well as more effective monitoring of changes in public attitudes in response to policy
innovation.
2. A conceptual roadmap for indicators of trust and legitimacy
A coherent conceptual framework is needed within which to think about trust in the police and criminal
courts and the legitimacy of these legal authorities. The ESS module measures a number of complex
concepts (see Appendix for the full question wording and
http://www2.lse.ac.uk/methodologyInstitute/whosWho/Jackson/jackson_ESS.aspx for more details).
The focus of this paper is on the two main constructs: trust and legitimacy, each as applied to the police
and criminal courts. We begin with trust.
Trust in the police and criminal courts
Trust becomes relevant when we depend upon the actions of others, when there is risk to us inherent in
the other’s actions (or inactions), and when we feel uncertainty about the other’s behaviour. By
‘bracketing out’ possible future events, through (and with) trust we are anchored in social situations that
would otherwise appear overwhelmingly complicated – especially regarding the motives, intentions and
2
potential actions of others. Trust reduces the complexity of our social world (Luhmann, 1979, 1988;
Giddens, 1991).
Smith (2010) identifies three types of trust. The first is an individual tendency or psychological
disposition to trust ‘most people.’ This is a generalised form of trust, based largely on social learning
and developmental processes. The second is particularised trust. This is the idea that people ‘like me’
can be trusted, but that other groups may not share my moral values; in-groups and out-groups (and
thereby moral communities) are defined by social categories and stereotypes. The third is strategic trust,
which is the belief that specific others have the appropriate motives and intentions. Significant others
can be relied upon to act in one’s own interests in particular situations.
We define trust in the police2 as the belief that the police have the right intentions towards
citizens and are competent to act in specific ways in specific situations (cf. Hardin, 2002). More
strategic than particularised, to trust in the police is to believe that officers have appropriate motives
and are technically competent (in the roles assigned to them within social relationships and systems) to
carry out their fiduciary obligations (that is, in certain situations place the interests of others above their
own) Placing trust in the police involves assuming that officers will act according to expectations
framed by the twin functions of the institution as a public service and a state-sponsored institution of
violence and intrusion. We look to officers to deal with crime and disorder in terms of prevention and
of apprehending those who disobey the law, and to be impartial, and fair and restrained in their use of
authority. This accords with Stoutland’s (2001) definition of trust as expectations of behaviour covering
both abilities and intentions. Her four dimensions of trust are: (1) priorities, (2) competence, (3)
dependability and (4) respectfulness.
Trust refers to general beliefs that the police are performing their social function to the best of
their ability. But it is also important in expectations about future encounters (as well as important in
actual encounters). Coming into contact with the police as victims, witnesses and suspects, we have
vested interests in the outcomes of such encounters. Because officers have a significant degree of
freedom in how they might act, these outcomes are placed at risk from their (mis)behaviour (cf. Tilly
2005). Trust is thus demonstrated, earned, justified in these encounters. It is won and lost by acting
effectively and fairly, by taking the interests of citizens into account, and by communicating and
engaging with citizens (Bradford et al., 2009; Hohl et al., 2010). While people’s views on legal
authorities are not expected to be in a state of constant review, trust is subject to revision through
experience, whether direct, vicarious, or mediated, including:
•single incidents, involving accidents, incompetence, incivility or malpractice;
•perceived changes in levels of police visibility;
•perceived declines in availability and readiness to intervene; and
•increasingly widespread ideas that police do not treat everyone the same.
Yet, most people do not have the expertise, time, or interest needed to fully evaluate police
activity. They are seldom in the position to know what the police and criminal courts are really doing.
Despite often having little substantial knowledge of how officers or judges generally act and behave,
respondents can still answer survey questions about what are quite narrow aspects of performance and
motives (e.g. Sunshine & Tyler, 2003; Skogan, 2006; Reisig et al., 2007; Tankebe, 2010a; Jackson &
Bradford, 2010a). One explanation for this turns on the idea that trust is based on what Hardin (2002)
calls ‘encapsulated interests.’ Applied to the police, this is a general belief that officers understand the
interests of the general public (or not) and take these interests into account when acting (or not). Tyler’s
notion of motive-based trust is also important (Lind & Tyler 1988; Tyler 2006; Tyler & Huo 2002),
where people invest in the police and courts a sense of shared values and motives. Premised on the idea
that the parties involved have shared social bonds (making it possible for the one to imagine,
understand and influence the interests of the other), motive-based trust turns on whether the police are
seen as having the best interests of the community at heart, that they share their goals and priorities
(Jackson & Sunshine, 2007; Jackson & Bradford, 2010a), that they are working on behalf of the
community. Given the important place of crime, punishment and institutions of justice in the political
2 Because of space pressure, we focus on the ‘police.’ However the module also addresses the ‘criminal courts’
and the theory applies equally to this agent of criminal justice.
3
and social life of a nation, people look to the police to deliver order, to defend norms and values, and to
secure a sense of justice for offenders that represents the rights and dignity of citizens.
Intriguingly, this sense of shared motives and interests may be central to trust in justice, in part
because motive-based trust may drive more specific assessments of competence. For Earle (2010: 542):
‘Knowing whether the intentions of the other are good or bad (relative to oneself) is more important
than knowing what the other can do.’ While people may not possess much concrete knowledge of
police performance (relying on the mass media for information), people may nevertheless b elieve that
officers take their interests into account (or not), that the police are ‘on their side’ (or not). This sense of
shared interests, rules and motivates may then influence more concrete beliefs of their competence to
act effectively and fairly in given situations. This may also be reciprocal, where concrete encounters
reveal information about the effectiveness and fairness of the police, which in turn shapes socio-
relational trust based on shared motives and interests.
Legitimacy of the police and criminal courts
Defined most generally, legitimacy is the right to govern and the recognition by the governed of that
right (Beetham, 1991; Coicaud, 2002; Tankebe, 2007). The legitimacy of the criminal justice system
resides most fundamentally in recognition of its right to exist, and in the justification of its authority in
determining the law, governing through the use of coercive force, and punishing those who act illegally.
Although they vary widely, a key feature of many definitions is that legitimacy confers the right to
command and promotes the duty to obey (Weber, 1948; Tyler, 1990). Most importantly, people defer
to, and cooperate with, legitimate authorities because they feel it is right to do so (Tyler, 2006a, 2006b,
2008).
For Weber, the legitimacy of institutions denoted approval or sincere recognition of a norm,
law or social arrangement. The law is legitimate when people see the legal system and its authorities as
providing an appropriate standard of conduct. This authorisation involves the belief that the law is to be
complied with not because of external sanction but because it is the correct standard. Such a citizen-
conferred (‘subjective’ and ‘empirical’3) account states that a system is legitimate when the public grant
it legitimacy. An observer sitting outside the system might find a particular arrangement unjust and
unacceptable, yet they must nevertheless conclude that it is legitimate when those governed believe it to
be so. According to this perspective, to say something is legitimate is to make a factual claim about the
subjective state of mind of particular individuals that belong to one political society (Beetham, 1991;
Caocaud, 2002; Tankebe, 2010b).
A normative concept of legitimacy sets out further, more ‘objective’ criteria according to which
an authority or institution is legitimate (Hinsch, 2008). Normative legitimacy refers not to the subjective
state of mind of the governed, but to whether the actions of authorities meet certain substantive
requirements (usually requirements of justice and rationality, for which objective evidence can be
adduced). Operating at the level of the institution, legitimacy is here a property of performance and
structure (captured by national-level statistics of efficiency, accountability, legality, and so forth). Any
normative conception of legitimacy has to describe why meeting these criteria confers authority on
institutions or persons. Normative legitimacy means substantive recognition that the truth (or validity)
of these arrangements is right and just.4
What qualities must an institution possess to warrant its justified authority? Some political
philosophers have followed a loosely Weberian tradition, situating legitimacy in individual’s
perceptions that the rule of an authority is justified. Here there might be just one criterion of legitimacy
for the justice system: that individuals feel an obligation to obey the rules set forth by the legal system
and enforced by legal authorities such as the police. According to the empirical concept of legitimacy,
we might thus say that the police are legitimate when citizens feel obligated to obey police directives.
3 Tankebe (2010b) refers to this as ‘mass or exogenous legitimation of power’.
4 Tankebe (2010b) highlights another source of legitimation: namely, the sense of legitimacy felt by the holders of
power themselves. Endogenous legitimacy is ‘…the attempts of the powerful to justify the rightness of their
authority in their own eyes and to cultivate their own sense of identity’ (p. xxx). This is particularly important
with regard to police discretion. Do officers on the ground follow their own internal rules because of their own
sense of the organisation’s legitimacy (Tyler et al., 2007)? Tankebe (2010b) argues that what is needed is ‘…a
recognition that, whilst claims to legitimacy may be authenticated by rulers themselves for their own governing
and justificatory needs, equally critical – depending on the performance task an authority system faces – are the
subjective normative aspects and verification of those claims by the citizenry.’ (p. xxx).
4
Tyler’s work would then suggest that legitimacy is won and lost partly through the experience of
procedural justice and injustice. Yet, others maintain that, because legitimacy is granted by the
individual to the institution, it must rest in part on the value judgements of those individuals. This is a
decision by the individual, whether conscious or not, that the institution shares a certain moral or ethical
position (Beetham, 1991). Legitimacy is not just about power, it is also about the justification of power.
Such accounts point to normative concepts of legitimacy, and extend our area of interest to perceptions
of the justice system that provide a normative justifiability of power.5
A guiding premise of the ESS module is that judgements among individuals about the
legitimacy of an institution are based to some degree on assessments of the congruence between its
goals, practises and behaviours, and their own. Legitimacy becomes a kind of ‘moral alignment’
between individuals and the legal/criminal system around them. Beetham (ibid.) adds a third criterion:
the perceived legality of legal authorities. Legitimacy, for him, turns on the moral justifiability of the
power relations bound up in the state and its justice system – conformity to people’s values, its ability
to satisfy public interests and normative expectations, and in the legality of power (cf. Tankebe, 2009:
1280-1281). All considerations of legitimacy must involve a normative, ideological or moral element,
he argues: those granting legitimacy always do so on the basis that it is an expression of common
shared values.
Following Tyler (2006, 2008), Beetham (1991) and Tankebe (2010b), our framework is
premised on the idea that the justice system can be considered empirically legitimate when individuals
governed by it feel (a) an obligation to obey the authority (as a special case of Beetham’s notion of
expressed consent6), (b) that the authority expresses shared morals – people justify the existence of legal
authorities when they judge that they enact, defend and strengthen the moral values they themselves
share, and (c) that the justice system follows its own internal rules as these are understood by individual
citizens. We reason that the moral alignment between citizens and police officers (for example)
provides a measure of whether the police are to seen to operate according to a shared ethnical and moral
framework; but also that such alignment provides the police with the moral authority to act as a source
of guidance or an exemplar of proper conduct.7
3. European indicators of trust and legitimacy: Process of development
In summary, our social indicators perspective is premised on the idea that citizens have a right to expect
the police and criminal courts to be effective, fair and responsive to local needs and priorities. We
define trust as the belief that the police and courts have the right intentions and are competent to do
what citizens trust them to do, i.e. to act in ways that reflect effectiveness, fairness and responsiveness
to local needs and priorities (cf. Stoutland, 2001). Implying the recognition of a shared moral
commitment – where police and criminal courts take the interests of citizens into account – trust is
revealed in beliefs about current affairs, but also in expectations about future behaviour. We define
legitimacy as (a) expressed consent, (b) normative justifiability of power and (c) legality of action. We
also assume that the legitimacy of the justice system is both citizen-conferred and system-conferred, so
5 Tankebe (2010b) refers to normative justifiability as being ‘…the need for rules and police practices to be rooted
in the shared beliefs of society’, which includes working towards the general interests and collective goods of
society, and limiting the use of power through a restrained use of authority. Legitimacy is thus sustained by
avoiding unnecessary and discriminatory restriction of the freedom of citizens and avoiding the deployment of
power against political opponents. A parallel can be drawn here with Tyler & Blader’s (2003: 359) group
engagement model, which argues that the procedural injustice experienced by those on the receiving end of racial
profiling ‘…communicates marginality and exclusion from important protections that are extended to most other
group members – for example, “freedom from arbitrary arrest and seizure”.’
6 We might even include acts of cooperation in the very definition of legitimacy. Bradford & Jackson (2010)
argue that acts of cooperation imply recognition of the police role in maintaining order and ‘fighting crime’: the
recognition of the right of the police to act in, and gain control over, certain social situations. Expressed consent
may by found not only in acts of blunt obedience to officer’s instructions, but also in general public deference, in
offers of assistance, and in an acceptance (indeed encouragement) of officers’ presence at specific moments.
Moments of cooperation may thus be both an outcome of the conferral of authority and an act of legitimation.
7 We consider it an open and empirical question whether obligation to obey and moral alignment scale together, or
whether they constitute separate latent constructs. It may well be the case that they scale together in some
countries (and therefore would be combined into one measurement part of a broader structural model) but do not
in other countries (and therefore just be separate parts of the structural model).
5
legitimacy is to be captured not just by survey data of European citizens, but also by national data on
accountability, transparency, democratic principles, corruption, and so forth. The criminal justice
system can be considered normatively legitimate when their actions conform to certain minimum
standards with regards to fairness, efficiency and honesty. These standards refer to expressed consent
(e.g. defined as national-level ESS estimates of intentions to cooperate with police); normative
justifiability of power (e.g. defined as levels of democratic accountability and transparency); and
legality of action (e.g. defined as levels of cooperation and abuse). For further discussion see: Hough et
al. (2010), Jackson et al, (in press) and Jackson & Bradford (2010b).
These central tenets are expected to hold in many situations, but we do expect local social and
cultural contexts to play an important role. The countries under consideration in the ESS module will
provide a specific set of factors that influence how citizens view their criminal justice agencies: what it
means to trust them; what the relationship between trust and legitimacy is; and how the connection
between legitimacy and compliance is constituted.
4. From concepts to measures in a cross-national context: The European Social Survey
It is important to stress the methodological guidelines underpinning the trust in justice module. The ESS
module works within a tripartite structure set out by previous work on social indicators of poverty and
social exclusion (Atkinson et al., 2002). Primary indicators (level-1) constitute a small number of lead
survey measures of trust and legitimacy. Secondary indicators (level-2) support the primary indicators
by providing greater detail using survey measures of the various dimensions of trust and legitimacy.
Finally, country-based indicators (level-3) highlight local specificities and help interpret survey data. A
basket of indicators is then developed for monitoring trust in justice, permitting the comparison
between countries using standardised and agreed measures. Mapping onto the relevant conceptual
domains of confidence and legitimacy, these questions must be balanced, mutually consistent (with
proportionate weighting of single indicators), transparent and accessible. Specific methodological
principles guide the selection of indicators: they need to have a clear and consensually achieved
definition; they need to be robust and statistically valid; they need to be able to measure the impact of
policy interventions; they need to have an international foundation so that data are comparable across
Member States; they need to be subject to revision if necessary; and they should not impose too large a
burden in their collection.
In the rest of this paper we focus on the development of the level-1 and level-2 indicators of
trust in justice, which were developed in part through the ESS process. Figure 1 illustrates the s tages of
question development and design in the ESS (Saris and Gallhofer, 2007; Fitzgerald and Jowell, 2008).
At Stage 1, proposals are received from question designers using the ESS Question Module Design
Template (Fitzgerald, 2007). These proposals outline the substantive measurement aims of the module
and its theoretical framework, and identify the key concepts and dimensions. The template is then used
throughout the questionnaire design process to document decisions and to ensure that the process is
continually informed by the agreed measurement aims of the module. Once the conceptual basis of the
module is agreed, the proposed questions are reviewed by a multi-disciplinary cross-national specialist
panel (stage 2) and assessed by the Survey Quality Predictor Program (SQP) (Saris and Gallhofer,
2007). SQP is used to estimate the reliability and validity of new items based on the structural form of
the items (stage 3).8 The questions are then revised by the question designers (stage 4) and following
this, stages 1-4 are repeated reflecting the iterative nature of the question design process. The ESS
National Coordinators are then consulted on both substantive and translation issues (stage 5).
In the next stage the proposed questions are tested in a large-scale, two-nation quantitative pilot
including some split ballot MTMM experiments (stages 6 and 7 – finding of which are outlined below)
for particular concepts. Extensive analysis of the pilot data - including examination of item non-
response, scalability, factor structure, correlations, analysis of the MTMM experiments, assessment of
translation as well as consideration of the Pilot Reports from the pilot countries (stage 8) - feed into
further specialist review of the proposed questions (stage 9). This is subsequently followed by further
consultation with the National Coordinators (stage 10) before the final source questionnaire is produced
(stage 11). The final source questionnaire contains annotations of words and phrases where their
meaning within the context of the survey question is thought to need clarification to aid translation. The
8 SQP is usually only used once.
6
questionnaire is then translated in all participating countries and all languages spoken as first language
by 5% or more of the population before fieldwork commences.
The quantitative pilot
Conducted in January and February 2010 in Bulgaria (by the Agency for Social Analyses in Bulgaria)
and the UK (by Ipsos-MORI), 820 face-to-face interviews were carried out (400 in Bulgaria and 420 in
the UK9). The objectives were to test the wording of new and amended questions to make sure they
were correctly understood by respondents; to test the expected interrelationships between variables in
the data; to identify suitable candidates for deletion in order to meet the target number of questions; and
to check the timing of the new questions. All interviewers working on the pilot were briefed before
conducting fieldwork and were asked to complete a short questionnaire outlining any problems
experienced during the interview. A selection of interviewers in both countries were also invited to a
debrief where researchers from the fieldwork agencies facilitated a discussion about the questionnaire.
Members of the ESS research team observed live interviews conducted in the UK. Interviewer
debriefing reports were prepared by the respective fieldwork teams. They are drawn upon later in this
paper. Let us turn first to the country contexts of the pilot.
Different contexts: UK and Bulgaria
Any comparative piece of survey research needs to demonstrate equivalence of concepts and measures
if the conclusions are to be valid and the empirical findings are to be convincing. Conceptual
equivalence refers to the meaning of theories and concepts across diverse concepts: do they apply and
are they equivalent in different social, cultural and political settings? Measurement equivalence refers to
the comparability of measures: do the scales reflect the same underlying concepts? The specific
contexts under investigation influence the adaptation of concepts, development of scales, translation of
wording and empirical assessment of measurement equivalence
The UK was selected as one pilot country since it provided a test of the questionnaire in the
‘source’ language that had been used when designing the module. Bulgaria was selected to provide a
good contrast as a former communist country as well as for reasons of cost effectiveness. Testing in a
former communist country was deemed to be important for historical reasons, e.g. existing ESS data
suggested that levels of trust in the police and legal system were lower in Bulgaria and some other
former East European countries than elsewhere in Europe. Feedback from some ESS National
Coordinators (who run the survey conducted in each specific country) had also suggested that
corruption was a salient issue in regard to criminal justice.
In the UK, for instance, there may be something specific to the relationship between individuals
and the police – in particular, the police’s position as representative of nation, state, community and
even an idealised national past (Reiner 2010; Loader and Mulcahy, 2003; Jackson & Bradford, 2009).
Although the strength of these images may have been attenuated, not least by the long decline in public
trust in almost all state institutions, a persistent residue remains that may powerfully affect what it
means to trust the police. Many in Britain today would probably still concur with the opinions of their
parents and grandparents, who in research carried out for the 1962 Royal Commission on Policing
indicated overwhelmingly that they thought the British police were the best in the world (Loader and
Mulcahy 2003). That such an opinion might be based in large part on negative stereotypes of police in
other countries would only serve to underline that the image of the British police is tightly bound to
notions of identity, even patriotism and pride.
Bulgaria, by contrast, continues to experience high levels of corruption, an ineffective justice
system, and high levels of organized crime. Cases of political corruption, flagrant conflicts of interest
and the use of public resources for personal benefit continue, and are considered by the public as “crime
without punishment”. High levels of public mistrust in state institutions and justice persist, and common
public dissatisfaction with quality of life and welfare (Tilkidjiev, 2010). Indeed, data collected in the
9 The target population was specified to be the same as in the ESS main stage; namely individuals in residential
property aged 15 and over with no upper age limit. Due to funding constraints the samples were not based on the
representative principles of random probability sampling. Instead the samples were designed to broadly mirror the
population of each country on key demographic characteristics. Quota sampling strategies were employed in each
country although the precise sampling specification differed slightly. Respondents from both countries were
selected according to the following quotas: age, gender and working status (working v. not working). In the UK,
manual v. non-manual occupation was also used as an additional quota.
7
two last ESS rounds (and through other international studies10) show that Bulgarian citizens have the
lowest degree of satisfaction with life in general and with the activity of key state institutions in the EU.
Trust in justice thus operates against a backdrop of Bulgaria as a ‘low-trust society.’11
Method and analytical strategy
In the context of the UK and Buglaria, we assess the measurement equivalence of our scales of trust and
legitimacy through a statistical analysis of data from the pilot. Multiple-item survey scales are
commonly analysed using latent variable models, which include such methods as linear factor analysis
and structural equation modelling, latent trait (item response) models, and latent class models (for
overviews, see Bartholomew and Knott, 1999, Bollen, 1989, McCutcheon, 1987, and Skrondal and
Rabe-Hesketh, 2004). All of these operationalise the basic idea that the individual survey items are
regarded as fallible measures of some directly unobservable (i.e. latent) constructs, and that it is those
latent constructs that are the main focus of interest. We can also use latent variable modelling to test the
‘measurement equivalence’ of the items, examining whether the questions seem to be measuring the
same thing in both UK and Bulgaria. Do the measures ‘work’ equally well in our different contexts?
Examining the measurement equivalence of the indicators of the theoretical concepts, we
present (because of limited space) models for only three constructs, each measured by three survey
items. The constructs are (a) trust in police effectiveness, (b) perceptions of police corruption and (c)
perceived risk of sanctions. The English wordings of the questions and the response options are given in
Table 1.
INSERT TABLE 1 ABOUT HERE
Results
Latent variable models are used to examine both how the survey items measure the latent constructs,
and how the latent variables relate to each other and to observed explanatory variables. As an example,
Table 2 shows estimates for a model for the three items on trust in police effectiveness, conditional on
one latent variable and with country (UK vs. Bulgaria) used as an explanatory variable. The model is a
linear factor analysis model, of the kind defined by equations (1) and (2) below, and represented by
diagram (1) of Figure 1. The measurement part of the model in Table 2 shows that scores on all three
items are positively associated with the latent variable, high values of which can thus be interpreted to
represent high levels of trust in police effectiveness. Comparing the two countries, the variability of
confidence levels is roughly similar, but there is a statistically significant difference in the means. The
average level of perceived effectiveness is around 0.8 units lower among the Bulgarian than the UK
respondents, on a similar 11-point scale as the items.
INSERT FIGURE 1 ABOUT HERE
INSERT TABLE 2 ABOUT HERE
Before we place too firm an interpretation on such results, we need to consider whether the
survey responses from different countries are actually comparable. This is one of the key concerns of
our pilot: are the measures measuring equivalent ‘things’ in the two different countries? Latent variable
models can be used to examine such cross-national comparability of survey measurement. We can
illustrate some ways of doing so with models for the three sets of items in Table 3. We use linear factor
analysis models to analyse the items. Suppose
ij
η
denotes the value of a single latent variable
η
for
respondent j in country i, and let Yijk denote the value of survey item k for that respondent. Suppose that
items k=1,…K are all measures of
η
(in our examples K=3). The models we consider are represented
as diagrams in Figure 1. In all of the models, we assume about the latent variable that
ij
η
is normally distributed with mean
)(i
µ
and variance
)(i
τ
. (1)
The superscript (i) indicates that in general the parameters may have different values in different
countries, i.e. that there may be differences in the average level and variability of the latent construct
10 European Quality of Life Survey 2003 and 2007; European Value Survey 2008 (cited in Tilkidjiev 2010)
11 The lowest trust is expressed towards political institutions - politicians, parties and parliament, followed by the
judiciary and the police.
8
among individuals in different countries. This is typically a substantively interesting possibility that we
want to allow in the model; it is indicated by the arrow from “Country” to “Latent variable” in Figure 1.
The measurement model is a model for the items Yijk given the latent variable
ij
η
. We will
consider different possibilities for this model. The simplest of them states that
ijkijkkijk
Y
δηλν
++=
(2)
for all items k=1,…K. Here
ijk
δ
is a normally distributed random variable with mean 0 and variance
kk
θ
. All of the parameters
k
ν
,
k
λ
, and
kk
θ
of this model are the same in all of the countries. The
measurement is then said to be fully equivalent across the countries. This case is represented by
diagram (1) of Figure 1, where the equivalence is indicated by the fact that the diagram contains no
arrows directly from country to the survey items.
If any parameters of the measurement model do vary between countries, the measurement is not
fully equivalent. Equivalence is “partial” if it holds for some, but not all, the items. For example in
diagram (2) of Figure 1, measurement of item 3 is non-equivalent. We will consider two forms of non-
equivalence, depending on which parameters of the measurement model it affects. In the first, only the
intercept terms
k
ν
may vary between countries, so that
ijkijk
i
kijk
Y
δηλν
++=
)(
(3)
for one or more items k. In Figure 1, this “direct effect” from country to an item is represented by the
arrow labelled (a). The second possibility is that both the intercepts and the loadings
k
λ
may vary, so
that for some items
ijkij
i
k
i
kijk
Y
δηλν
++=
)()(
. (4)
The variation in the loadings can be thought of as an interaction between country and the latent variable
in the measurement model; it is represented by the line labelled (b) in Figure 1. In both models (3) and
(4) we assume that the measurement variances
kk
θ
are the same across countries, but other choices for
them are also possible.
These ideas of equivalence and non-equivalence are not limited to factor analysis. They apply
also to other kinds of latent variable models. For example the items on perceived risk of sanctions have
only four response options. So we might have chosen to model them as ordinal categorical variables
rather than as effectively continuous as in factor analysis. This would lead to the use of latent trait
models for ordinal items. The form of the measurement model would then be different. But the question
of measurement equivalence would still be whether its parameters are the same across countries. The
methods of model comparison for examining measurement equivalence are also essentially similar for
all types of latent variable models.
A summary of the results of our analyses is given in Table 3. The procedure for assessing the
measurement equivalence of items was as follows. We examined one item at a time. As an example,
consider the police effectiveness item EFF2, which is shown second in Table 3. First, a model of full
equivalence was fitted, for the three items on police effectiveness and with one of the other items (here
EFF1) as the anchor item. This is the model given by equations (1) and (2), and is labelled
“Equivalence” in the table. The estimates of the intercepts
)(i
ν
and loadings
)(i
k
λ
of the measurement
model for item EFF2 are shown in this row of the table, separately for UK and Bulgaria. For the
equivalence model, these are by definition equal for the two countries. Estimates of the rest of the
parameters are not shown in the table; for the equivalence model for police effectiveness, they are the
estimates shown in Table 2.
INSERT TABLE 3 ABOUT HERE
The row labelled “Direct effect” shows estimates of the intercepts and loadings of the item
being tested when measurement model is (3) for that item (and the equivalence model (2) holds for the
other two items). Here the intercept for EFF2 is 5.16 for the UK but 4.93 for Bulgaria. Finally, the row
labelled “Interaction” shows estimates when model (4) is used for the item being tested, i.e. when both
the intercept and slope of that item vary by country. These three models are then repeated for the other
9
two trust in police effectiveness items in turn, and for models for the items on perceived risk of
sanctions and perceptions of police corruption similarly.12
The different models for each item are compared using standard statistical likelihood ratio (LR)
tests (see Agresti and Finlay 2009, S. 15.3). This test always compares a pair of models, where one
model is a constrained version of the other. Three comparisons are considered here: between the
Equivalence and Direct-effect models; between the Direct-effect and Interaction models; and directly
between the Equivalence and Interaction models. The P-values of these tests are shown in Table 4. A
small P-value indicates that the less constrained of the two models fits substantially better than the more
constrained one, and a large P-value that the models are not significantly different. For example, in the
comparison of the Equivalence and Direct-effect models for item EFF1 we have P=0.12, which
indicates that the less constrained Direct-effect model does not fit significantly better than the more
constrained Equivalence model. For CORR1 we have P<0.01 for the same comparison, so the opposite
conclusion is reached. In other words, when the intercept parameters of the measurement model of an
item are allowed to have different values in the two countries (while keeping the measurement models
of the other two items for the same concept the same across countries), their difference is statistically
significant for CORR1 but not for EFF1. The measurement is cross-nationally equivalent in this respect
for EFF1 but not for CORR1.13
Lessons drawn
Statistical conclusions from these analyses are relatively clear-cut. We can combine these conclusions
with findings from the interviewer debriefings in order to validate measurement tools and highlight
potential problems. First, no significant deviations from measurement equivalence were found for the
measures of trust in police effectiveness. Consequently, these items appear to work in comparable ways
for both British and Bulgarian respondents. Consistent with this, the interviewer debriefings indicated
that respondents found these questions relatively straight-forward. But two caveats should be drawn.
First, according to the debriefing report drafted by the Agency for Social Analyses in Bulgaria, a
significant number of Bulgarian respondents felt that their answer to EFF3 ( “If an emergency were to
occur near to where you live and the police were called, how quickly do you think they would arrive at
the scene?”) depended on the type of emergency. EFF3 was subsequently changed to ‘If a violent crime
or house burglary were to occur near to where you live and the police were called, how slowly or
quickly do you think they would arrive at the scene?’ In translation to Norwegian it further emerged
that the police would react immediately to the former (violence) but hardly arrive at the scene in the
latter case (burglary), so the final item specified only violent crime. Second, a good number of UK
respondents – when asked whether they had difficulties with certain questions in the module – felt that
they did not have much concrete knowledge about the police performance with respect to EFF1 (“How
successful do you think the police are at preventing crimes in [country] where violence is used or
threatened?” and EFF2 (“How successful do you think the police are at catching people who commit
burglaries in [country]?”). Yet they still felt comfortable giving an answer. Perhaps respondents were
drawing upon a broader set of expectations about the assumed motives and interests of the police
(Jackson & Sunshine, 2007; Jackson & Bradford, 2010a), inferring competence from shared values.
Second, all three measures of perceptions of police corruption showed evidence of non-
equivalence in the intercepts of the measurement models between the countries, but not their loadings
between the countries. The three indicators were designed to jointly measure the unobserved construct
12 The models for perceived police corruption involved a minor complication in that the estimated residual
variance of CORR2 in the invariance model was negative. This so-called “Heywood case” occurs sometimes in
factor analysis models when the sample size and/or number of items is small (see Bartholomew and Knott 1999,
S. 3.18). We resolved it by fixing, in all three models for a given item, the residual variance of the anchor item at
the value it obtained for the Direct-effect model when freely estimated. This exercise amounts to a further
constraint on the variance of the latent variable. It has little effect on the conclusions of the model comparisons
discussed below, in that these were qualitatively unchanged even when the Heywood case was allowed to stand.
13 This effectively assesses one item at a time against the benchmark where the other two items are assumed
invariant. The interpretation of such a procedure is somewhat ambiguous when more than one of the items may be
non-equivalent. Instead, we could then relax the model further, by testing the invariance of one item within a
model where one other item was already non-invariant. With three items, this means that the common scale of the
latent variable is established by one item only, so the interpretation of the model is somewhat questionable. For
the items considered here, the conclusions about their measurement invariance remain unchanged after such tests.
10
of ‘perception of police legality of action’. But the lack of measurement equivalence suggested that the
items were not measuring ‘the same thing,’ in some sense. The intercepts were different in Bulgaria
compared to the UK. In particular, in Bulgaria the intercept for bribery was larger in magnitude than it
was in the UK. This suggests that bribery is a bigger objective (and subjective) problem in Bulgaria.
Moreover, the Bulgarian interviewer debriefing revealed that the bribery question ‘touched upon a very
sensitive topic.’ (In the UK interviewer debriefing, no problems were raised by these three questions,
although again there was some concern that respondents had little direct knowledge of police activities).
Given these findings, the question design team went back to the drawing board with the measures. The
focus was changed to bribery and what political scientists call ‘state capture’ (i.e. the undue influence of
government and big business on the police and criminal courts).
Third, the items on perceived risk of sanctions also deviated from equivalence: for SANCT1
intercepts were comparable in two countries but loadings only marginally so; for SANCT2 intercepts
were comparable but loadings not; and for SANCT3 neither loadings nor intercepts were comparable.
This suggests that perceptions of the risk of sanction related to the three crime types considered here are
not correlated equally strongly with each other, and with overall risk of sanction, in the two countries.
The interviewer debriefings further indicated that people found the basic item of perceptions of being
caught and punishment to be clear and understandable, but that the specific criminal acts were more
problematic. In particular, in both countries (but especially Bulgaria) there was confusion as to the
notion of ‘paying cash with no receipt to avoid paying VAT.’ This, alongside the ESS expert review
process, suggested that we needed to changed the crime types (with implications not just for the
perceived risk of sanction indicators but also the personal morality and compliance scales). The final
questions used covered the following crimes:
•make an exaggerated or false insurance claim;
•buy something you thought might be stolen; and,
•commit a traffic offence like speeding or crossing a red light.
Similar analyses were conducted for all the other scales, and where appropriate, improvements
were made on the basis of these analyses (as well as the interviewer debriefings and the continuation of
the expert review process). However, space precludes us reporting full details. (Please contact the first
author). The final questions can be found in the Appendix.
5. The significance of trust in justice for crime-control policies across Europe
Given the relevance of the ESS module to models of cooperation and social regulation, we finish with
some brief thoughts on the importance of crime-control policy to our work. By measuring contact, trust,
legitimacy, cooperation and compliance (as well as perceived risk of sanction and personal morality),
we can test Tyler’s procedural justice model across diverse social and political contexts. We can test
whether public cooperation and compliance is higher when institutions enjoy the right to govern and the
recognition of that right by the populace (Tyler, 2006b). We can ask whether people who are aligned
with their society’s legal structures are also more likely to abide by the rules of that society and assist
the police and courts through reporting crimes, identifying culprits and giving evidence (Sunshine &
Tyler, 2003; Tyler et al., 2010; Tyler, 2011)?
A key idea in procedural justice theory is that normative compliance — the ‘ought to’
reasoning — is secured more economically, and is more stable over time, than instrumental compliance
(the ‘what’s the risk / benefit for me’ reasoning). Legitimacy encourages people to follow the rules and
cooperate with authorities, not out of fear of punishment, but because they believe they ought to (Tyler,
2011). Fair procedures and the restrained use of authority mean there is less need for costly and
minimally-efficient crime-control policies that focus on surveillance, intrusion, capture and punishment
(Tyler, 2008). Compared to crime-control policies based around deterrence and instrumental models of
cooperation (which seek to demonstrate to citizens that the police are effective and the courts are
punitive), a values-based model may be a more efficient and effective basis for encouraging people to
bring their behaviour in line with law and legal institutions (Tyler, this volume). If most people obey
the law without the active force of deterrence and punishment, then the police can target the hard-core
whose behaviour is motivated not by values, but by the rational choice of likelihood being caught and
11
the severity of subsequent punishment. Without voluntary compliance, and without most people
obeying most laws most of the time, there is significant cost for criminal justice.
Prior research on procedural justice and compliance has been explored primarily in English-
speaking cultures: it has not been sufficiently tested in the kind of diverse societies and legal
jurisdictions found across Europe. Using data from the ESS, we hope that the European social scientific
community can now put to rigorous empirical test the validity of a value-based approach to legal
regulation that designs institutions in ways that promote the development of social values such as
legitimacy.
References
Agresti, A. and Finlay, B. (2009). Statistical Methods for the Social Sciences (4th ed.). Upper Saddle
River, NJ: Pearson.
Atkinson T, Cantillon B, Marlier E, Nolan B (2002). Social Indicators: The EU and Social Inclusion.
Oxford University Press, Oxford.
Bartholomew D. J. and Knott M. (1999). Latent Variable Models and Factor Analysis. London: Arnold.
Beetham, D. (1991). The Legitimation of Power. London: Macmillan.
Bollen, K. A. (1989). Structural Equations with Latent Variables. New York: Wiley.
Bradford, B. and Jackson, J. (2010). ‘Cooperating with the Police: Social Control and the Reproduction
of Police Legitimacy’, working paper available at SSRN: http://ssrn.com/abstract=1640958
Bradford, B., Jackson, J. and Stanko, E. (2009). ‘Contact and Confidence: Revisiting the Impact of
Public Encounters with the Police’, Policing and Society, 19, 1, 20-46.
Coicaud, J.-M. (2002). Legitimacy and Politics. Cambridge: Cambridge University Press.
Earle, T. C. (2010). ‘Trust in Risk Management: A Model-Based Review of Empirical Research’, Risk
Analysis, 30, 541–574.
Giddens, A. (1991). Modernity and Self-Identity. Self and Society in the Late Modern Age. Cambridge:
Polity.
Hardin, R. (2002). Trust and Trustworthiness. New York: Russell Sage.
Hinsch, W. (2008). ‘Legitimacy and Justice’, in Kuhnelt, J. (ed) Political Legitimation without
Morality? London: Springer
Hohl, K., Bradford, B. and Stanko, E. A. (2010). ‘Influencing Trust and Confidence in the Metropolitan
Police: Results from an Experiment Testing the Effect of Leaflet Drops on Public Opinion’, British
Journal of Criminology, 50,3, 491–513.
Hough, M., Jackson, J., Bradford, B., Myhill, A. and Quinton, P. (2010). ‘Procedural Justice, Trust and
Institutional Legitimacy’, Policing: A Journal of Policy and Practice, 4, 3, 203-210.
Jackson, J. and Bradford, B. (2009). ‘Crime, Policing and Social Order: On the Expressive Nature of
Public Confidence in Policing’, British Journal of Sociology, 60, 3, 493-521.
Jackson, J. and Bradford, B. (2010a). ‘What is Trust and Confidence in the Police?’, Policing: A
Journal of Policy and Practice, 4, 3, 241-248.
Jackson, J. and Bradford, B. (2010b). ‘Police Legitimacy: A Conceptual Review’, National Policing
Improvement Agency Wiki, available at SSRN: http://ssrn.com/abstract=1684507.
Jackson, J. and Sunshine, J. (2007). ‘Public Confidence in Policing: A Neo-Durkheimian Perspective’,
British Journal of Criminology, 47, 2, 214-233.
Jackson, J., Bradford, B., Hough, M. and Murray, K. H. (in press). ‘Policing by Consent and Public
Compliance with the Law: Notes on Legal Legitimacy and Cynicism’, in Crawford, A. and
Hucklesby, A. (eds.) Legitimacy and Compliance in Criminal Justice. Routledge.
Jowell, R. and Eva, G. (2009). ‘Happiness is not Enough: Cognitive Judgements as Indicators of
National Wellbeing’, Social Indicators Research, 91, 3, 317-328.
Karstedt, S. and Farrall, S. (2006). ‘The Moral Economy of Everyday Crime: Markets, Consumers and
Citizens’, British Journal of Criminology, 46,6, 1011-1036,
Lind, E. and Tyler, T. (1988). The Social Psychology of Procedural Justice. New York: Plenum Press.
Loader, I. and Mulcahy, A. (2003). Policing and the Condition of England. Oxford: Oxford University
Press.
Luhmann, N. (1979). Trust and Power. New York: Wiley.
Luhmann, N. (1988). ‘Familiarity, Confidence, Trust: Problems and Alternatives’, in Gambetta, D (ed.)
Trust: Making and Breaking Cooperative Relations. Oxford: Blackwell, pp 95-107.
12
McCutcheon, A. L. (1987). Latent Class Analysis. Newbury Park: Sage.
Reiner, R. (2010). The Politics of the Police (4th edition). Oxford: Oxford University Press.
Reisig, M. D., Bratton, J. and Gertz, M.G. (2007). ‘The Construct Validity and Refinement of Process-
based Policing Measures’, Criminal Justice and Behaviour, 34, 1005-1027.
Saris, W. E. and Gallhofer, I. N. (2007) Design, Evaluation and Analysis of Questionnaires for Survey
Research. Wiley, Hoboken, New Jersey.
Skogan, W.G. (2006). ‘Asymmetry in the Impact of Encounters with the Police’, Policing and Society,
16, 2, 99-126.
Skrondal, A. and Rabe-Hesketh, S. (2004). Generalized latent variable modeling: multilevel,
longitudinal and structural equation models. Boca Raton: Chapman & Hall/CRC.
Smith, S. S. (2010). ‘Race and Trust’, Annual Review of Sociology, 36, 453-475.
Stoutland, S. E. (2001). ‘The Multiple Dimensions of Trust in Resident/Police Relations in Boston’,
Journal of Research in Crime and Delinquency, 38: 226–256.
Sunshine, J. and Tyler, T.R. (2003). ‘The Role of Procedural Justice and Legitimacy in Shaping Public
Support for Policing’, Law and Society Review, 37, 3, 513-548.
Tankebe, J. (2007). Policing and Legitimacy in a Post-Colonial Democracy: A Theoretical and
Empirical Study of Ghana. Ph.D. Thesis (Unpublished), University of Cambridge
Tankebe, J. (2009) ‘Public Cooperation with the Police in Ghana: Does Procedural Fairness Matter?’
Criminology 47(4): 1265 – 1293
Tankebe, J. (2010a). ‘Public Confidence in the Police: Testing the Effects of Public Experiences of
Police Corruption in Ghana’, British Journal of Criminology, 50, 2, 296-319.
Tankebe, J. (2010b) ‘Legitimation and Resistance: Police Reform in the (un)making’ in L. K. Cheliotis
(ed.) Roots, Rites and Sites of Resistance: The Banality of Good. Basingstoke: Palgrave Macmillan.
Tilkidjiev, N. (2010). ‘Trust and Well-being’, in: Tilkidjiev, N. and Dimova, L. (eds) Well-being and
Trust: Bulgaria in Europe? Comparative Analysis after ESS Rounds 2006/2009. Sofia: Iztok-
Zapad Publishers, 33-62
Tilly C. (2005). Trust and Rule. Cambridge: Cambridge University Press.
Tyler, T. R. (2006a). ‘Psychological Perspectives on Legitimacy and Legitimation’, Annual Review of
Psychology, 57, 375-400.
Tyler, T. R. (2006b). Why People Obey the Law (second edition). New Haven: Yale University Press.
Tyler, T. R. (2008). ‘Psychology and Institutional Design’, Review of Economics and Law, 4, 3, 801-
887.
Tyler, T. R. (2011). Why People Cooperate? The Role of Social Motivations. New Jersey: Princeton
University Press.
Tyler, T. R. and Huo, Y. (2002). Trust in the Law: Encouraging Public Cooperation with the Police
and Courts. New York: Russell Sage Foundation.
Tyler, T. R., Callahan, P. and Frost, J. (2007). ‘Armed, and Dangerous(?): Can Self-regulatory
Approaches Shape Rule Adherence among Agents of Social Control’, Law and Society Review, 41,
2, 457-492.
Tyler, T. R., Schulhofer, S. J. and Huq, A. Z. (2010). ‘Legitimacy and Deterrence Effects in
Counterterrorism Policing: A Study of Muslim Americans’, Law & Society Review, 44, 2, 365-402.
Weber, M. (1948). From Max Weber: Essays in Sociology (translated by H. Gerth and C. W. Mills).
London: Routledge and Kegan Paul.
13
Table 1: English wordings of the survey items for three concepts that are used for illustration in Section
4. The short labels listed here are used to refer to the items in Tables 3 and 4, and elsewhere in the
Section.
Items on trust in police effectiveness :
EFF1: “How successful do you think the police are at preventing crimes in [country] where
violence is used or threatened?” Responses on an 11-point scale with the end points labelled as
“Extremely unsuccessful” for 0 and “Extremely successful” for 10.
EFF2: “How successful do you think the police are at catching people who commit burglaries
in [country]?” 11-point scale with the end points labelled as “Extremely unsuccessful” for 0 and
“Extremely successful” for 10.
EFF3: “If an emergency were to occur near to where you live and the police were called, how
quickly do you think they would arrive at the scene?” 11-point scale with the end points labelled as
“Not at all quickly” for 0 and “Extremely quickly” for 10.
Items on perceptions of police corruption :
CORR1: “How often would you say that the police in [country] take bribes?”
CORR2: “How often would you say that the police in [country] deliberately provide false
evidence to the courts?”
CORR3: “How often would you say the police in [country] use more force than is legally
allowed when making arrests?”
For each of these three items, the responses are an 11-point scale with the end points labelled as
“Never” for 0 and “Always” for 10.
Items on perceived risk of sanctions :
Introduction: “Now some questions about how likely it is that you would be caught and punished if you
did certain things in [country]. How likely is it that you would be caught and punished if you…”
SANCT1: “…paid cash with no receipt to avoid paying VAT?”
SANCT2: “...bought something you thought might be stolen?”
SANCT3: “...over or falsely claimed government benefits?”
For each of these three items, the responses are on a 4-point scale with labels “Not at all likely”, “Not
very likely”, “Likely” and “Very likely” for responses numbered 1-4 respectively.
14
Table 2: Parameter estimates for a factor analysis model with one factor for the three items on Police
effectiveness, with country as an explanatory variable for the latent variable.
* Two parameters are fixed at given values, which is necessary to render the model identifiable. The constraints used here
equate the scale of the latent variable with that of one “anchor” item (here EFF1), and assign the average of the latent variable
in one country (here the UK) the value 0.
Measurement model:
Item Intercept Loading
Residual standard
deviation
EFF1 5.14 1* 0.90
EFF2 4.71 0.84 1.52
EFF3 5.81 0.59 2.28
Distribution of the latent variable:
Country Mean
(Standard error of
mean estimate)
Standard
deviation
UK 0* 1.79
Bulgaria -0.79 (0.15) 1.93
15
Table 3: Results of likelihood ratio (LR) tests and estimated parameters of one-factor models for the
survey items listed in Table 2, comparing models with different levels of measurement equivalence for
each item in turn. See the text for more details.
Items for trust in police effectiveness
P-value for LR comparison
with
Parameter estimates for item tested
Intercept Loading
Item tested Model Equivalence Direct effect UK
Bulgari
a UK
Bulgari
a
EFF1
Equivalence
5.1
4 5.14
1.2
0 1.20
Direct
effect 0.12
5.1
6 4.93
1.1
2 1.12
Interaction 0.25 0.53
5.1
6 4.92
1.1
6 1.10
EFF2
Equivalence
4.7
1 4.71
0.8
4 0.84
Direct
effect 0.09
4.6
3 4.87
0.8
9 0.89
Interaction 0.23 0.82
4.6
3 4.87
0.8
8 0.90
EFF3
Equivalence
5.8
1 5.81
0.5
9 0.59
Direct
effect 0.91
5.8
2 5.80
0.5
9 0.59
Interaction 0.67 0.37
5.8
2 5.83
0.5
4 0.63
Items for perceptions of police corruption
P-value for LR comparison
with
Parameter estimates for item tested
Intercept Loading
Item tested Model Equivalence Direct effect UK
Bulgari
a UK
Bulgari
a
CORR1
Equivalence
3.3
5 3.35
1.0
6 1.06
Direct
effect <0.01
2.9
1 4.85
0.9
1 0.91
Interaction <0.01 0.27
2.9
2 4.88
0.9
5 0.87
CORR2
Equivalence
2.7
7 2.77
0.8
3 0.83
Direct
effect <0.01
3.0
1 1.79
0.9
7 0.97
Interaction <0.01 0.18
3.0
1 1.67
0.9
2 1.02
CORR3
Equivalence
4.1
0 4.10
0.5
5 0.55
Direct
effect <0.01
4.6
3 2.79
0.7
5 0.75
Interaction <0.01 0.25
4.6
3 2.67
0.7
1 0.80
Items for perceived risk of sanctions
P-value for LR comparison
with
Parameter estimates for item tested
Intercept Loading
Item tested Model Equivalence Direct effect UK
Bulgari
a UK
Bulgari
a
SANCT1
Equivalence
2.1
4 2.14
0.8
3 0.83
Direct
effect 0.11
2.1
2 2.24
0.9
4 0.94
Interaction 0.07 0.09
2.1
2 2.17
0.9
7 0.80
SANCT2
Equivalence
2.4
4 2.44
1.2
1 1.21
Direct 0.19 2.4 2.55 1.3 1.36
16
effect 3 6
Interaction <0.01 <0.01
2.4
3 2.79
1.3
3 1.93
SANCT3
Equivalence
2.8
8 2.88
0.6
8 0.68
Direct
effect <0.01
2.9
8 2.72
0.5
9 0.59
Interaction <0.01 0.01
2.9
8 2.64
0.7
4 0.38
17
Figure 1: ESS questionnaire development process (Round 5)
4. Revised
proposals
from question
designers
Stages 1, 2 and 4
repeated
2. Expert
review of
questions
1. Proposals
from question
designers
13. Mainstage
Fieldwork
3. Use of Survey
Quality Predictor
Program (SQP)
5. Consultation
with ESS
National
Coordinators
6. Split ballot
MTMM
experiments
developed
7. Large-scale two-
nation quantitative
pilot
10. Consultation
with ESS
National
Coordinators
8. Pilot
analysis
12. Translation
of questions
9. Expert review 11. Final source
Questionnaire
produced
18
Figure 2: Graphical representations of the types of models considered in the analysis. Diagram (1)
represents a model of complete measurement equivalence, where the distribution of the latent
variable may vary between countries, but the measurement model for survey items as measures of the
latent variable is the same in all countries. Diagram (2) represents partial non-equivalence where the
measurement model of Item 3 varies between countries. The arrow labelled (a) represents a direct
effect of country on the average level of the Item, i.e. an effect on the intercept term of the
measurement model. The line marked (b) represents a situation where, furthermore, the loading of
the latent variable in the measurement model depends on the country.
19
Appendix: Trust in justice module survey measures
Personal morality
Please tell me how wrong it is to…
(1) make an exaggerated or false insurance claim
(2) buy something you thought might be stolen
(3) commit a traffic offence like speeding or crossing a red light.
Response alternatives: not wrong at all, a bit wrong, wrong, and seriously wrong.
Perceived risk of sanction
Now just suppose you were to do any of these things in [country]. Please tell me how likely it is that
you would be caught and punished if you…
(1) make an exaggerated or false insurance claim
(2) buy something you thought might be stolen
(3) commit a traffic offence like speeding or crossing a red light.
Response alternatives: not at all likely, not very likely, likely, and very likely.
Confidence in the police
Now some questions about the police in [country]. Taking into account all the things the police are
expected to do, would you say they are doing a good job or a bad job?
Response alternatives: very good job, good job, neither good nor bad job, bad job, and very bad job.
Contact with the police
(1) In the past two years, did the police in [country] approach you, stop you or make contact with you
for any reason?
Response alternatives: yes, no
(2) ASK IF HAS BEEN APPROACHED / STOPPED / CONTACTED BY THE POLICE FOR ANY
REASON IN PAST 2 YEARS. How dissatisfied or satisfied were you with the way the police treated
you the last time this happened?
Response alternatives: Very dissatisfied, dissatisfied, neither dissatisfied nor satisfied, satisfied, and
very satisfied.
Trust in police distributive fairness
Now some questions about whether or not the police in [country] treat victims of crime equally.
Please answer based on what you have heard or your own experience.
(1) When victims report crimes, do you think the police treat rich people worse, poor people worse, or
are rich and poor treated equally?
Response alternatives: Rich people treated worse, poor people treated worse, and rich and poor
treated equally,
(2) And when victims report crimes, do you think the police treat some people worse because of their
race or ethnic group or is everyone treated equally?
Response alternatives: People from a different race or ethnic group than most [country] people
treated worse, people from the same race or ethnic group as most [country] people treated worse,
and everyone treated equally regardless of their race or ethnic group.
Trust in police effectiveness
(1) Based on what you have heard or your own experience how successful do you think the police are
at preventing crimes in [country] where violence is used or threatened?
Choose your answer from this card, where 0 is extremely unsuccessful and 10 is extremely successful.
(2) And how successful do you think the police are at catching people who commit house burglaries
in [country]? Use the same card.
(3) If a violent crime were to occur near to where you live and the police were called, how slowly or
quickly do you think they would arrive at the scene?
Choose your answer from this card, where 0 is extremely slowly and 10 is extremely quickly.
20
Trust in police procedural fairness
Now some questions about when the police deal with crimes like house burglary and physical assault.
(1) Based on what you have heard or your own experience how often would you say the police
generally treat people in [country] with respect.
Response alternatives: not at all often, not very often, often, and very often.
(2) About how often would you say that the police make fair, impartial decisions in the cases they
deal with?
Response alternatives: not at all often, not very often, often, and very often.
(3) And when dealing with people in [country], how often would you say the police generally explain
their decisions and actions when asked to do so?
Response alternatives: not at all often, not very often, often, very often, and (no one ever asks the
police to explain their decisions and actions).
Perceived legitimacy of the police: obligation to obey
Now some questions about your duty towards the police in [country]. To what extent is it your duty
to…
(1) back the decisions made by the police even when you disagree with them?
(2) do what the police tell you even if you don’t understand or agree with the reasons?
(3) do what the police tell you to do, even if you don’t like how they treat you?
Use this card where 0 is not at all your duty and 10 is completely your duty.
Perceived legitimacy of the police: moral alignment
Please say to what extent you agree or disagree with each of the following statements about the police
in [country]:
(1) The police generally have the same sense of right and wrong as I do.
(2) The police stand up for values that are important to people like me.
(3) I generally support how the police usually act.
Response alternatives: Agree strongly, agree, neither agree nor disagree, disagree, and disagree
strongly
Perceived legitimacy of the police: corruption and legality of action
(1) The decisions and actions of the police are unduly influenced by pressure from political parties
and politicians.
Response alternatives: Agree strongly, agree, neither agree nor disagree, disagree, and disagree
strongly
(2) How often would you say that the police in [country] take bribes?
Choose your answer from this card where 0 is never and 10 is always.
Confidence in the criminal courts
I am now going to ask you some questions about the courts in [country] that deal with crimes such as
house burglary and physical assault. Again please answer based on what you have heard or your own
experience. Taking into account all the things the courts are expected to do, would you say they are
doing a good job or a bad job?
Response alternatives: very good job, good job, neither good nor bad job, bad job, and very bad
job.
Trust in court effectiveness
Please tell me how often you think the courts make mistakes that let guilty people go free?
Use this card where 0 is never and 10 is always.
Trust in court procedural fairness
How often do you think the courts make fair, impartial decisions based on the evidence made
available to them?
Use this card where 0 is never and 10 is always.
21
Trust in court distributive fairness
Now some questions about the chances of different people in [country] being found guilty of crimes
they did not commit.
(1) Suppose two people - one rich, one poor - each appear in court, charged with an identical crime
they did not commit. Choose an answer from this card to show who you think would be more likely
to be found guilty.
Response alternatives: The rich person is more likely to be found guilty, the poor person is more
likely to be found guilty, and they both have the same chance of being found guilty.
(2) Now suppose two people from different race or ethnic groups each appear in court, charged with
an identical crime they did not commit. Choose an answer from this card to show who you think
would be more likely to be found guilty.
Response alternatives: The person from a different race or ethnic group than most [country] people
is more likely to be found guilty, the person from the same race or ethnic group as most [country]
people is more likely to be found guilty, and they both have the same chance of being found guilty
Perceived legitimacy of the courts: corruption and legality of action
(1) Please tell me how often you would say that judges in [country] take bribes? Use this card where
0 is never and 10 is always.
(2) Please say to what extent you agree or disagree with each of the following statements about
[country] nowadays: Courts generally protect the interests of the rich and powerful above those of
ordinary people.
Response alternatives: Agree strongly, agree, neither agree nor disagree, disagree, and disagree
strongly
(3) The decisions and actions of the courts are unduly influenced by pressure from political parties
and politicians.
Response alternatives: Agree strongly, agree, neither agree nor disagree, disagree, and disagree
strongly
Perceived legitimacy of the courts: obligation to obey
Everyone has a duty to back the final verdict of the courts.
Response alternatives: Agree strongly, agree, neither agree nor disagree, disagree, and disagree
strongly
Perceived legality of the law (legal cynicism)
(1) All laws should be strictly obeyed.
(2) Doing the right thing sometimes means breaking the law.
Response alternatives: Agree strongly, agree, neither agree nor disagree, disagree, and disagree
strongly
Punitive attitudes
(1) People who break the law should be given much harsher sentences than they are these days.
Response alternatives: Agree strongly, agree, neither agree nor disagree, disagree, and disagree
strongly
(2) People have different ideas about the sentences which should be given to offenders. Take for
instance the case of a 25 year old man who is found guilty of house burglary for the second time.
Which one of the following sentences do you think he should receive?
Response alternatives: Prison sentence, suspended prison sentence, fine, community service, and any
other sentence
(3) ASK IF PRISON SENTENCE GIVEN. And which of the answers on this card comes closest to
the length of time you think he should spend in prison?
Response alternatives: 1-3 months, 4-6 months, 7– 11 months, about 1 year, about 2 years, about 3
years, about 4 years, about 5 years, 6-10 years, and more than 10 years
Cooperation with the police and courts
22
Now some questions about what you would do if you were the only witness to a crime. Imagine that
you were out and saw someone push a man to the ground and steal his wallet
(1) How likely would you be to call the police?
(2) How willing would you be to identify the person who had done it?
(3) And how willing would you be to give evidence in court against the accused?
Response alternatives: not at all likely, not very likely, likely, and very likely.
Compliance with the law
How often you have done each of these things in the last five years?
(1) make an exaggerated or false insurance claim
(2) buy something you thought might be stolen
(3) commit a traffic offence like speeding or crossing a red light.
Response alternatives: never, once, twice, 3 or 4 times, and 5 times or more.
23
