Content uploaded by Giovanni Cassani
Author content
All content in this area was uploaded by Giovanni Cassani on Jan 30, 2023
Content may be subject to copyright.
Voting, fast and slow: Ballot order and likeability effects in
the Five Star Movement’s 2012 online primary election
Francesco Marolla∗1,2, Angelica Maineri†1,3, Jacopo Tagliabue‡4, and Giovanni
Cassani§5
11Department of Sociology, Tilburg School of Behavioral Sciences, Tilburg University
2Department of Sociology and Social Research, University of Trento
3Erasmus School of Social and Behavioral Sciences, Erasmus University Rotterdam
4New York University, Tandon School of Engineering
5Department of Cognitive Science and Artificial Intelligence, Tilburg School of Humanities and Digital
Sciences, Tilburg University
January 30, 2023
This paper has been accepted for publication in Contemporary Italian Politics - submitted on
June 25th 2022, revised on September 11th 2022, accepted for publication on December 3rd 2022.
DOI: https://doi.org/10.1080/23248823.2023.2175124
∗f.marolla@tilburguniversity.edu
†maineri@essb.eur.nl
‡jt1339@nyu.edu
§g.cassani@tilburguniversity.edu
1
Abstract
We document ballot order effects in the 2012 Parlamentarie, the online primary election
held by the Italian Five-star Movement to select the candidate Members of Parliament in the
2013 Italian general elections. We show that candidates appearing towards the top of the
screen systematically ranked higher in preferences. This effect holds controlling for candidates’
socio-demographic features. We also show that the number of competing candidates moderates
ballot order effects, with a stronger penalty for candidates appearing at the bottom of the
page in more crowded competitions. Finally, we show the influence of candidates’ likeability.
Our results confirm for the first time that ballot order effects and likeability effects, already
documented in traditional paper-based elections, are also found in online set-ups. We conclude
by highlighting how the online medium, if properly leveraged, has the potential to reduce the
influence of such biases.
Keywords: Five Star Movement; Digital Democracy; Online elections; Cognitive heuristics;
Ballot order effects; Satisficing
2
1 Introduction
In December 2012, the Italian Movimento Cinque Stelle (Five-star Movement, M5s) launched an
unprecedented large-scale experiment in direct online democracy, the Parlamentarie. For the first
time in Italy, a political formation allowed its party members to vote for the MP candidates running
for the subsequent general election, through an online decision-making platform. Members, organised
in the same electoral districts as those used for the subsequent parliamentary elections, had to
decide among more than 1,400 candidate MPs. Holding an online primary where party members
could directly choose the candidates for the general election was consistent with the Movement’s
commitment to enhancing citizens’ participation and avoiding candidates being chosen by party
leaders. In this study, we analyse the role that a set of decision heuristics played in the process:
the evidence shows that voters relied on cognitive shortcuts, undermining the promises that ‘direct
democracy’ would remove the traditional biases of party-based politics. However, we also suggest
that most biases could have been reduced had there been a proper decision-making setting.
The 2012 Parlamentarie adopted a rather straightforward procedure. Voters visited a web page
with candidates listed alphabetically by surname, with a self-uploaded picture, the name and sur-
name, a few demographic details, and a voting button. By clicking on the candidate’s name, voters
could – but, crucially, were not required to – visit a separate page, which provided the candidate’s
CV and a short video presentation. Voters could choose up to three candidates and had four days
to cast their votes.
Based on these implementation choices and the election context, one can formulate two hypothe-
ses about cognitive heuristics. On the one hand, voters participating in the primary can be assumed
to be highly motivated: as the literature suggests, satisficing behaviour (Simon, 1956) – in which
participants provide the first satisfactory answer instead of the optimal one (Krosnick, 1991; Roberts
et al., 2019) – is known to decrease with higher motivation (Krosnick, 1991; Roßmann et al., 2017).
On the other hand, the fact that political differences might have been scarce in a primary and that
voters could vote without checking all candidates’ political stances might lead to cognitive heuristics
playing a larger role in reaching a decision. Moreover, following previous studies (Meredith and
Salant, 2013; S¨oderlund et al., 2021), the ballot order effect is expected to be moderated by the
number of candidates in a district: whereas candidates appearing first are always advantaged, we
hypothesise that candidates appearing last are penalised in districts with more candidates (due to
satisficing) and advantaged in districts with fewer candidates due to memory effects (Nairne, 1988).
We investigate whether ballot order and likeability influenced the election outcome in the 2012
3
Parlamentarie, considering the candidate’s rank as our target variable1. We controlled for gender,
age, and certain features of the self-uploaded pictures (see Section Parlamentarie 2012 for further
details) to ensure that any effect of ballot order was not tainted by other potential voting-cues. Fur-
thermore, we examined the influence of candidates’ likeability, broadly construed as the impression
elicited by a candidate picture and operationalised by asking participants in an online survey how
likely they would be to vote for a candidate based only on their picture, to investigate whether more
likeable candidates had an advantage (Lau and Redlawsk, 2001; Ballew and Todorov, 2007).
In line with previous research on ballot elections (van Erkel and Thijssen, 2016; Marcinkiewicz,
2014; Miller and Krosnick, 1998), we provide evidence of ballot order affecting the outcome of the
Parlamentarie. We further show a robust effect of likeability that exists alongside the ballot order
effect. Therefore, candidates were more likely to attract votes if they appeared towards the top
of the screen and if they appeared more likeable from the self-uploaded picture. The number of
candidates in a district moderated the ballot order effect in line with our hypothesis: candidates
appearing at the bottom of the list were advantaged in districts with fewer candidates. These results
challenge the rhetoric of the M5s according to which the mere shift to an online setting improves
the quality of crucial democratic processes such as intra-party elite selection. However, the online
set-up allows for more effective countermeasures than traditional paper-based elections. We return
to these points in the discussion after introducing the theoretical framework, describing the data
and statistical methods, and presenting the empirical results.
2 Theoretical framework
2.1 Decision-making under online settings: what can go wrong?
Elections represent a crucial opportunity for citizens to affect democracies through their decision-
making. Citizens’ rationality plays an important role in classic democratic theory, which posits
that an informed and attentive citizenry is required for democracy to work properly (Estlund and
Landemore, 2018). However, more recent evidence from cognitive approaches to voting has raised
several doubts about the optimism of such assumptions (Achen et al., 2017). Indeed, most citizens
know or care little about politics, making assumptions of rationality in political decision-making
unrealistic. In the light of evidence from cognitive theories of human decision-making such as the
dual-process theory (Kahneman, 2011), we know that individuals tend to adopt cognitive shortcuts to
1The share of votes was unfortunately not available for all districts.
4
deal with the cost that such tasks impose. For instance, according to Krosnick (1991), satisficing is a
function of task difficulty, respondent’s ability, and motivation, so that the reliance on such shortcuts
increases when complex tasks are perceived as low stakes (for applications in survey research, see,
e.g., Roberts et al., 2019; Roßmann et al., 2017). That is, people act mainly as ‘cognitive misers’,
given their tendency to adopt the easiest solutions to deal with problems (Fiske and Taylor, 1991),
including political decision-making.
In view of these arguments, questions on the reliability of elections as important democratic
processes have been raised. In seminal studies conducted by Lau and Redlawsk (2006, 2001), it
has been argued that, despite the recourse to cognitive shortcuts, voters can vote ‘correctly’, where
correct voting is defined as ‘one that is the same as the choice that would have been made under
conditions of full information’ (Lau and Redlawsk, 2006, pp. 75). In complex contexts such as
electoral campaigns, voters can make sense of politics and decide how to vote by relying on heuristics
such as party affiliation, ideology, group endorsement or viability. In addition, it has been argued
that while single individuals are more likely to produce biased decisions, when taken in the aggregate,
individuals can make rational decisions if the proper conditions are met (Surowiecki, 2005). However,
given that some biases are systematic and therefore errors are never truly random (Bartels, 1996),
errors are unlikely to cancel out in the aggregate (Lau and Redlawsk, 2001).
In the realm of ballot elections, ballot order effects are amongst the most documented heuristics
in the literature, as several studies have shown how candidates appearing first on the ballot were
systematically advantaged compared to those appearing in the middle or last (Marcinkiewicz, 2014;
S¨oderlund et al., 2021; Lutz, 2010; Miller and Krosnick, 1998; D¨aubler and Rudolph, 2020). Such
evidence indicates that response order effects can impact electoral outcomes. Moreover, in the
absence of the political cues typical of party-based elections (e.g., party affiliation and ideology,
Marcinkiewicz, 2014; Lutz, 2010), ballot order can play an even more relevant role in candidate-
based primary elections like the Parlamentarie. While these elections aim to reduce the influence of
parties on decisions and force citizens to make more informed choices based on candidate competence,
they also increase the complexity of voters’ decisions: as it is unlikely that voters will collect sufficient
information on all candidates, such situations increase the cognitive load and the consequent use
of cognitive shortcuts (Meredith and Salant, 2013; S¨oderlund et al., 2021). Moreover, voters might
be more inclined toward cognitive shortcuts in primary elections, where the lack of ideological cues
deprives them of a critical discriminative cue (Marcinkiewicz, 2014).
While studies on human-computer interactions provide additional evidence of the pervasiveness
of response order effects (Burghardt et al., 2017, 2018; Lerman and Hogg, 2014; Dev et al., 2019;
5
Burghardt et al., 2017, 2018), we argue that any electoral setting should ensure that candidates
have fair chances of being elected due to their competence and stances rather than because of their
positions on the ballot. To the best of our knowledge, no study to date has replicated this evidence
in online elections. In particular, the set-up of the Parlamentarie is unique, as it made it possible for
voters to get further information about candidates within the platform itself, which may facilitate
access to information about candidates when voters make their choices.
2.2 The M5s 2012 online primaries
Since its foundation in 2007, the M5s has strategically used the web to organise its grassroots ac-
tivism and spread its anti-establishment messages (Bordignon and Ceccarini, 2015). While Beppe
Grillo’s blog was articulating the movement’s ideological messages in a ‘top-down’ fashion, grass-root
activists were aggregating and organising using the digital platform Meetup. These two organisa-
tional features correspond to what Mair and Katz (2002) called the ‘party in central office’ and the
‘party on the ground’, whereas the ‘party in public office’ was as yet virtually non-existent, aside
from a few council members elected in a handful of towns.
However, the fundamental element of innovation the M5s brought into Italian politics was the use
of digital media to actualise their narrative of citizens’ empowerment. Before the 2013 Italian general
elections, the M5s launched the Parlamentarie, the primary election to recruit the candidates that
would compete in the general elections to become MPs. Unlike any other primary election previously
held in Italy, the Parlamentarie were organised entirely online and aimed not simply to select the
party leader but the entire pool of candidates.
The M5s’ example falls within a framework of digital direct democracy experiments that Eu-
ropean democracies have witnessed over the last decade (other examples being the Spanish party
Podemos (Vittori, 2017), the France Insoumise party (Guglielmo, 2021), and the pirate parties (Ger-
baudo, 2018)). Despite some ideological differences, these parties share the use of ICTs to bring
citizens closer to institutions in an optic of power decentralisation. In so doing, they aim to improve
the quality of political participation to actualise their project of digital direct democracy. To this
end, internet voting platforms have been used as a tool for decision-making to afford different pref-
erence aggregation models, such as intra-party consultations and agenda-setting. By hosting these
deliberative processes on online platforms, parties reduce the costs of political participation for both
citizens and institutions, facilitating monitoring of constituencies (Deseriis, 2021).
The Parlamentarie represent an interesting case-study for at least two reasons. First, they
6
involved more than 20, 000 voters who chose amongst more than 1, 400 activists as candidate MPs
(Biorcio and Sampugnaro, 2019) in an ecologically valid context, i.e., one in which every actor
involved (party, candidates, voters) has genuine stakes. Second, their outcome had a far-reaching
impact on Italian politics, as the M5s went on to win about 25% of the popular vote at the ensuing
general election, with 163 candidates selected via the Parlamentarie being elected to the Chamber
of Deputies (109) and the Senate (54).
On top of being regular registered members of the M5s by September 2012, candidates needed
to meet several requirements to compete in the Parlamentarie2, meant to shield the movement from
infiltration and last-minute opportunists (Tronconi, 2018). Most notably, candidates were ineligible
if they had had previous experience as an MP, but had to have run in a local election, either under
the M5s banner, or as part of a local list affiliated with the M5s.
These requirements reflect characteristics typical of populist discourses (Mudde, 2004), being a
product of the anti-establishment narrative of the M5s, where the wisdom of the common citizen is
opposed to the corruption of the ruling political elite. Furthermore, political inexperience is seen as a
positive feature, political expertise being associated with the moral corruption of the Italian political
establishment. In addition, the M5s’ ideology has often been linked with technopopulism (Bickerton
and Accetti, 2018), that is, an ideology blending elements of populism (power to the people) with
elements of technocracy (power to the experts). Whereas such a mix might look contradictory at
first, the two arguments are jointly used to legitimise the Movement’s populistic claims and eschew
ideological confrontation. Thus, citizens are idealised as the real protagonists of political life and as
having greater expertise than professional politicians.
The purpose of the primary was also to let voters decide, albeit indirectly, on the order in
which candidates would appear on the ballot in the general election, which would in principle be
determined by the number of votes obtained in the Parlamentarie3. Having voters decide on the
order of candidates was particularly important in the M5s’ rhetoric since the electoral law in force
at the time made no provision for preference voting. Candidates were elected to Parliament based
on the share of votes obtained by the party in each district: the more votes, the more candidates
were elected, with candidates obtaining seats (or not) according to the order in which they featured
on the ballot. Therefore, even though some adjustments to the electoral lists for the general election
were made post-hoc (new names were added to the ballots and some candidates withdrew from the
competition before the election), the outcome of the primary may have played a role in determining
2https://www.ilpost.it/2012/12/02/le-primarie- del-movimento- 5-stelle/
3The procedure by which lists for the general election were put together was however not openly communicated
by the FSM.
7
who entered Parliament in the end.
Voters had to be at least 18-years old at the time of the election, be enrolled in the M5s as of 30
September 2012 and certify their identity digitally by uploading their ID. The voting interface was
rather simple, as shown in Figure 1. Voters landed on a page listing all candidates in the district
in alphabetical order based on the initial letters of their surnames. The landing page provided
information about each candidate’s appearance through a self-uploaded picture, name, gender, age,
and profession. Further details about the candidates and their political priorities could be accessed
by clicking on their names and being redirected to a separate page. Here, voters could find the
candidate’s CV, a short introductory video and a statement about the political projects the candidate
would pursue in the event that s/he was elected. It was not however possible to cast a vote on a
candidate’s personal page: for this, voters had to go back to the landing page. The voting procedures
lasted from 3 December to 6 December 2012 and involved 20,252 people (Biorcio and Sampugnaro,
2019).
In light of previous studies on ballot order effects, it is immediately clear that this interface created
considerable risks that voting heuristics helped to determine the outcome. First, some names were
immediately visible while others could only be seen after scrolling, and more so in districts with
more candidates competing. Second, voters could cast a vote directly on the landing page without
accessing candidates’ personal information and political statements. Finally, the availability of the
candidate picture on the landing page could have given more likeable candidates an advantage over
others.
3 Materials and Methods
3.1 Data
3.1.1 Parlamentarie 2012
The main dataset for this study was scraped in 2013 from the website, then accessible under a CC-
BY-NC-ND license, which hosted the results of the Parlamentarie4. Candidates were grouped into
31 districts, corresponding to the 27 districts in which the Italian territory was divided according to
the electoral system in 2013 and the 4 districts covering the rest of the world for Italians resident
abroad. For each candidate, we scraped their name, surname, age, profession, and picture (wherever
available) and annotated gender based on other demographics. We also derived their position on the
4The page can be now accessed through a way-back machine: https://web.archive.org/web/20121217093818/
https://www.beppegrillo.it/votazioni/candidati/elenco_circoscrizioni.php.
8
Figure 1: Example of candidate selection page on the Rousseau platform where the Parlamentarie
took place. The first column shows the self-uploaded picture, followed by the surname and name.
Other columns show age, place of birth, occupation, with the last column devoted to the voting
button.
screen, based on the alphabetical order of the candidates’ surnames, and scraped their final rank
(separately for each district) in the election.
We then manually tagged each available picture according to whether it contained a party logo.
To this end, we also considered the founder of the party as a logo to account for a possible party-
endorsement voting-cue, under the hypothesis that candidates featuring a party logo or the party
founder in their picture would boost their credibility in the eyes of voters and so gain an advantage.
We flagged pictures containing the scan of an ID document or drawings/comics/writings as not
containing a picture.
9
3.1.2 Likeability ratings
To account for the role of likeability, we collected judgments through an online survey distributed on
Prolific and hosted on Qualtrics. To collect likeability ratings, we selected three target districts that
i) varied in terms of the number of candidates competing, thereby improving the generalisability of
our results, and (ii) maximised the number of usable pictures5. We converged on districts number 5
(Lombardia-3; 18 candidates; 18 pictures available), 11 (Emilia-Romagna; 99 candidates; 80 pictures
available) and 21 (Puglia; 61 candidates; 49 pictures available).
Since some of the candidates have since 2012 acquired national significance and would be easily
recognised by Italian citizens, we recruited participants from France, Spain, Portugal, and Greece, to
ensure comparable aesthetic standards to those Italian voters are most likely to have, while limiting
the possibility of participants answering on the basis of what they knew of the candidate. We
recruited 176 participants, asking each of them to rate 25 pictures, to obtain 30 ratings per picture
and so improve the reliability of ratings per picture. Participants were paid £7.20/h for their
participation. The experimental design and data management plan were approved by the Research
Ethics and Data Management Committee (REDC) of the Tilburg School of Humanities and Digital
Sciences (TSHD) of Tilburg University, code REDC2020.201. The Online Supplementary Materials,
datasets, figures generated during the analyses, and R scripts are available for replication purposes
on Dataverse (https://doi.org/10.34894/KE8VVY).
After providing their informed consent, participants were asked to provide details of their gen-
der, age, education, and country, these being collected in order to assess whether there were any
systematic differences in likeability ratings. Participants who did not provide their consent were
redirected back to Prolific and did not receive payment.
Likeability ratings were obtained by presenting subjects with a candidate’s picture (of the same
size and resolution as the one uploaded to the website, in order to preserve the conditions of the
Parlamentarie as much as possible) and were asked to drag a slider to indicate how likely they would
be to vote for the candidate basing their decision simply on the picture. They were instructed to
move the slider more towards a pole the stronger their intuition about the candidate. The slider
was initially presented in the middle and was anchored between -50 (not at all likely ) and +50
(extremely likely). Candidates were presented randomly to each participant. Participants were
scanned for uncooperative behaviour, considering whether they always left the slider at the initial
position or whether they always dragged the slider to the same extreme of the bar. No participant
5We provide further details about the exclusion criteria for candidate pictures as well as about the experimental
interface and procedure, inclusion criteria for participants, and participants demographics in Appendix A.
10
showed uncooperative behaviour as defined in this way.
3.2 Statistical approach
We adopted a step-wise regression approach: we started by including a set of control variables and
added the variable of interest (with possible interactions) to assess whether it improved the model’s
fit. The control variables included the candidate’s age, their gender (binary), and the presence
of a party logo in the candidate’s picture (to create a categorical variable with three levels: no
picture, picture without a logo, and picture with a logo). We further considered possible composition
effects to test the possibility that, for example, a 50-year-old candidate would appear young if the
median age in the district was 65 but old if the median age in the district was 35, or that a logo
would draw more attention if only 5% of the pictures in a district showed one as opposed to 20%
of the pictures showing one. Therefore, we derived two measures for age: one applying median
centring considering the global median and one applying median centring separately per district,
i.e. subtracting the district median age from the age of each candidate in the district. We further
computed the proportion of pictures with a logo in each district and the proportion of women in each
district). Finally, we quantified the model fit using the Akaike Information Criterion (AIC), which
penalises models for the number of parameters they estimate. We thus checked whether adding the
predictor of interest resulted in a reduction of the AIC.
The dependent variable in the main analysis was the candidate’s rank per district. However,
screen position and rank are inevitably related and co-vary, and while every district features a
candidate at screen position 1 and a candidate ranking first, not all districts feature a candidate in
screen position 57 and a candidate ranking 57th. It is very different for a candidate to be in position
18 in a district with 18 rather than 80 candidates: considering screen position as the absolute distance
from the top of the page does not allow us to take account of the fact that the last position in the list
can be very salient as well. Therefore, we transformed both rank and screen position by normalising
the values to the unit range, using the formula unit(xi) = xi−min(X)/max(X)−min(X)∀xi∈X
where X is the vector of screen position values or ranks: candidates appearing at the top of a page
or ranking first would get a score of 0, and candidates appearing last on the page or ranking last
would get a score of 1, with intermediate values depending on the number of candidates in a district.
We used Generalised Additive Mixed Models6(GAMMs, Baayen et al., 2017) to model rank
(unit-normalised) as a function of the independent variables of interest. Both GAMMs and CLMMs
6Appendix C presents a replication of the analysis of rank using Cumulative Link Mixed Models (CLMM), treating
rank (not unit normalised) as an ordinal variable.
11
implement a multilevel approach to account for district-specific variance by including random inter-
cepts and (non-linear) slopes for the relevant independent variables. Moreover, GAMMs allowed us
to model possible non-linear effects that continuous predictors may have on rank: this is particularly
important considering that we have hypothesised that the ballot order effect may be quadratic in
smaller districts. Unless otherwise specified when describing results, we included continuous pre-
dictors as simple smooths; used splines to model non-linearities, and did not limit the number of
inflection points an estimated effect was allowed to have.
To check whether the number of candidates moderates the effect of screen position, we took the
log of the number of candidates (base 2), to reduce the long right tail that would make estimates
brittle for districts with several competing candidates, and included a partial tensor product between
number of candidates and screen position to fit an interaction between the two.
Finally, we analysed the possible effect of likeability on rank. We again used GAMMs predicting
rank as a function of age, gender, screen position and participants’ ratings in the online survey,
including random intercepts for rater ID to account for the fact that ratings provided by the same
participant are likely to have a higher correlation than ratings provided by different raters. We did
not consider the presence of party logos in the pictures because we selected pictures which did not
have a party logo in the first place to avoid biases due to subjects’ recognition of the logo.
4 Results
4.1 Ballot order effects on rank
Our first analysis focused on possible ballot order effects on the ranking of candidates in each district.
We fitted a baseline GAMM predicting unit-normalised rank as a linear combination of age, gender
and presence of a logo in the picture7. We first compared global median centring and district median
centring, and observed that the latter provided a better fit. We then tested whether the effect of
gender was moderated by the proportion of women in each district, testing whether an interaction
between gender and proportion of women per district improved model fit over the simple effect of
gender. This interaction resulted in a higher AIC and was thus discarded. The same happened
with the interaction between presence of a logo and share of pictures with a logo per district, which
was also discarded from further analyses. The baseline model thus included a smooth term for
age (median centred by district), parametric terms for gender and presence of logo, and a random
7We used the mgcv package (Wood, 2017) in R to fit GAMMs and the itsadug package (van Rij et al., 2020) to
plot effects.
12
intercept for district identifiers (AIC = 222.718). No random slope improved model fit.
We then added screen position (unit-normalised)8to this model (random slopes for screen po-
sition did not improve model fit). This model (AI C = 171.795, adj.r2= 0.243) improved over
the baseline (∆AIC = 50.923), showing that screen position is a significant predictor of rank
(edf = 5.144, Ref.df = 6.253, F= 9.767, p < 0.001). Figure 2 displays all the effects visually
(all effects were statistically significant, exact coefficients are provided in Appendix B).
0.0 0.2 0.4 0.6 0.8 1.0
0.3 0.4 0.5 0.6 0.7
Screen position
screen position (normalized)
rank (normalized)
fitted values, excl. random
−20 −10 0 10 20 30
0.3 0.4 0.5 0.6 0.7 0.8
Age (median centered by district)
age (centered)
rank (normalized)
fitted values, excl. random
m
w
0.2 0.3 0.4 0.5 0.6
Gender
rank (normalized)
no_pic
no_logo
logo
0.4 0.5 0.6 0.7 0.8 0.9 1.0
Party logo in picture
rank (normalized)
Figure 2: Effects of screen position (unit-normalized; top-left), age (median centred by district, top-
right), gender (bottom-left), presence of a logo in the candidate picture (’logo’ indicates pictures with
a party logo; ’no logo’ indicates pictures without a party logo; ’no pic’ indicates candidates who did
not upload a picture at all; bottom-right). Low values on the y axis indicate better rankings. The
underlying statistical model is a GAMM with ranking (uni-normalized as the dependent variable),
continuous predictors included as simple smooths, and a random intercept for district.
8To ensure that the normalisation did not introduce any artefact, we also fitted a GAMM with the same DV and
IVs but replacing normalised screen position with the original variable. The predictor was significant (edf = 3.258,
Ref.df = 4.070, F= 11.24, p < 0.001) but the model had a worse fit (AI C = 183.5, adj.r2= 0.235), showing that
normalised screen position best captures the relation between screen position and rank in each district.
13
The effect of screen position appears roughly quadratic: candidates appearing towards the top
of the screen (left side of the x-axis) had an advantage over other candidates. However, candidates
appearing towards the bottom of the screen (right end of the x-axis) tended to rank higher (bet-
ter) than candidates appearing in the middle of the list. We also observe a linear effect of age,
with younger candidates relative to their district ranking first, as well as an advantage for women.
Candidates showcasing a party logo in their picture had a slight yet significant advantage over can-
didates who did not, but the dominant effect is the stark penalty for candidates who did not upload
a picture.
We then included an interaction between screen position (unit-normalised) and number of can-
didates per district (logged)9to check whether the quadratic effect was primarily driven by smaller
districts, in line with our predictions. This interaction was statistically significant (edf = 3.129,
Ref.df = 3.701, F= 3.869, p < 0.01) and improved the model fit over the model which included
screen position alone (AIC = 163.871, ∆AIC = 7.924). Visual inspection of the tensor product
revealed the expected pattern, visualised in the centre panel of Figure 3 (coefficients are provided
in Appendix B). Darker shades of blue indicate lower predicted ranks, while orange shades indicate
higher predicted ranks. In the right-hand panel we display the effect of screen position at different
numbers of candidates, showing the composite effect of the main effect and the partial tensor prod-
uct: the advantage of candidates displayed at the top exists regardless of number of candidates, but
is slightly weaker when fewer candidates compete. In contrast, we see that the advantage for candi-
dates shown at the bottom of the list is clear when fewer candidates compete but almost vanishes
in the most crowded districts.
4.2 Likeability effect on rank
We fitted a GAMM model predicting unit normalised rank (to address the differences in number of
candidates in the three target districts). The independent variables included age (median centred
by district), screen position (unit normalised), gender, and likeability ratings, all in interaction with
district ID. We first tested whether all interactions were needed by comparing AIC scores and found
that they all improved model fit, showing that the target effects are different in the three target
districts. The model also included a random intercept for rater ID. Figure 4 shows the effects of
interest, allowing for direct comparisons between districts (see Appendix B for the exact coefficients
of screen position and likeability in each district).
9We included screen position as a smooth term and the interaction between screen position and number of candi-
dates as a partial tensor product, limiting the number of allowed inflection points to 4.
14
0.0 0.2 0.4 0.6 0.8 1.0
0.3 0.4 0.5 0.6 0.7
s(Screen Position)
screen position (normalized)
rank (normalized)
fitted values, excl. random
0.0 0.2 0.4 0.6 0.8 1.0
4.0 4.5 5.0 5.5 6.0 6.5
ti(screen position, n candidates)
screen position (normalised)
n candidates
0.35
0.4
0.4
0.45
0.45
0.5
0.5
0.55
0.55
0.6
0.6
0.0 0.2 0.4 0.6 0.8 1.0
0.3 0.4 0.5 0.6 0.7
Screen Position by n candidates
screen position (normalized)
rank (normalized)
fitted values, excl. random
n candidates
17
41
81
Figure 3: Non-linear interaction between screen position and number of candidates on rank (unit-
normalised) estimated using a GAMM while controlling for gender, age, and presence of party logos
in the candidate picture. Left: main effect of screen position (x axis, unit normalised) on rank (y axis,
unit normalised). Center: partial tensor product between screen position and number of candidates.
Darker shades of blue indicate lower (better) predicted ranks while orange shades indicate higher
(worse) predicted ranks. Red lines connect points with the same predicted rank. Right: effect of
screen position (x axis, unit normalised) on rank (y axis, unit normalised) for districts with different
number of candidates (colour and line type legend), showing how the partial tensor product and the
main effect of screen position combine.
Screen position had quite different effects, consistent with our previous analysis: candidates
appearing at the bottom were favoured in district 5 (the smallest). In contrast, in the mid- size and
large districts, candidates appearing at the top were favoured. Likeability had a largely linear effect,
with a slight yet robust non-linearity in the largest district:10 candidates rated as more likeable from
the picture ranked better in all districts, with a stronger effect in district10 (the smallest). Finally,
we see again that women were favoured, and more so in the largest district (district 5 is not shown as
only men competed there). Our results thus suggest that more likeable candidates were advantaged
regardless of how many candidates were competing.
10We probed this by fitting a model only on data points from district 21. First, we included the effect of likeability
as a simple smooth, and then we included it as a parametric term, forcing the model to estimate a linear effect. The
AIC of the first model was 17 points lower, confirming that allowing the model to estimate a non-linear effect for
likeability on rank improves model fit.
15
0.0 0.2 0.4 0.6 0.8 1.0
0.3 0.4 0.5 0.6 0.7
Screen position
screen position
rank
fitted values, excl. random
5
11
21
−40 −20 0 20 40
0.2 0.3 0.4 0.5 0.6
Likability
would vote from pic
rank
fitted values, excl. random
5
11
21
Figure 4: Effect of screen position on rank (left) and likeability on rank (right) moderated by district
id (blue, solid line: district 5 (Lombardia 3, 18 candidates); red, dotted line: district 11 (Emilia
Romagna, 100 candidates); yellow, dashed line: district 21 (Puglia, 61 candidates)), estimated using
GAMMs controlling for sex and age (median centred per district). The non-linearity was limited to
a third order polynomial to prevent overfitting.
5 Discussion
The M5s’ 2012 Parlamentarie provided a unique experiment in online democracy and nearly ideal
conditions for isolating cognitive biases’ effects on elections, which are typically difficult to achieve in
conventional primaries. In this election, voters’ decisions were not influenced by well-known biases
such as candidates’ affiliations to a specific party faction, or viability (Lau and Redlawsk, 2006,
2001). At the same time, the Parlamentarie offer a real-life case-study of election biases, allowing a
thorough test of our hypotheses outside the lab.
Our results show that the choices of M5s party members are likely to have been affected by
ballot order effects — reported in conventional paper-based elections (van Erkel and Thijssen, 2016;
Lutz, 2010; Marcinkiewicz, 2014) — and candidate likeability (Ballew and Todorov, 2007; Lau and
Redlawsk, 2001). We document how, in general, candidates appearing at the top of the screen were
advantaged as compared to candidates appearing further down the list. We further qualified this
effect by showing that the number of candidates competing in a district moderates order effects, with
16
stronger penalties in districts with more candidates competing (S¨oderlund et al., 2021; Meredith and
Salant, 2013). This seems to confirm that the higher the number of candidates, the more voters
resort to satisficing behaviour (Krosnick, 1991). In the context of the Parlamentarie, this effect can
be explained by considering that the larger the number of candidates competing, the more voters
had to scroll and the more time they would have had to spend if they had sought to survey all of
them. In small districts, however, we found evidence of a recency effect, suggesting that candidates
who appeared at the bottom of the page were advantaged, likely due to easier recall and higher
salience after exhausting the candidate list (Nairne, 1988). These findings are robust net of control
variables related to candidates’ sociodemographic characteristics and composition effects.
We further found a robust effect of likeability in a sample of districts, with more likeable candi-
dates ranking higher. The effect of screen position held even after controlling for likeability. However,
in the smaller district, the effect of screen position was reversed, with candidates appearing at the
bottom of the page ranking higher in the election. This suggests that different cognitive biases may
interact in non-trivial ways and paves the way for future studies, which should also consider halo
effects, to assess whether appearing closer to popular candidates may provide a spillover advantage
purely because voters will be more likely to consider a candidate whose name appears close to
another candidate which, for any reason, draws more attention. In addition, future research may
look into which features respondents evaluate when rating candidates’ likeability, since our set-up
did not disentangle which specific criteria respondents adopted.
The implications of these findings are amplified when considering (i) the electoral results achieved
by the M5s (25.55% for the Chamber of Deputies in 2013) and (ii) the electoral rules in place during
the 2013 general election, which ‘projected onto’ the general election the biases of the primary election
(see Appendix D for an analysis showing that candidates’ screen positions in the Parlamentarie
indeed had an effect on their probability of entering parliament in the 2013 general election).
Finally, the covariates we included in the statistical model showed interesting effects on their
own. First, candidates who are younger than their competitors tended to rank higher. This finding
aligns with the Movement’s rhetoric and the requirement of candidates not to have had prior political
experience in a public office: younger candidates might appear less compromised with the political
establishment. Moreover, showing the party logo or party founder in the candidate’s picture gave
him/her an advantage, suggesting that in a field where differences among candidates are small, being
in a position to attest one’s history in the movement provided an advantage. Moreover, not uploading
a picture resulted in a strong penalty. The reason for this cannot be gleaned from our analysis, but
we hypothesise that, in a party with a strong emphasis on digital tools and transparency, candidates
17
not providing their pictures might appear less trustworthy, less committed or less technologically
capable, all liabilities on the M5s platform. Finally, we saw that women had an advantage. However,
the number of women competing in the election was very small (196 out of 1,486 candidates, of which
154 appeared on the ballot in the 2013 general election), suggesting that self-selection might have
played a role, with women competing only if they felt sufficiently qualified.
Considering the hypotheses we started from, therefore, we should conclude that the allegedly
higher motivation of voters was not enough to counter the effect of satisficing, operationalised here
through order effects and likeability. It is possible that the design choices of the interface played
a part in determining these effects by creating more favourable conditions for voters to resort to
shortcuts and heuristics and countering the possible advantages offered by the online medium in
making information about candidates more readily available. In addition, such effects may also have
hampered the Movement’s objective of leveraging digital democracy tools to improve the process of
recruitment of new candidates by using open online primaries in place of the conventional approaches
adopted by other parties. The Parlamentarie were successful in removing several barriers to political
access, especially as compared to more structured primaries where parties exert tighter control of
the lists: unfortunately, our analyses show that the implementation hampered these possibilities for
several candidates, who may have been disadvantaged by platform design choices.
Even though we document a robust effect of ballot order and likeability on election ranks in
the Parlamentarie, our analyses remain correlational and cannot be taken to provide evidence that
screen position caused the election outcome. A controlled A/B test would be required to test a causal
relation, e.g., showing to a random subset of voters candidates in alphabetical order vs candidates
listed in a random yet fixed order vs candidates randomly shuffled at each access. This experiment
is however not feasible in a real election as it would systematically manipulate the electoral lists
for subsets of the electorate, likely biasing the decision-making process of some of them. For this
reason, we further argue that such an experimental manipulation would not be ethical in a real
election, particularly given the results we have illustrated – results that suggest a robust relation
between ballot order and election outcome, in line with previous studies (van Erkel and Thijssen,
2016; Lutz, 2010; Miller and Krosnick, 1998). Considering the stakes in such processes, extant
knowledge should be leveraged to make a platform that minimises the influence of cognitive biases,
adopting the more pervasive randomisation allowed by the medium and the electoral system. Mock
elections are a viable option for implementing such a manipulation (Lau and Redlawsk, 2006), but
they lack ecological validity and would not allow us to draw firm conclusions given that participants
are unlikely to adopt the same decision-making processes in an election with no real stakes.
18
Although we were unable to analyse the Movement’s subsequent online primary elections due to
changes in the license under which the website was made public, the 2012 Parlamentarie provided an
environment that was less influenced by external dynamics. Candidates did not have any advantage
due to incumbent status, and media coverage was limited due to the lack of a party in public
office and to a ban on television appearances on the part of party members. Our findings further
highlight the wide effects of biases in the 2012 Parlamentarie: candidates receiving an advantage
due to likeability or screen position went on to gain an incumbent advantage and name recognition,
improving their chances in the following elections (Meredith and Salant, 2013). However, it could
be the case that some of the candidates enjoyed local popularity because they had run for previous
local elections and lost or received occasional media coverage in the local press. While our focus in
this work was on information that was immediately available or determined by the voting platform,
future work can look into ways of quantifying name recognition for candidates at the time, using
news or social media to track user mentions and to test whether name recognition predicts election
rank. Future work can also leverage the data that we release to test further predictors. Information
provided in videos and personal pages would have been particularly interesting in this respect but
could not be scraped as it was only accessible after log-in.
Despite showing that online elections are subject to the same biases found in paper-based elec-
tions, our results actually suggest that the online set-up offers several opportunities to limit the
influence of such biases. If anything, the Parlamentarie represent a wasted opportunity to counter
or limit the impact of cognitive heuristics, whose effects have been known for a long time in the
political science literature (van Erkel and Thijssen, 2016; Lutz, 2010; Miller and Krosnick, 1998).
Useful suggestions come from research on clicking behaviour (Dev et al., 2019). For instance, the
online set-up offers the chance to randomise the order of the candidates for each user’s access to
reduce ballot order effects. Such a solution has already been tested in conventional elections (Ho
and Imai, 2006; Darcy and Mackerras, 1993), but it is costly and challenging to implement. In
contrast, the online setting provides a cheap and feasible opportunity in this direction. Even though
the primacy bias is impossible to eliminate due to the way in which human attention works (Simon,
1956), randomising the order of the candidates for each voter’s access could at least reduce the effect
of this bias on the election outcome.
Moreover, connected to our results on gender and age effects, another suggestion would be to
conceal impression signals to avoid the influence on voting decisions of factors other than candidate
competence. Finally, drawing from the evidence that suggests that primacy effects are more likely
to be found in cases of quick completion response (Malhotra, 2008), another counter measure would
19
be delaying voting. Platforms could set a fixed time before allowing the user to vote to encourage
them to consult as many candidate profiles as possible and to counter cost-minimising behaviour.
Such advice can be especially crucial in primary elections, where ideological voting-cues are lacking
and the cognitive costs for voters tend to be higher.
In conclusion, our study has investigated the electoral outcomes of – what was at the time
– an unprecedented experiment in digital democracy – one that took place in Italy in 2012 and
had important implications in terms of institutional representation. We reported a robust ballot
order effect, moderated by the number of competing candidates, and a likeability effect in the 2012
Parlamentarie – the effects – we suggested - depending on satisficing (Krosnick, 1991; Roberts et al.,
2019). Our main wager is that digital platforms have the potential to address existing problems but,
left unchecked, do not necessarily lead to a better quality of decision-making. The good news is that
policymakers can profit from robust evidence and suggestions that the literature offers. Although
cognitive biases will continue to affect the decision- making of many voters, digital platforms provide
several possible solutions if properly leveraged.
6 Acknowledgments
This project started almost ten years ago, and followed Hofstadter’s Law11 as the original design
changed and life had its course. We wish to thank the people that at that time helped us in
understanding the FSM, collecting and labelling the data, and taking a first stab at all of this.
Francesco Oggiano, Ileana Bego, Camilla Tagliabue and Davide Rossi: your contribution has not
been forgotten.
We further thank Giuseppe Arena, Federico Bianchi, Gabriele Mari, and Matteo Colombo for
useful feedback about this work, and Andrea Polonioli, Bingqing Christine Yu and Ciro Greco for
their help in a previous iteration of this research project. We also thank the people that helped
us translate the survey in the target languages: Raquel Garrido Alhama, Kalliopi Dilaveraki, Paris
Mavromoustakos Blom, Nikos Aggelakis, Miguel Egler, and Audrey Patenaude.
7 Authors biographies
Francesco Marolla is a joint PhD candidate for the Sociology and Social research department of
University of Trento and the sociology department of Tilburg University. His work focuses on
11It always takes longer than you expect, even when you take into account Hofstadter’s Law.
20
populist voting behaviour in a comparative perspective.
Angelica M. Maineri is a PhD candidate in Sociology at Tilburg University and Data Manager at
ODISSEI. Her work focuses on privacy risks in the datafied society and (web) survey methodology.
Jacopo Tagliabue is the Director of AI at Coveo and Adj. Professor of MLSys at NYU. He divides
his time between industry and research, with a focus on natural language processing, information
retrieval and reasoning.
Giovanni Cassani is Assistant Professor at the Department of Cognitive Science and Artificial
Intelligence at the Faculty of Humanities and Digital Sciences of Tilburg University. His work focuses
on cognitive science, computational linguistics, and data science.
8 Disclosure statement
The authors declare no conflict of interest.
Appendices
A Online survey for likeability ratings
As mentioned in the main article, we recruited participants from other countries than Italy to avoid
that respondents judged pictures considering their political inclinations. We used Prolific’s screeners
to limit the survey availability to subjects who indicated their country of residence is one of the four
aforementioned ones and also indicated one of Spanish, French, Portuguese and Greek as their native
language. The survey was translated from English in each target language by native speakers who
are also fluent in English.
As indicated in the main text, after providing their informed consent, participants were asked
to provide a few demographic details. Gender was coded as a three-level factor (Male, Female,
Non-binary / Third gender); age was binned (18-29, 30-39, 40-49, 50-59, 60-69, 70+); education was
collected by asking whether participants held a college degree (Bachelor, Master’s, PhD) or not;
country was collected by presenting a text box and asking participants in which country they lived.
Subjects wishing not to provide this information could select the option Prefer not to say for the
first three questions and could type NA for the last. Demographics were collected to assess whether
systematic differences in likeability ratings were observed but will not be released to preserve raters’
full anonymity.
21
After completing the demographic questions, participants were presented with a practice trial in
which they were asked to drag a slider to signal whether a rock is animate or inanimate. Candidates
dragging a slider towards the animate pole were presented with a message explaining the intended
use of the slider and asked to drag it towards the inanimate pole. Once the slider was dragged
towards the correct pole, participants could proceed to rate candidates’ pictures. likeability ratings
were obtained using the interface in Figure 5.
Figure 5: Example trial from the online survey we used to collect likeability ratings.
In each trial, participants saw a picture of a candidate (at the same size and resolution as uploaded
on the website to preserve the actual conditions from the Parlamentarie as much as possible) and
were asked the following question: Simply basing your decision on the picture, how likely would you
be to the vote for this candidate?. They then had to drag a slider between the poles Not at all likely
displayed on the left and Extremely likely displayed on the right.
We did not collect likeability ratings for any picture featuring other people than the candidate
for both ethical and practical concerns. First, we could not be sure these other people provided
their consent for their picture to appear on the website. Second, specifically for minors, we preferred
to avoid uploading such pictures online. Third, it would be unclear for raters which person to rate
when more than one appears in the picture. We also excluded pictures featuring party logos, since
their presence could have a different impact on party members voting in a primary election than it
does on raters.
22
The survey was conducted in September 2021. First of all, there was a strong imbalance along
several demographic variables. As far as the raters’ country is concerned, 109 participants indicated
they came from Portugal, 26 from Spain, 24 from Greece, 12 from France, while 5 did not specify
their provenance. Moreover, 154 raters indicated they were between 18 and 29 years old, 19 between
30 and 39, 2 between 40 and 49, 1 between 50-59. 102 participants indicated having a college-
level degree, 73 indicated they do not, 1 preferred not to share this detail. Finally, 81 participants
identified as females, 93 as males, 2 as non-binary.
We plotted likeability ratings against rank in the election separately by district, overlaying a
LOESS fit separately by category level for each demographic attribute to inspect for possible sys-
tematic differences that would prevent us from conflating all judgments when predicting rank. We
show the outcome for the variable Country in Figure 6 to illustrate the approach. Similar figures
for the variables Gender, Age, and Education are available in the data package, as well as the script
we used to conduct the pre-processing of the likeability ratings as downloaded from Qualtrics. It
can be seen that, with some small differences notwithstanding, the LOESS fit between likeability
and rank is comparable across ratings provided by subjects from different countries. The same was
observed for all other demographic variables, indicating that no systematic biases attributable to
demographic characteristics affected likeability ratings.
Next to the aforementioned visual approach, we also fit a linear mixed effect regression (LMER)
model with the lme4 (Bates et al., 2015) and lmerTest (Kuznetsova et al., 2017) packages predicting
likeability ratings as a linear combination of raters’ gender, country, education, and age (binarised,
with the category 18-29 labeled as young and the other categories lumped together and labeled as
notYoung due to sparsity). We also included a random intercept for rater ID and nested random
intercepts for candidate ID within district ID to account for systematic variation attributable to
the specific rater and the specific candidate being rated. Finally, due to sparsity, we excluded
participants who self-identified as non-binary, and who did not disclose their education or country.
No predictor was significant, confirming that no systematic differences in ratings can be attributed
to demographic characteristics. We, therefore, included all likeability ratings in the analysis of rank
reported in the main analysis. This analysis is also available in the data package, with the output
included in the script.
23
0
5
10
15
−50 −25 0 25
rank
Country France Greece Portugal Spain
d5
Likability ~ rank (by Country)
0
25
50
75
100
−50 −25 0 25 50
rank
d11
0
20
40
60
−50 −25 0 25 50
likability
rank
d21
Figure 6: Relation between likeability and rank, plotted separately for each district, with superim-
posed LOESS fit. The black line shows the LOESS fit disregarding country information. The fit
over ratings provided by participants who did not disclose their country has been removed due to
the small sample size.
24
B GAMMs regression for rank prediction
We report here the full regression coefficients and corresponding statistics that resulted from the
model visualised in Figure 2 in the main article. These results are visualised and further described
in Section 4.1 of the main article.
Table 1: Estimates of the GAMM fitted to predict unit normalised rank as a linear combination of
gender, presence of a party logo in the picture, age (median centred per district), and screen position
(unit normalised). Parametric terms are presented first, with the estimated bcoefficient, corresponding
standard error (se), tstatistic and pvalue. For the non-linear smooths (age and screen position), the
table shows the estimated degrees of freedom (edf), the reference degrees of freedom (Ref.df ), F statistic,
and pvalue.
Estimate se t p
(Intercept) 0.5254 0.0082
gender:woman -0.2642 0.0202 -13.106 < .001
logo:yes -0.0624 0.0234 -2.665 < .01
logo:no picture 0.3316 0.0314 10.571 < .001
edf Ref.df F p
age 1.4097 1.721 33.033 < .001
screen position 5.1438 6.253 9.767 < .001
We further provide here the table showing the regression coefficients of the GAMM fitted to
predict unit-normalised rank considering an interaction between number of candidates and screen
position, visualized in Figure 3 of the main article. See Section 4.1 of the main paper for a detailed
presentation and a visualization of these results.
Table 2: Estimates of the GAMM fitted to predict unit normalised rank as a linear combination of gender,
presence of a party logo in the picture, and age (median centred per district, included a simple non-
linear smooth) and a non-linear interaction of unit normalised screen position and number of candidates
(n candidates, logged). The non-linear interaction was coded as a partial tensor product. The table
provides the estimated degrees of freedom (edf), the reference degrees of freedom (Ref.df), F statistic,
and pvalue.
Estimate se t p
(Intercept) 0.5257 0.0082
gender:woman -0.2638 0.0201 -13.122 < .001
logo:yes -0.0592 0.0234 -2.532 < .05
logo:no picture 0.3275 0.0313 10.476 < .001
edf Ref.df F p
age 1.3318 1.598 37.723 < .001
screen position 4.92 6.004 10.056 < .001
screen position*n candidates 3.129 3.701 3.869 < .01
Finally, we provide the coefficients and corresponding statistics for the analysis of the effect of
likeability on rankings in a sample of districts. See section 4.2 for an elaboration on these results.
25
Table 3: Effect of likeability on rank in the three target districts, extracted from district-specific models
also including gender (if applicable), age (centred), and screen position. The table provides the adjusted
r2of the full model, estimated degrees of freedom (edf ) and reference degrees of freedom (Ref.df ) of
the likeability effect, with the corresponding Fstatistic and pvalue.
Estimate se t p
(Intercept) 0.4802 0.0120 40.176 < .001
gender:woman -0.2589 0.0224 -11.536 < .001
d11 0.0440 0.0136 3.245 < .01
d21 0.0296 0.0142 2.081 < .05
gender:women*d11 0.0394 0.0258 1.526 0.127
edf Ref.df F p
age*d5 1.000 1.000 63.548 < .001
age*d11 1.986 2.000 180.218 < .001
age*d21 1.217 1.386 73.514 < .001
screen position*d5 1.852 1.978 20.677 < .001
screen position*d11 1.954 1.998 22.926 < .001
screen position*d21 1.947 1.997 122.240 < .001
likeability*d5 1.100 1.189 23.107 < .001
likeability*d11 1.000 1.000 26.843 < .001
likeability*d21 1.277 1.478 8.752 < .001
C Ordinal regression models for rank prediction
Before carrying out any statistical analysis on rank, we ensured that no relation exists between our
control variables, gender and age, on the one side, and screen position on the other, which would
confound the interpretation of the target effect of screen position on rank. A GAMM predicting
gender as a linear combination of simple smooths for age and screen position (unit-normalised per
district) with a random intercept for district identifier, showed no reliable relation between these
variables. The same outcome was observed in a GAMM predicting age as a function of gender and
unit-normalised screen position, again including a random intercept for district identifier.
To check that ballot order effects do not depend on the normalisation of rank and the use
of GAMMs, we also fitted Cumulative Link Mixed Models (CLMMs), using the ordinal package
Christensen (2019) in R to fit CLMMs and a combination of the ggeffects (L¨udecke, 2018) ggplot2
(Wickham, 2016) packages to plot ordinal effects. We again started by fitting a baseline statistical
model predicting rank, this time untransformed, as a linear combination of age (centred), gender,
and presence of a party logo in the picture, adding a random intercept over district ID (AIC =
11,002.92). We tested the inclusion of random slopes over the different predictors but never observed
an improvement in model fit. All independent variables were significant predictors of rank (p < 0.001)
with effects consistent with what we reported in the main analysis. Predicted rank increases with
age, women tended to rank better than men, not having a picture was detrimental while showing a
party logo only marginally helped.
26
We then added screen position (unit-normalised) as a predictor variable and included a ran-
dom slope for unit-normalised screen position over district ID. Since in the GAMM we observed
a quadratic effect of screen position on rank (unit-normalised), we also included a quadratic ef-
fect of screen position. Screen position was normalised to avoid the mechanistic relation between
screen position and rank, by which only larger districts had higher ranks and screen position val-
ues. By normalising screen position, we have the same range of values for all districts and can
estimate to what extent a candidate’s relative position on screen influenced the final rank. Consis-
tent with the main analysis, screen position had a reliable quadratic effect on rank and a reliable
linear effect. The CLMM, which only included a linear effect of screen position, had an AIC of
10,979.22 (∆AIC = 28.49 compared to the baseline CLMM), while the CLMM, which also included
a quadratic effect of screen position, had an AIC of 10,957.68 (∆AI C = 50.02 compared to the
baseline CLMM), confirming that the quadratic term improves the model fit. Detailed statistics and
output are available in the R workspaces and can be replicated with the provided scripts. All other
predictors remained significant. Figure 7 shows the effect of screen position on rank, estimated using
CLMMs. The figure shows that, for candidates featuring at the top of the screen, it is likelier to rank
at the top; this probability decreases when the normalised screen position increases, with a small
increase for candidates towards the bottom of the page, in line with the quadratic effect observed in
the GAMM. The probability of ranking lower follows the opposite pattern, although the magnitude
is lower - this follows from the fact that while every district has a candidate ranking at position
1, fewer and fewer districts have candidates with ranks over 50, which automatically decreases the
probability of this outcome. This is one of the main reasons why we presented the GAMM in the
main analysis.
We thus replicate the ballot order effect on rank in the 2012 Parlamentarie using a CLMM which
construes the DV as a fully ordinal variable, complementing the analysis we present in the main
paper using GAMMs. Candidates appearing towards the bottom of the page were systematically
disadvantaged and ranked lower on average.
We now move to consider the relationship between the ballot order effect and district size. To this
end, we fitted separate CLMs, one per district (hence we did not have to consider random effects),
predicting the candidate rank as a function of gender (if applicable), age, and screen position.
We did not consider the presence of a party logo in the candidate picture because the model was
unidentifiable in some districts due to sparsity. We then extracted the tstatistic of screen position
in each district-specific CLM and plotted it against the number of candidates in each district. The
result is shown in Figure 8.
27
0.00
0.02
0.04
0.06
0.00 0.25 0.50 0.75 1.00
unit normalised screen position
predicted probability
first mid last
Rank
Effect of screen position (unit normalised) on rank
Figure 7: Effect of screen position (unit-normalized) on rank estimated via a Cumulative Link Mixed
Model (CLMM) also including age (centred), gender and presence of a party logo in the candidate
picture, including a random intercept for district ID and random slopes for screen position over
district ID. The plot shows the predicted probability (y-axis) that a candidate featuring in a given
screen position (x-axis) ended up at a given rank (colour legend).
28
r = 0.591 [ 0.286 , 0.787 ]r = 0.591 [ 0.286 , 0.787 ]r = 0.591 [ 0.286 , 0.787 ]r = 0.591 [ 0.286 , 0.787 ]r = 0.591 [ 0.286 , 0.787 ]r = 0.591 [ 0.286 , 0.787 ]r = 0.591 [ 0.286 , 0.787 ]r = 0.591 [ 0.286 , 0.787 ]r = 0.591 [ 0.286 , 0.787 ]r = 0.591 [ 0.286 , 0.787 ]r = 0.591 [ 0.286 , 0.787 ]r = 0.591 [ 0.286 , 0.787 ]r = 0.591 [ 0.286 , 0.787 ]r = 0.591 [ 0.286 , 0.787 ]r = 0.591 [ 0.286 , 0.787 ]r = 0.591 [ 0.286 , 0.787 ]r = 0.591 [ 0.286 , 0.787 ]r = 0.591 [ 0.286 , 0.787 ]r = 0.591 [ 0.286 , 0.787 ]r = 0.591 [ 0.286 , 0.787 ]r = 0.591 [ 0.286 , 0.787 ]r = 0.591 [ 0.286 , 0.787 ]r = 0.591 [ 0.286 , 0.787 ]r = 0.591 [ 0.286 , 0.787 ]r = 0.591 [ 0.286 , 0.787 ]r = 0.591 [ 0.286 , 0.787 ]r = 0.591 [ 0.286 , 0.787 ]r = 0.591 [ 0.286 , 0.787 ]r = 0.591 [ 0.286 , 0.787 ]
−1
0
1
2
3
25 50 75 100
n candidates in district
screen position t stat
Screen position t stat on rank ~ n of candidates
Figure 8: Relation between district size (x-axis) and the tstatistic of the effect of screen position on
rank (y-axis) in a series of CLMs fitted for each district, with a linear fit and the linear correlation
coefficient with the corresponding CI.
29
In small districts, the effect of screen position tends to be weak and does not have a consistent
direction. However, districts feature more than 35 candidates, and the effect of screen position on
rank becomes stronger. The linear fit and the correlation confirm that the effect of screen position
tends to get stronger and penalise candidates appearing at the bottom of the list more when more
candidates compete.
These robustness checks confirm a reliable effect of a candidate’s position on screen in each
district and the final ranking of the candidate in the district election during the 2012 Parlamentarie,
such that candidates appearing at the top of the page, due to the initials of their surname, had
a higher chance of ranking on top in the election outcome. This effect is, however, moderated by
the number of candidates competing in a district, with candidates at the bottom of the page being
favoured in small districts and further penalised in larger districts. Figure 9 and Figure 10 show plots
of the effect of screen position on rank, estimated using CLMs and GAMs respectively, separately for
each district to provide a more complete picture of the effect under investigation. It is interesting to
note that darker lines in Figure 9, indicating the probability of ranking first, have mixed directions
in smaller districts, appearing at the top, while dark blue lines consistently decrease while going to
the right on the x-axis in larger districts. This indicates that the probability of ranking at the top
decreases when the candidate is found towards the bottom of the page. A similar pattern is observed
with GAMs in Figure 10, where we see fitted lines pointing in different directions for smaller districts
while consistently pointing upward for larger districts, indicating that the predicted rank increases
when the candidate is found at the bottom of the screen.
D Ballot order effect spillover to parliamentary elections
Finally, we analysed whether the effect of screen position (unit-normalised) on the probability a
candidate had of being elected to parliament correlated with district size. Figure 11 shows a scatter-
plot with district size on the x-axis and the Z statistic of screen position in a GLMER model fitted
separately for each district (controlling for age (global median centering here provided a better fit
than median centering per district) and gender where applicable). A linear fit is overlayed, and an
indication of Pearson’s correlation coefficient with the corresponding 95% CI is provided.
The correlation is not reliable, suggesting that the number of candidates in a district did not mod-
ulate the effect of screen position (unit-normalised) on the probability that a candidate competing
in the 2012 Parlamentarie entered the Italian parliament in 2013.
30
7
12
23
13
11
3
19
1
21
8
15
20
4
22
24
14
2
18
17
26
25
6
16
29
30
5
9
10
31
0 20 40 60 0 20 40 60 80 0 20 40 60 80 0 20 40 60 80 0 25 50 75 100
0 20 40 60 0 20 40 60 0 20 40 60 0 20 40 60 0 20 40 60 0 20 40 60
0 10 20 30 0 10 20 30 0 10 20 30 0 10 20 30 40 0 10 20 30 40 0 20 40
0 10 20 0 10 20 0 10 20 0 10 20 30 0 10 20 30 0 10 20 30
5 10 5 10 15 5 10 15 0 5 10 15 20 0 5 10 15 20 0 5 10 15 20 25
0.038
0.040
0.042
0.044
0.02
0.03
0.04
0.010
0.015
0.020
0.025
0.01
0.02
0.03
0.04
0.05
0.06
0.00
0.01
0.02
0.03
0.04
0.00
0.01
0.02
0.03
0.04
0.005
0.010
0.015
0.020
0.025
0.005
0.010
0.02
0.04
0.06
0.02
0.03
0.04
0.05
0.02
0.04
0.06
0.01
0.02
0.03
0.04
0.01
0.02
0.03
0.03
0.06
0.09
0.12
0.01
0.02
0.03
0.04
0.05
0.025
0.030
0.035
0.01
0.02
0.03
0.004
0.008
0.012
0.016
0.04
0.05
0.06
0.07
0.02
0.03
0.04
0.05
0.01
0.02
0.03
0.04
0.010
0.015
0.020
0.025
0.030
0.01
0.02
0.03
0.05
0.06
0.07
0.08
0.09
0.00
0.05
0.10
0.02
0.04
0.06
0.00
0.01
0.02
0.03
0.04
0.01
0.02
0.03
screenOrder
predicted probability
first mid last
Rank
Rank probability by screen position in every district
Figure 9: Effect of screen position on rank estimated separately for every district using CLMs, controlling for age (centred) and gender (when
applicable). Districts are ordered from left to right, top to bottom, according to the number of candidates in the district (x-axis). The y-axis shows
the predicted probability that a candidate appearing on a given screen position ended up at a certain rank (colour legend) after the election.
31
7
12
23
13
11
3
19
1
21
8
15
20
4
22
24
14
2
18
17
26
25
6
16
29
30
5
9
10
31
0 25 50 75 100 0 25 50 75 100 0 25 50 75 100 0 25 50 75 100 0 25 50 75 100
0 25 50 75 100
0.00
0.25
0.50
0.75
0.00
0.25
0.50
0.75
0.00
0.25
0.50
0.75
0.00
0.25
0.50
0.75
0.00
0.25
0.50
0.75
position on screen
predicted rank
Rank by screen position (GAM)
Figure 10: Effect of screen position on rank estimated separately for every district using GAMs, controlling for age (centred) and gender (when
applicable). Districts are ordered from left to right, top to bottom, according to the number of candidates in the district (x-axis). The y-axis shows
the predicted rank.
32
r = −0.292 [ −0.616 , 0.116 ]r = −0.292 [ −0.616 , 0.116 ]r = −0.292 [ −0.616 , 0.116 ]r = −0.292 [ −0.616 , 0.116 ]r = −0.292 [ −0.616 , 0.116 ]r = −0.292 [ −0.616 , 0.116 ]r = −0.292 [ −0.616 , 0.116 ]r = −0.292 [ −0.616 , 0.116 ]r = −0.292 [ −0.616 , 0.116 ]r = −0.292 [ −0.616 , 0.116 ]r = −0.292 [ −0.616 , 0.116 ]r = −0.292 [ −0.616 , 0.116 ]r = −0.292 [ −0.616 , 0.116 ]r = −0.292 [ −0.616 , 0.116 ]r = −0.292 [ −0.616 , 0.116 ]r = −0.292 [ −0.616 , 0.116 ]r = −0.292 [ −0.616 , 0.116 ]r = −0.292 [ −0.616 , 0.116 ]r = −0.292 [ −0.616 , 0.116 ]r = −0.292 [ −0.616 , 0.116 ]r = −0.292 [ −0.616 , 0.116 ]r = −0.292 [ −0.616 , 0.116 ]r = −0.292 [ −0.616 , 0.116 ]r = −0.292 [ −0.616 , 0.116 ]
−2
−1
0
1
25 50 75 100
n candidates
Z statistic
Screen position Z statistic ~ n candidates
Figure 11: Relation between number of candidates in a district (x-axis) and the Zstatistic of the
effect of screen position on rank (y-axis) in a series of CLMs fitted for each district, with a linear fit
and the linear correlation coefficient with the corresponding CI.
33
References
Achen, C., L. Bartels, C. H. Achen, and L. M. Bartels (2017). Democracy for realists. Princeton
University Press.
Baayen, R. H., S. Vasishth, R. Kliegl, and D. Bates (2017). The cave of shadows: Addressing the
human factor with generalized additive mixed models. Journal of Memory and Language 94,
206–234.
Ballew, Charles C., I. and A. Todorov (2007). Predicting political elections from rapid and unre-
flective face judgments. Proceedings of the National Academy of Sciences 104(46), 17948–53.
Bartels, L. M. (1996). Uninformed votes: Information effects in presidential elections. American
Journal of Political Science 40, 194–230.
Bates, D., M. M¨achler, B. Bolker, and S. Walker (2015). Fitting linear mixed-effects models using
lme4. Journal of Statistical Software 67 (1), 1–48.
Bickerton, C. J. and C. I. Accetti (2018, 5). ‘techno-populism’ as a new party family: the case of
the five star movement and podemos. Contemporary Italian Politics 10, 132–150.
Biorcio, R. and R. Sampugnaro (2019). Introduction: The five-star movement from the street to
local and national institutions. Contemporary Italian Politics 11 (1), 5–14.
Bordignon, F. and L. Ceccarini (2015, 12). Five stars and a cricket. beppe grillo shakes italian
politics. South European Society and Politics 18, 427–449.
Burghardt, K., E. F. Alsina, M. Girvan, W. Rand, and K. Lerman (2017, 3). The myopia of crowds:
Cognitive load and collective evaluation of answers on stack exchange. PLoS ONE 12.
Burghardt, K., T. Hogg, and K. Lerman (2018). Quantifying the impact of cognitive biases in
question-answering systems. In Twelfth International AAAI Conference on Web and Social Media.
Christensen, R. H. B. (2019). ordinal—regression models for ordinal data. R package version
2019.12-10. https://CRAN.R-project.org/package=ordinal.
Darcy, R. and M. Mackerras (1993). Rotation of ballots: Minimizing the number of rotations.
Electoral Studies 12 (1), 77–82.
D¨aubler, T. and L. Rudolph (2020). Cue-taking, satisficing, or both? quasi-experimental evidence
for ballot position effects. Political behavior 42 (2), 625–652.
34
Deseriis, M. (2021). Rethinking the digital democratic affordance and its impact on political repre-
sentation: Toward a new framework. New Media & Society 23, 2452–2473.
Dev, H., K. Karahalios, and H. Sundaram (2019). Quantifying voter biases in online platforms: An
instrumental variable approach. Proceedings of the ACM on Human-Computer Interaction 3.
Estlund, D. and H. Landemore (2018). The epistemic value of democratic deliberation, pp. 113–131.
Oxford University Press New York.
Fiske, S. T. and S. E. Taylor (1991). Social cognition. From brains to culture. Mcgraw-Hill Book
Company.
Gerbaudo, P. (2018). The digital party: Political organisation and online democracy. Pluto Press.
Guglielmo, M. (2021). Anti-party digital parties between direct and reactive democracy. the case of
la france insoumise. In Digital Parties, pp. 127–148. Springer.
Ho, D. E. and K. Imai (2006, 9). Randomization inference with natural experiments: An anal-
ysis of ballot effects in the 2003 california recall election. Journal of the American Statistical
Association 101, 888–900.
Kahneman, D. (2011). Thinking, fast and slow. Macmillan.
Krosnick, J. A. (1991). Response strategies for coping with the cognitive demands of attitude
measures in surveys. Applied Cognitive Psychology 5 (3), 213–236.
Kuznetsova, A., P. B. Brockhoff, and R. H. B. Christensen (2017). lmerTest package: Tests in linear
mixed effects models. Journal of Statistical Software 82 (13), 1–26.
Lau, R. and D. P. Redlawsk (2006). How voters decide: Information processing during electoral
campaigns. Cambridge University Press.
Lau, R. R. and D. P. Redlawsk (2001). Advantages and disadvantages of cognitive heuristics in
political decision making. American Journal of Political Science 45, 951.
Lerman, K. and T. Hogg (2014). Leveraging position bias to improve peer recommendation. PloS
one 9 (6), e98914.
Lutz, G. (2010). First come, first served: The effect of ballot position on electoral success in open
ballot pr elections. Representation 46, 167–181.
35
L¨udecke, D. (2018). ggeffects: Tidy data frames of marginal effects from regression models. Journal
of Open Source Software 3 (26), 772.
Mair, P. and R. Katz (2002). The ascendancy of the party in public office: Party organizational
change in twentieth-century democracies. Political parties 24, 113–36.
Malhotra, N. (2008). Completion time and response order effects in web surveys. Public Opinion
Quarterly 72, 914–934.
Marcinkiewicz, K. (2014, 3). Electoral contexts that assist voter coordination: Ballot position effects
in poland. Electoral Studies 33, 322–334.
Meredith, M. and Y. Salant (2013, 3). On the causes and consequences of ballot order effects.
Political Behavior 35, 175–197.
Miller, J. M. and J. A. Krosnick (1998). The impact of candidate name order on election outcomes.
The Public Opinion Quarterly 62, 291–330.
Mudde, C. (2004). The populist zeitgeist. Government and opposition 39 (4), 541–563.
Nairne, J. S. (1988). A framework for interpreting recency effects in immediate serial recall. Memory
& Cognition 16 (4), 343–352.
Roberts, C., E. Gilbert, N. Allum, and L. Eisner (2019, 11). Research Synthesis: Satisficing in
Surveys: A Systematic Review of the Literature. Public Opinion Quarterly 83 (3), 598–626.
Roßmann, J., T. Gummer, and H. Silber (2017, 10). Mitigating Satisficing in Cognitively Demanding
Grid Questions: Evidence from Two Web-Based Experiments. Journal of Survey Statistics and
Methodology 6 (3), 376–400.
Simon, H. A. (1956). Rational choice and the structure of the environment. Psychological Re-
view 63 (2), 129–38.
S¨oderlund, P., ˚
A. von Schoultz, and A. Papageorgiou (2021). Coping with complexity: Ballot
position effects in the finnish open-list proportional representation system. Electoral Studies 71,
102330.
Surowiecki, J. (2005). The wisdom of crowds. Anchor.
Tronconi, F. (2018, 1). The italian five star movement during the crisis: Towards normalisation?
South European Society and Politics 23, 163–180.
36
van Erkel, P. F. and P. Thijssen (2016). The first one wins: Distilling the primacy effect. Electoral
Studies 44, 245–254.
van Rij, J., M. Wieling, R. H. Baayen, and H. van Rijn (2020). itsadug: Interpreting time series
and autocorrelated data using gamms. R package version 2.4.
Vittori, D. (2017). Podemos and the five-star movement: Populist, nationalist or what? Contem-
porary Italian Politics 9 (2), 142–161.
Wickham, H. (2016). ggplot2: Elegant Graphics for Data Analysis. Springer-Verlag New York.
Wood, S. N. (2017). Generalized Additive Models.
37
