Voting, fast and slow: Ballot order and likeability eﬀects in
the Five Star Movement’s 2012 online primary election
Francesco Marolla∗1,2, Angelica Maineri†1,3, Jacopo Tagliabue‡4, and Giovanni
11Department of Sociology, Tilburg School of Behavioral Sciences, Tilburg University
2Department of Sociology and Social Research, University of Trento
3Erasmus School of Social and Behavioral Sciences, Erasmus University Rotterdam
4New York University, Tandon School of Engineering
5Department of Cognitive Science and Artiﬁcial Intelligence, Tilburg School of Humanities and Digital
Sciences, Tilburg University
January 30, 2023
This paper has been accepted for publication in Contemporary Italian Politics - submitted on
June 25th 2022, revised on September 11th 2022, accepted for publication on December 3rd 2022.
We document ballot order eﬀects in the 2012 Parlamentarie, the online primary election
held by the Italian Five-star Movement to select the candidate Members of Parliament in the
2013 Italian general elections. We show that candidates appearing towards the top of the
screen systematically ranked higher in preferences. This eﬀect holds controlling for candidates’
socio-demographic features. We also show that the number of competing candidates moderates
ballot order eﬀects, with a stronger penalty for candidates appearing at the bottom of the
page in more crowded competitions. Finally, we show the inﬂuence of candidates’ likeability.
Our results conﬁrm for the ﬁrst time that ballot order eﬀects and likeability eﬀects, already
documented in traditional paper-based elections, are also found in online set-ups. We conclude
by highlighting how the online medium, if properly leveraged, has the potential to reduce the
inﬂuence of such biases.
Keywords: Five Star Movement; Digital Democracy; Online elections; Cognitive heuristics;
Ballot order eﬀects; Satisﬁcing
In December 2012, the Italian Movimento Cinque Stelle (Five-star Movement, M5s) launched an
unprecedented large-scale experiment in direct online democracy, the Parlamentarie. For the ﬁrst
time in Italy, a political formation allowed its party members to vote for the MP candidates running
for the subsequent general election, through an online decision-making platform. Members, organised
in the same electoral districts as those used for the subsequent parliamentary elections, had to
decide among more than 1,400 candidate MPs. Holding an online primary where party members
could directly choose the candidates for the general election was consistent with the Movement’s
commitment to enhancing citizens’ participation and avoiding candidates being chosen by party
leaders. In this study, we analyse the role that a set of decision heuristics played in the process:
the evidence shows that voters relied on cognitive shortcuts, undermining the promises that ‘direct
democracy’ would remove the traditional biases of party-based politics. However, we also suggest
that most biases could have been reduced had there been a proper decision-making setting.
The 2012 Parlamentarie adopted a rather straightforward procedure. Voters visited a web page
with candidates listed alphabetically by surname, with a self-uploaded picture, the name and sur-
name, a few demographic details, and a voting button. By clicking on the candidate’s name, voters
could – but, crucially, were not required to – visit a separate page, which provided the candidate’s
CV and a short video presentation. Voters could choose up to three candidates and had four days
to cast their votes.
Based on these implementation choices and the election context, one can formulate two hypothe-
ses about cognitive heuristics. On the one hand, voters participating in the primary can be assumed
to be highly motivated: as the literature suggests, satisﬁcing behaviour (Simon, 1956) – in which
participants provide the ﬁrst satisfactory answer instead of the optimal one (Krosnick, 1991; Roberts
et al., 2019) – is known to decrease with higher motivation (Krosnick, 1991; Roßmann et al., 2017).
On the other hand, the fact that political diﬀerences might have been scarce in a primary and that
voters could vote without checking all candidates’ political stances might lead to cognitive heuristics
playing a larger role in reaching a decision. Moreover, following previous studies (Meredith and
Salant, 2013; S¨oderlund et al., 2021), the ballot order eﬀect is expected to be moderated by the
number of candidates in a district: whereas candidates appearing ﬁrst are always advantaged, we
hypothesise that candidates appearing last are penalised in districts with more candidates (due to
satisﬁcing) and advantaged in districts with fewer candidates due to memory eﬀects (Nairne, 1988).
We investigate whether ballot order and likeability inﬂuenced the election outcome in the 2012
Parlamentarie, considering the candidate’s rank as our target variable1. We controlled for gender,
age, and certain features of the self-uploaded pictures (see Section Parlamentarie 2012 for further
details) to ensure that any eﬀect of ballot order was not tainted by other potential voting-cues. Fur-
thermore, we examined the inﬂuence of candidates’ likeability, broadly construed as the impression
elicited by a candidate picture and operationalised by asking participants in an online survey how
likely they would be to vote for a candidate based only on their picture, to investigate whether more
likeable candidates had an advantage (Lau and Redlawsk, 2001; Ballew and Todorov, 2007).
In line with previous research on ballot elections (van Erkel and Thijssen, 2016; Marcinkiewicz,
2014; Miller and Krosnick, 1998), we provide evidence of ballot order aﬀecting the outcome of the
Parlamentarie. We further show a robust eﬀect of likeability that exists alongside the ballot order
eﬀect. Therefore, candidates were more likely to attract votes if they appeared towards the top
of the screen and if they appeared more likeable from the self-uploaded picture. The number of
candidates in a district moderated the ballot order eﬀect in line with our hypothesis: candidates
appearing at the bottom of the list were advantaged in districts with fewer candidates. These results
challenge the rhetoric of the M5s according to which the mere shift to an online setting improves
the quality of crucial democratic processes such as intra-party elite selection. However, the online
set-up allows for more eﬀective countermeasures than traditional paper-based elections. We return
to these points in the discussion after introducing the theoretical framework, describing the data
and statistical methods, and presenting the empirical results.
2 Theoretical framework
2.1 Decision-making under online settings: what can go wrong?
Elections represent a crucial opportunity for citizens to aﬀect democracies through their decision-
making. Citizens’ rationality plays an important role in classic democratic theory, which posits
that an informed and attentive citizenry is required for democracy to work properly (Estlund and
Landemore, 2018). However, more recent evidence from cognitive approaches to voting has raised
several doubts about the optimism of such assumptions (Achen et al., 2017). Indeed, most citizens
know or care little about politics, making assumptions of rationality in political decision-making
unrealistic. In the light of evidence from cognitive theories of human decision-making such as the
dual-process theory (Kahneman, 2011), we know that individuals tend to adopt cognitive shortcuts to
1The share of votes was unfortunately not available for all districts.
deal with the cost that such tasks impose. For instance, according to Krosnick (1991), satisﬁcing is a
function of task diﬃculty, respondent’s ability, and motivation, so that the reliance on such shortcuts
increases when complex tasks are perceived as low stakes (for applications in survey research, see,
e.g., Roberts et al., 2019; Roßmann et al., 2017). That is, people act mainly as ‘cognitive misers’,
given their tendency to adopt the easiest solutions to deal with problems (Fiske and Taylor, 1991),
including political decision-making.
In view of these arguments, questions on the reliability of elections as important democratic
processes have been raised. In seminal studies conducted by Lau and Redlawsk (2006, 2001), it
has been argued that, despite the recourse to cognitive shortcuts, voters can vote ‘correctly’, where
correct voting is deﬁned as ‘one that is the same as the choice that would have been made under
conditions of full information’ (Lau and Redlawsk, 2006, pp. 75). In complex contexts such as
electoral campaigns, voters can make sense of politics and decide how to vote by relying on heuristics
such as party aﬃliation, ideology, group endorsement or viability. In addition, it has been argued
that while single individuals are more likely to produce biased decisions, when taken in the aggregate,
individuals can make rational decisions if the proper conditions are met (Surowiecki, 2005). However,
given that some biases are systematic and therefore errors are never truly random (Bartels, 1996),
errors are unlikely to cancel out in the aggregate (Lau and Redlawsk, 2001).
In the realm of ballot elections, ballot order eﬀects are amongst the most documented heuristics
in the literature, as several studies have shown how candidates appearing ﬁrst on the ballot were
systematically advantaged compared to those appearing in the middle or last (Marcinkiewicz, 2014;
S¨oderlund et al., 2021; Lutz, 2010; Miller and Krosnick, 1998; D¨aubler and Rudolph, 2020). Such
evidence indicates that response order eﬀects can impact electoral outcomes. Moreover, in the
absence of the political cues typical of party-based elections (e.g., party aﬃliation and ideology,
Marcinkiewicz, 2014; Lutz, 2010), ballot order can play an even more relevant role in candidate-
based primary elections like the Parlamentarie. While these elections aim to reduce the inﬂuence of
parties on decisions and force citizens to make more informed choices based on candidate competence,
they also increase the complexity of voters’ decisions: as it is unlikely that voters will collect suﬃcient
information on all candidates, such situations increase the cognitive load and the consequent use
of cognitive shortcuts (Meredith and Salant, 2013; S¨oderlund et al., 2021). Moreover, voters might
be more inclined toward cognitive shortcuts in primary elections, where the lack of ideological cues
deprives them of a critical discriminative cue (Marcinkiewicz, 2014).
While studies on human-computer interactions provide additional evidence of the pervasiveness
of response order eﬀects (Burghardt et al., 2017, 2018; Lerman and Hogg, 2014; Dev et al., 2019;
Burghardt et al., 2017, 2018), we argue that any electoral setting should ensure that candidates
have fair chances of being elected due to their competence and stances rather than because of their
positions on the ballot. To the best of our knowledge, no study to date has replicated this evidence
in online elections. In particular, the set-up of the Parlamentarie is unique, as it made it possible for
voters to get further information about candidates within the platform itself, which may facilitate
access to information about candidates when voters make their choices.
2.2 The M5s 2012 online primaries
Since its foundation in 2007, the M5s has strategically used the web to organise its grassroots ac-
tivism and spread its anti-establishment messages (Bordignon and Ceccarini, 2015). While Beppe
Grillo’s blog was articulating the movement’s ideological messages in a ‘top-down’ fashion, grass-root
activists were aggregating and organising using the digital platform Meetup. These two organisa-
tional features correspond to what Mair and Katz (2002) called the ‘party in central oﬃce’ and the
‘party on the ground’, whereas the ‘party in public oﬃce’ was as yet virtually non-existent, aside
from a few council members elected in a handful of towns.
However, the fundamental element of innovation the M5s brought into Italian politics was the use
of digital media to actualise their narrative of citizens’ empowerment. Before the 2013 Italian general
elections, the M5s launched the Parlamentarie, the primary election to recruit the candidates that
would compete in the general elections to become MPs. Unlike any other primary election previously
held in Italy, the Parlamentarie were organised entirely online and aimed not simply to select the
party leader but the entire pool of candidates.
The M5s’ example falls within a framework of digital direct democracy experiments that Eu-
ropean democracies have witnessed over the last decade (other examples being the Spanish party
Podemos (Vittori, 2017), the France Insoumise party (Guglielmo, 2021), and the pirate parties (Ger-
baudo, 2018)). Despite some ideological diﬀerences, these parties share the use of ICTs to bring
citizens closer to institutions in an optic of power decentralisation. In so doing, they aim to improve
the quality of political participation to actualise their project of digital direct democracy. To this
end, internet voting platforms have been used as a tool for decision-making to aﬀord diﬀerent pref-
erence aggregation models, such as intra-party consultations and agenda-setting. By hosting these
deliberative processes on online platforms, parties reduce the costs of political participation for both
citizens and institutions, facilitating monitoring of constituencies (Deseriis, 2021).
The Parlamentarie represent an interesting case-study for at least two reasons. First, they
involved more than 20, 000 voters who chose amongst more than 1, 400 activists as candidate MPs
(Biorcio and Sampugnaro, 2019) in an ecologically valid context, i.e., one in which every actor
involved (party, candidates, voters) has genuine stakes. Second, their outcome had a far-reaching
impact on Italian politics, as the M5s went on to win about 25% of the popular vote at the ensuing
general election, with 163 candidates selected via the Parlamentarie being elected to the Chamber
of Deputies (109) and the Senate (54).
On top of being regular registered members of the M5s by September 2012, candidates needed
to meet several requirements to compete in the Parlamentarie2, meant to shield the movement from
inﬁltration and last-minute opportunists (Tronconi, 2018). Most notably, candidates were ineligible
if they had had previous experience as an MP, but had to have run in a local election, either under
the M5s banner, or as part of a local list aﬃliated with the M5s.
These requirements reﬂect characteristics typical of populist discourses (Mudde, 2004), being a
product of the anti-establishment narrative of the M5s, where the wisdom of the common citizen is
opposed to the corruption of the ruling political elite. Furthermore, political inexperience is seen as a
positive feature, political expertise being associated with the moral corruption of the Italian political
establishment. In addition, the M5s’ ideology has often been linked with technopopulism (Bickerton
and Accetti, 2018), that is, an ideology blending elements of populism (power to the people) with
elements of technocracy (power to the experts). Whereas such a mix might look contradictory at
ﬁrst, the two arguments are jointly used to legitimise the Movement’s populistic claims and eschew
ideological confrontation. Thus, citizens are idealised as the real protagonists of political life and as
having greater expertise than professional politicians.
The purpose of the primary was also to let voters decide, albeit indirectly, on the order in
which candidates would appear on the ballot in the general election, which would in principle be
determined by the number of votes obtained in the Parlamentarie3. Having voters decide on the
order of candidates was particularly important in the M5s’ rhetoric since the electoral law in force
at the time made no provision for preference voting. Candidates were elected to Parliament based
on the share of votes obtained by the party in each district: the more votes, the more candidates
were elected, with candidates obtaining seats (or not) according to the order in which they featured
on the ballot. Therefore, even though some adjustments to the electoral lists for the general election
were made post-hoc (new names were added to the ballots and some candidates withdrew from the
competition before the election), the outcome of the primary may have played a role in determining
2https://www.ilpost.it/2012/12/02/le-primarie- del-movimento- 5-stelle/
3The procedure by which lists for the general election were put together was however not openly communicated
by the FSM.
who entered Parliament in the end.
Voters had to be at least 18-years old at the time of the election, be enrolled in the M5s as of 30
September 2012 and certify their identity digitally by uploading their ID. The voting interface was
rather simple, as shown in Figure 1. Voters landed on a page listing all candidates in the district
in alphabetical order based on the initial letters of their surnames. The landing page provided
information about each candidate’s appearance through a self-uploaded picture, name, gender, age,
and profession. Further details about the candidates and their political priorities could be accessed
by clicking on their names and being redirected to a separate page. Here, voters could ﬁnd the
candidate’s CV, a short introductory video and a statement about the political projects the candidate
would pursue in the event that s/he was elected. It was not however possible to cast a vote on a
candidate’s personal page: for this, voters had to go back to the landing page. The voting procedures
lasted from 3 December to 6 December 2012 and involved 20,252 people (Biorcio and Sampugnaro,
In light of previous studies on ballot order eﬀects, it is immediately clear that this interface created
considerable risks that voting heuristics helped to determine the outcome. First, some names were
immediately visible while others could only be seen after scrolling, and more so in districts with
more candidates competing. Second, voters could cast a vote directly on the landing page without
accessing candidates’ personal information and political statements. Finally, the availability of the
candidate picture on the landing page could have given more likeable candidates an advantage over
3 Materials and Methods
3.1.1 Parlamentarie 2012
The main dataset for this study was scraped in 2013 from the website, then accessible under a CC-
BY-NC-ND license, which hosted the results of the Parlamentarie4. Candidates were grouped into
31 districts, corresponding to the 27 districts in which the Italian territory was divided according to
the electoral system in 2013 and the 4 districts covering the rest of the world for Italians resident
abroad. For each candidate, we scraped their name, surname, age, profession, and picture (wherever
available) and annotated gender based on other demographics. We also derived their position on the
4The page can be now accessed through a way-back machine: https://web.archive.org/web/20121217093818/
Figure 1: Example of candidate selection page on the Rousseau platform where the Parlamentarie
took place. The ﬁrst column shows the self-uploaded picture, followed by the surname and name.
Other columns show age, place of birth, occupation, with the last column devoted to the voting
screen, based on the alphabetical order of the candidates’ surnames, and scraped their ﬁnal rank
(separately for each district) in the election.
We then manually tagged each available picture according to whether it contained a party logo.
To this end, we also considered the founder of the party as a logo to account for a possible party-
endorsement voting-cue, under the hypothesis that candidates featuring a party logo or the party
founder in their picture would boost their credibility in the eyes of voters and so gain an advantage.
We ﬂagged pictures containing the scan of an ID document or drawings/comics/writings as not
containing a picture.
3.1.2 Likeability ratings
To account for the role of likeability, we collected judgments through an online survey distributed on
Proliﬁc and hosted on Qualtrics. To collect likeability ratings, we selected three target districts that
i) varied in terms of the number of candidates competing, thereby improving the generalisability of
our results, and (ii) maximised the number of usable pictures5. We converged on districts number 5
(Lombardia-3; 18 candidates; 18 pictures available), 11 (Emilia-Romagna; 99 candidates; 80 pictures
available) and 21 (Puglia; 61 candidates; 49 pictures available).
Since some of the candidates have since 2012 acquired national signiﬁcance and would be easily
recognised by Italian citizens, we recruited participants from France, Spain, Portugal, and Greece, to
ensure comparable aesthetic standards to those Italian voters are most likely to have, while limiting
the possibility of participants answering on the basis of what they knew of the candidate. We
recruited 176 participants, asking each of them to rate 25 pictures, to obtain 30 ratings per picture
and so improve the reliability of ratings per picture. Participants were paid £7.20/h for their
participation. The experimental design and data management plan were approved by the Research
Ethics and Data Management Committee (REDC) of the Tilburg School of Humanities and Digital
Sciences (TSHD) of Tilburg University, code REDC2020.201. The Online Supplementary Materials,
datasets, ﬁgures generated during the analyses, and R scripts are available for replication purposes
on Dataverse (https://doi.org/10.34894/KE8VVY).
After providing their informed consent, participants were asked to provide details of their gen-
der, age, education, and country, these being collected in order to assess whether there were any
systematic diﬀerences in likeability ratings. Participants who did not provide their consent were
redirected back to Proliﬁc and did not receive payment.
Likeability ratings were obtained by presenting subjects with a candidate’s picture (of the same
size and resolution as the one uploaded to the website, in order to preserve the conditions of the
Parlamentarie as much as possible) and were asked to drag a slider to indicate how likely they would
be to vote for the candidate basing their decision simply on the picture. They were instructed to
move the slider more towards a pole the stronger their intuition about the candidate. The slider
was initially presented in the middle and was anchored between -50 (not at all likely ) and +50
(extremely likely). Candidates were presented randomly to each participant. Participants were
scanned for uncooperative behaviour, considering whether they always left the slider at the initial
position or whether they always dragged the slider to the same extreme of the bar. No participant
5We provide further details about the exclusion criteria for candidate pictures as well as about the experimental
interface and procedure, inclusion criteria for participants, and participants demographics in Appendix A.
showed uncooperative behaviour as deﬁned in this way.
3.2 Statistical approach
We adopted a step-wise regression approach: we started by including a set of control variables and
added the variable of interest (with possible interactions) to assess whether it improved the model’s
ﬁt. The control variables included the candidate’s age, their gender (binary), and the presence
of a party logo in the candidate’s picture (to create a categorical variable with three levels: no
picture, picture without a logo, and picture with a logo). We further considered possible composition
eﬀects to test the possibility that, for example, a 50-year-old candidate would appear young if the
median age in the district was 65 but old if the median age in the district was 35, or that a logo
would draw more attention if only 5% of the pictures in a district showed one as opposed to 20%
of the pictures showing one. Therefore, we derived two measures for age: one applying median
centring considering the global median and one applying median centring separately per district,
i.e. subtracting the district median age from the age of each candidate in the district. We further
computed the proportion of pictures with a logo in each district and the proportion of women in each
district). Finally, we quantiﬁed the model ﬁt using the Akaike Information Criterion (AIC), which
penalises models for the number of parameters they estimate. We thus checked whether adding the
predictor of interest resulted in a reduction of the AIC.
The dependent variable in the main analysis was the candidate’s rank per district. However,
screen position and rank are inevitably related and co-vary, and while every district features a
candidate at screen position 1 and a candidate ranking ﬁrst, not all districts feature a candidate in
screen position 57 and a candidate ranking 57th. It is very diﬀerent for a candidate to be in position
18 in a district with 18 rather than 80 candidates: considering screen position as the absolute distance
from the top of the page does not allow us to take account of the fact that the last position in the list
can be very salient as well. Therefore, we transformed both rank and screen position by normalising
the values to the unit range, using the formula unit(xi) = xi−min(X)/max(X)−min(X)∀xi∈X
where X is the vector of screen position values or ranks: candidates appearing at the top of a page
or ranking ﬁrst would get a score of 0, and candidates appearing last on the page or ranking last
would get a score of 1, with intermediate values depending on the number of candidates in a district.
We used Generalised Additive Mixed Models6(GAMMs, Baayen et al., 2017) to model rank
(unit-normalised) as a function of the independent variables of interest. Both GAMMs and CLMMs
6Appendix C presents a replication of the analysis of rank using Cumulative Link Mixed Models (CLMM), treating
rank (not unit normalised) as an ordinal variable.
implement a multilevel approach to account for district-speciﬁc variance by including random inter-
cepts and (non-linear) slopes for the relevant independent variables. Moreover, GAMMs allowed us
to model possible non-linear eﬀects that continuous predictors may have on rank: this is particularly
important considering that we have hypothesised that the ballot order eﬀect may be quadratic in
smaller districts. Unless otherwise speciﬁed when describing results, we included continuous pre-
dictors as simple smooths; used splines to model non-linearities, and did not limit the number of
inﬂection points an estimated eﬀect was allowed to have.
To check whether the number of candidates moderates the eﬀect of screen position, we took the
log of the number of candidates (base 2), to reduce the long right tail that would make estimates
brittle for districts with several competing candidates, and included a partial tensor product between
number of candidates and screen position to ﬁt an interaction between the two.
Finally, we analysed the possible eﬀect of likeability on rank. We again used GAMMs predicting
rank as a function of age, gender, screen position and participants’ ratings in the online survey,
including random intercepts for rater ID to account for the fact that ratings provided by the same
participant are likely to have a higher correlation than ratings provided by diﬀerent raters. We did
not consider the presence of party logos in the pictures because we selected pictures which did not
have a party logo in the ﬁrst place to avoid biases due to subjects’ recognition of the logo.
4.1 Ballot order eﬀects on rank
Our ﬁrst analysis focused on possible ballot order eﬀects on the ranking of candidates in each district.
We ﬁtted a baseline GAMM predicting unit-normalised rank as a linear combination of age, gender
and presence of a logo in the picture7. We ﬁrst compared global median centring and district median
centring, and observed that the latter provided a better ﬁt. We then tested whether the eﬀect of
gender was moderated by the proportion of women in each district, testing whether an interaction
between gender and proportion of women per district improved model ﬁt over the simple eﬀect of
gender. This interaction resulted in a higher AIC and was thus discarded. The same happened
with the interaction between presence of a logo and share of pictures with a logo per district, which
was also discarded from further analyses. The baseline model thus included a smooth term for
age (median centred by district), parametric terms for gender and presence of logo, and a random
7We used the mgcv package (Wood, 2017) in R to ﬁt GAMMs and the itsadug package (van Rij et al., 2020) to
intercept for district identiﬁers (AIC = 222.718). No random slope improved model ﬁt.
We then added screen position (unit-normalised)8to this model (random slopes for screen po-
sition did not improve model ﬁt). This model (AI C = 171.795, adj.r2= 0.243) improved over
the baseline (∆AIC = 50.923), showing that screen position is a signiﬁcant predictor of rank
(edf = 5.144, Ref.df = 6.253, F= 9.767, p < 0.001). Figure 2 displays all the eﬀects visually
(all eﬀects were statistically signiﬁcant, exact coeﬃcients are provided in Appendix B).
0.0 0.2 0.4 0.6 0.8 1.0
0.3 0.4 0.5 0.6 0.7
screen position (normalized)
fitted values, excl. random
−20 −10 0 10 20 30
0.3 0.4 0.5 0.6 0.7 0.8
Age (median centered by district)
fitted values, excl. random
0.2 0.3 0.4 0.5 0.6
0.4 0.5 0.6 0.7 0.8 0.9 1.0
Party logo in picture
Figure 2: Eﬀects of screen position (unit-normalized; top-left), age (median centred by district, top-
right), gender (bottom-left), presence of a logo in the candidate picture (’logo’ indicates pictures with
a party logo; ’no logo’ indicates pictures without a party logo; ’no pic’ indicates candidates who did
not upload a picture at all; bottom-right). Low values on the y axis indicate better rankings. The
underlying statistical model is a GAMM with ranking (uni-normalized as the dependent variable),
continuous predictors included as simple smooths, and a random intercept for district.
8To ensure that the normalisation did not introduce any artefact, we also ﬁtted a GAMM with the same DV and
IVs but replacing normalised screen position with the original variable. The predictor was signiﬁcant (edf = 3.258,
Ref.df = 4.070, F= 11.24, p < 0.001) but the model had a worse ﬁt (AI C = 183.5, adj.r2= 0.235), showing that
normalised screen position best captures the relation between screen position and rank in each district.
The eﬀect of screen position appears roughly quadratic: candidates appearing towards the top
of the screen (left side of the x-axis) had an advantage over other candidates. However, candidates
appearing towards the bottom of the screen (right end of the x-axis) tended to rank higher (bet-
ter) than candidates appearing in the middle of the list. We also observe a linear eﬀect of age,
with younger candidates relative to their district ranking ﬁrst, as well as an advantage for women.
Candidates showcasing a party logo in their picture had a slight yet signiﬁcant advantage over can-
didates who did not, but the dominant eﬀect is the stark penalty for candidates who did not upload
We then included an interaction between screen position (unit-normalised) and number of can-
didates per district (logged)9to check whether the quadratic eﬀect was primarily driven by smaller
districts, in line with our predictions. This interaction was statistically signiﬁcant (edf = 3.129,
Ref.df = 3.701, F= 3.869, p < 0.01) and improved the model ﬁt over the model which included
screen position alone (AIC = 163.871, ∆AIC = 7.924). Visual inspection of the tensor product
revealed the expected pattern, visualised in the centre panel of Figure 3 (coeﬃcients are provided
in Appendix B). Darker shades of blue indicate lower predicted ranks, while orange shades indicate
higher predicted ranks. In the right-hand panel we display the eﬀect of screen position at diﬀerent
numbers of candidates, showing the composite eﬀect of the main eﬀect and the partial tensor prod-
uct: the advantage of candidates displayed at the top exists regardless of number of candidates, but
is slightly weaker when fewer candidates compete. In contrast, we see that the advantage for candi-
dates shown at the bottom of the list is clear when fewer candidates compete but almost vanishes
in the most crowded districts.
4.2 Likeability eﬀect on rank
We ﬁtted a GAMM model predicting unit normalised rank (to address the diﬀerences in number of
candidates in the three target districts). The independent variables included age (median centred
by district), screen position (unit normalised), gender, and likeability ratings, all in interaction with
district ID. We ﬁrst tested whether all interactions were needed by comparing AIC scores and found
that they all improved model ﬁt, showing that the target eﬀects are diﬀerent in the three target
districts. The model also included a random intercept for rater ID. Figure 4 shows the eﬀects of
interest, allowing for direct comparisons between districts (see Appendix B for the exact coeﬃcients
of screen position and likeability in each district).
9We included screen position as a smooth term and the interaction between screen position and number of candi-
dates as a partial tensor product, limiting the number of allowed inﬂection points to 4.
0.0 0.2 0.4 0.6 0.8 1.0
0.3 0.4 0.5 0.6 0.7
screen position (normalized)
fitted values, excl. random
0.0 0.2 0.4 0.6 0.8 1.0
4.0 4.5 5.0 5.5 6.0 6.5
ti(screen position, n candidates)
screen position (normalised)
0.0 0.2 0.4 0.6 0.8 1.0
0.3 0.4 0.5 0.6 0.7
Screen Position by n candidates
screen position (normalized)
fitted values, excl. random
Figure 3: Non-linear interaction between screen position and number of candidates on rank (unit-
normalised) estimated using a GAMM while controlling for gender, age, and presence of party logos
in the candidate picture. Left: main eﬀect of screen position (x axis, unit normalised) on rank (y axis,
unit normalised). Center: partial tensor product between screen position and number of candidates.
Darker shades of blue indicate lower (better) predicted ranks while orange shades indicate higher
(worse) predicted ranks. Red lines connect points with the same predicted rank. Right: eﬀect of
screen position (x axis, unit normalised) on rank (y axis, unit normalised) for districts with diﬀerent
number of candidates (colour and line type legend), showing how the partial tensor product and the
main eﬀect of screen position combine.
Screen position had quite diﬀerent eﬀects, consistent with our previous analysis: candidates
appearing at the bottom were favoured in district 5 (the smallest). In contrast, in the mid- size and
large districts, candidates appearing at the top were favoured. Likeability had a largely linear eﬀect,
with a slight yet robust non-linearity in the largest district:10 candidates rated as more likeable from
the picture ranked better in all districts, with a stronger eﬀect in district10 (the smallest). Finally,
we see again that women were favoured, and more so in the largest district (district 5 is not shown as
only men competed there). Our results thus suggest that more likeable candidates were advantaged
regardless of how many candidates were competing.
10We probed this by ﬁtting a model only on data points from district 21. First, we included the eﬀect of likeability
as a simple smooth, and then we included it as a parametric term, forcing the model to estimate a linear eﬀect. The
AIC of the ﬁrst model was 17 points lower, conﬁrming that allowing the model to estimate a non-linear eﬀect for
likeability on rank improves model ﬁt.
0.0 0.2 0.4 0.6 0.8 1.0
0.3 0.4 0.5 0.6 0.7
fitted values, excl. random
−40 −20 0 20 40
0.2 0.3 0.4 0.5 0.6
would vote from pic
fitted values, excl. random
Figure 4: Eﬀect of screen position on rank (left) and likeability on rank (right) moderated by district
id (blue, solid line: district 5 (Lombardia 3, 18 candidates); red, dotted line: district 11 (Emilia
Romagna, 100 candidates); yellow, dashed line: district 21 (Puglia, 61 candidates)), estimated using
GAMMs controlling for sex and age (median centred per district). The non-linearity was limited to
a third order polynomial to prevent overﬁtting.
The M5s’ 2012 Parlamentarie provided a unique experiment in online democracy and nearly ideal
conditions for isolating cognitive biases’ eﬀects on elections, which are typically diﬃcult to achieve in
conventional primaries. In this election, voters’ decisions were not inﬂuenced by well-known biases
such as candidates’ aﬃliations to a speciﬁc party faction, or viability (Lau and Redlawsk, 2006,
2001). At the same time, the Parlamentarie oﬀer a real-life case-study of election biases, allowing a
thorough test of our hypotheses outside the lab.
Our results show that the choices of M5s party members are likely to have been aﬀected by
ballot order eﬀects — reported in conventional paper-based elections (van Erkel and Thijssen, 2016;
Lutz, 2010; Marcinkiewicz, 2014) — and candidate likeability (Ballew and Todorov, 2007; Lau and
Redlawsk, 2001). We document how, in general, candidates appearing at the top of the screen were
advantaged as compared to candidates appearing further down the list. We further qualiﬁed this
eﬀect by showing that the number of candidates competing in a district moderates order eﬀects, with
stronger penalties in districts with more candidates competing (S¨oderlund et al., 2021; Meredith and
Salant, 2013). This seems to conﬁrm that the higher the number of candidates, the more voters
resort to satisﬁcing behaviour (Krosnick, 1991). In the context of the Parlamentarie, this eﬀect can
be explained by considering that the larger the number of candidates competing, the more voters
had to scroll and the more time they would have had to spend if they had sought to survey all of
them. In small districts, however, we found evidence of a recency eﬀect, suggesting that candidates
who appeared at the bottom of the page were advantaged, likely due to easier recall and higher
salience after exhausting the candidate list (Nairne, 1988). These ﬁndings are robust net of control
variables related to candidates’ sociodemographic characteristics and composition eﬀects.
We further found a robust eﬀect of likeability in a sample of districts, with more likeable candi-
dates ranking higher. The eﬀect of screen position held even after controlling for likeability. However,
in the smaller district, the eﬀect of screen position was reversed, with candidates appearing at the
bottom of the page ranking higher in the election. This suggests that diﬀerent cognitive biases may
interact in non-trivial ways and paves the way for future studies, which should also consider halo
eﬀects, to assess whether appearing closer to popular candidates may provide a spillover advantage
purely because voters will be more likely to consider a candidate whose name appears close to
another candidate which, for any reason, draws more attention. In addition, future research may
look into which features respondents evaluate when rating candidates’ likeability, since our set-up
did not disentangle which speciﬁc criteria respondents adopted.
The implications of these ﬁndings are ampliﬁed when considering (i) the electoral results achieved
by the M5s (25.55% for the Chamber of Deputies in 2013) and (ii) the electoral rules in place during
the 2013 general election, which ‘projected onto’ the general election the biases of the primary election
(see Appendix D for an analysis showing that candidates’ screen positions in the Parlamentarie
indeed had an eﬀect on their probability of entering parliament in the 2013 general election).
Finally, the covariates we included in the statistical model showed interesting eﬀects on their
own. First, candidates who are younger than their competitors tended to rank higher. This ﬁnding
aligns with the Movement’s rhetoric and the requirement of candidates not to have had prior political
experience in a public oﬃce: younger candidates might appear less compromised with the political
establishment. Moreover, showing the party logo or party founder in the candidate’s picture gave
him/her an advantage, suggesting that in a ﬁeld where diﬀerences among candidates are small, being
in a position to attest one’s history in the movement provided an advantage. Moreover, not uploading
a picture resulted in a strong penalty. The reason for this cannot be gleaned from our analysis, but
we hypothesise that, in a party with a strong emphasis on digital tools and transparency, candidates
not providing their pictures might appear less trustworthy, less committed or less technologically
capable, all liabilities on the M5s platform. Finally, we saw that women had an advantage. However,
the number of women competing in the election was very small (196 out of 1,486 candidates, of which
154 appeared on the ballot in the 2013 general election), suggesting that self-selection might have
played a role, with women competing only if they felt suﬃciently qualiﬁed.
Considering the hypotheses we started from, therefore, we should conclude that the allegedly
higher motivation of voters was not enough to counter the eﬀect of satisﬁcing, operationalised here
through order eﬀects and likeability. It is possible that the design choices of the interface played
a part in determining these eﬀects by creating more favourable conditions for voters to resort to
shortcuts and heuristics and countering the possible advantages oﬀered by the online medium in
making information about candidates more readily available. In addition, such eﬀects may also have
hampered the Movement’s objective of leveraging digital democracy tools to improve the process of
recruitment of new candidates by using open online primaries in place of the conventional approaches
adopted by other parties. The Parlamentarie were successful in removing several barriers to political
access, especially as compared to more structured primaries where parties exert tighter control of
the lists: unfortunately, our analyses show that the implementation hampered these possibilities for
several candidates, who may have been disadvantaged by platform design choices.
Even though we document a robust eﬀect of ballot order and likeability on election ranks in
the Parlamentarie, our analyses remain correlational and cannot be taken to provide evidence that
screen position caused the election outcome. A controlled A/B test would be required to test a causal
relation, e.g., showing to a random subset of voters candidates in alphabetical order vs candidates
listed in a random yet ﬁxed order vs candidates randomly shuﬄed at each access. This experiment
is however not feasible in a real election as it would systematically manipulate the electoral lists
for subsets of the electorate, likely biasing the decision-making process of some of them. For this
reason, we further argue that such an experimental manipulation would not be ethical in a real
election, particularly given the results we have illustrated – results that suggest a robust relation
between ballot order and election outcome, in line with previous studies (van Erkel and Thijssen,
2016; Lutz, 2010; Miller and Krosnick, 1998). Considering the stakes in such processes, extant
knowledge should be leveraged to make a platform that minimises the inﬂuence of cognitive biases,
adopting the more pervasive randomisation allowed by the medium and the electoral system. Mock
elections are a viable option for implementing such a manipulation (Lau and Redlawsk, 2006), but
they lack ecological validity and would not allow us to draw ﬁrm conclusions given that participants
are unlikely to adopt the same decision-making processes in an election with no real stakes.
Although we were unable to analyse the Movement’s subsequent online primary elections due to
changes in the license under which the website was made public, the 2012 Parlamentarie provided an
environment that was less inﬂuenced by external dynamics. Candidates did not have any advantage
due to incumbent status, and media coverage was limited due to the lack of a party in public
oﬃce and to a ban on television appearances on the part of party members. Our ﬁndings further
highlight the wide eﬀects of biases in the 2012 Parlamentarie: candidates receiving an advantage
due to likeability or screen position went on to gain an incumbent advantage and name recognition,
improving their chances in the following elections (Meredith and Salant, 2013). However, it could
be the case that some of the candidates enjoyed local popularity because they had run for previous
local elections and lost or received occasional media coverage in the local press. While our focus in
this work was on information that was immediately available or determined by the voting platform,
future work can look into ways of quantifying name recognition for candidates at the time, using
news or social media to track user mentions and to test whether name recognition predicts election
rank. Future work can also leverage the data that we release to test further predictors. Information
provided in videos and personal pages would have been particularly interesting in this respect but
could not be scraped as it was only accessible after log-in.
Despite showing that online elections are subject to the same biases found in paper-based elec-
tions, our results actually suggest that the online set-up oﬀers several opportunities to limit the
inﬂuence of such biases. If anything, the Parlamentarie represent a wasted opportunity to counter
or limit the impact of cognitive heuristics, whose eﬀects have been known for a long time in the
political science literature (van Erkel and Thijssen, 2016; Lutz, 2010; Miller and Krosnick, 1998).
Useful suggestions come from research on clicking behaviour (Dev et al., 2019). For instance, the
online set-up oﬀers the chance to randomise the order of the candidates for each user’s access to
reduce ballot order eﬀects. Such a solution has already been tested in conventional elections (Ho
and Imai, 2006; Darcy and Mackerras, 1993), but it is costly and challenging to implement. In
contrast, the online setting provides a cheap and feasible opportunity in this direction. Even though
the primacy bias is impossible to eliminate due to the way in which human attention works (Simon,
1956), randomising the order of the candidates for each voter’s access could at least reduce the eﬀect
of this bias on the election outcome.
Moreover, connected to our results on gender and age eﬀects, another suggestion would be to
conceal impression signals to avoid the inﬂuence on voting decisions of factors other than candidate
competence. Finally, drawing from the evidence that suggests that primacy eﬀects are more likely
to be found in cases of quick completion response (Malhotra, 2008), another counter measure would
be delaying voting. Platforms could set a ﬁxed time before allowing the user to vote to encourage
them to consult as many candidate proﬁles as possible and to counter cost-minimising behaviour.
Such advice can be especially crucial in primary elections, where ideological voting-cues are lacking
and the cognitive costs for voters tend to be higher.
In conclusion, our study has investigated the electoral outcomes of – what was at the time
– an unprecedented experiment in digital democracy – one that took place in Italy in 2012 and
had important implications in terms of institutional representation. We reported a robust ballot
order eﬀect, moderated by the number of competing candidates, and a likeability eﬀect in the 2012
Parlamentarie – the eﬀects – we suggested - depending on satisﬁcing (Krosnick, 1991; Roberts et al.,
2019). Our main wager is that digital platforms have the potential to address existing problems but,
left unchecked, do not necessarily lead to a better quality of decision-making. The good news is that
policymakers can proﬁt from robust evidence and suggestions that the literature oﬀers. Although
cognitive biases will continue to aﬀect the decision- making of many voters, digital platforms provide
several possible solutions if properly leveraged.
This project started almost ten years ago, and followed Hofstadter’s Law11 as the original design
changed and life had its course. We wish to thank the people that at that time helped us in
understanding the FSM, collecting and labelling the data, and taking a ﬁrst stab at all of this.
Francesco Oggiano, Ileana Bego, Camilla Tagliabue and Davide Rossi: your contribution has not
We further thank Giuseppe Arena, Federico Bianchi, Gabriele Mari, and Matteo Colombo for
useful feedback about this work, and Andrea Polonioli, Bingqing Christine Yu and Ciro Greco for
their help in a previous iteration of this research project. We also thank the people that helped
us translate the survey in the target languages: Raquel Garrido Alhama, Kalliopi Dilaveraki, Paris
Mavromoustakos Blom, Nikos Aggelakis, Miguel Egler, and Audrey Patenaude.
7 Authors biographies
Francesco Marolla is a joint PhD candidate for the Sociology and Social research department of
University of Trento and the sociology department of Tilburg University. His work focuses on
11It always takes longer than you expect, even when you take into account Hofstadter’s Law.
populist voting behaviour in a comparative perspective.
Angelica M. Maineri is a PhD candidate in Sociology at Tilburg University and Data Manager at
ODISSEI. Her work focuses on privacy risks in the dataﬁed society and (web) survey methodology.
Jacopo Tagliabue is the Director of AI at Coveo and Adj. Professor of MLSys at NYU. He divides
his time between industry and research, with a focus on natural language processing, information
retrieval and reasoning.
Giovanni Cassani is Assistant Professor at the Department of Cognitive Science and Artiﬁcial
Intelligence at the Faculty of Humanities and Digital Sciences of Tilburg University. His work focuses
on cognitive science, computational linguistics, and data science.
8 Disclosure statement
The authors declare no conﬂict of interest.
A Online survey for likeability ratings
As mentioned in the main article, we recruited participants from other countries than Italy to avoid
that respondents judged pictures considering their political inclinations. We used Proliﬁc’s screeners
to limit the survey availability to subjects who indicated their country of residence is one of the four
aforementioned ones and also indicated one of Spanish, French, Portuguese and Greek as their native
language. The survey was translated from English in each target language by native speakers who
are also ﬂuent in English.
As indicated in the main text, after providing their informed consent, participants were asked
to provide a few demographic details. Gender was coded as a three-level factor (Male, Female,
Non-binary / Third gender); age was binned (18-29, 30-39, 40-49, 50-59, 60-69, 70+); education was
collected by asking whether participants held a college degree (Bachelor, Master’s, PhD) or not;
country was collected by presenting a text box and asking participants in which country they lived.
Subjects wishing not to provide this information could select the option Prefer not to say for the
ﬁrst three questions and could type NA for the last. Demographics were collected to assess whether
systematic diﬀerences in likeability ratings were observed but will not be released to preserve raters’
After completing the demographic questions, participants were presented with a practice trial in
which they were asked to drag a slider to signal whether a rock is animate or inanimate. Candidates
dragging a slider towards the animate pole were presented with a message explaining the intended
use of the slider and asked to drag it towards the inanimate pole. Once the slider was dragged
towards the correct pole, participants could proceed to rate candidates’ pictures. likeability ratings
were obtained using the interface in Figure 5.
Figure 5: Example trial from the online survey we used to collect likeability ratings.
In each trial, participants saw a picture of a candidate (at the same size and resolution as uploaded
on the website to preserve the actual conditions from the Parlamentarie as much as possible) and
were asked the following question: Simply basing your decision on the picture, how likely would you
be to the vote for this candidate?. They then had to drag a slider between the poles Not at all likely
displayed on the left and Extremely likely displayed on the right.
We did not collect likeability ratings for any picture featuring other people than the candidate
for both ethical and practical concerns. First, we could not be sure these other people provided
their consent for their picture to appear on the website. Second, speciﬁcally for minors, we preferred
to avoid uploading such pictures online. Third, it would be unclear for raters which person to rate
when more than one appears in the picture. We also excluded pictures featuring party logos, since
their presence could have a diﬀerent impact on party members voting in a primary election than it
does on raters.
The survey was conducted in September 2021. First of all, there was a strong imbalance along
several demographic variables. As far as the raters’ country is concerned, 109 participants indicated
they came from Portugal, 26 from Spain, 24 from Greece, 12 from France, while 5 did not specify
their provenance. Moreover, 154 raters indicated they were between 18 and 29 years old, 19 between
30 and 39, 2 between 40 and 49, 1 between 50-59. 102 participants indicated having a college-
level degree, 73 indicated they do not, 1 preferred not to share this detail. Finally, 81 participants
identiﬁed as females, 93 as males, 2 as non-binary.
We plotted likeability ratings against rank in the election separately by district, overlaying a
LOESS ﬁt separately by category level for each demographic attribute to inspect for possible sys-
tematic diﬀerences that would prevent us from conﬂating all judgments when predicting rank. We
show the outcome for the variable Country in Figure 6 to illustrate the approach. Similar ﬁgures
for the variables Gender, Age, and Education are available in the data package, as well as the script
we used to conduct the pre-processing of the likeability ratings as downloaded from Qualtrics. It
can be seen that, with some small diﬀerences notwithstanding, the LOESS ﬁt between likeability
and rank is comparable across ratings provided by subjects from diﬀerent countries. The same was
observed for all other demographic variables, indicating that no systematic biases attributable to
demographic characteristics aﬀected likeability ratings.
Next to the aforementioned visual approach, we also ﬁt a linear mixed eﬀect regression (LMER)
model with the lme4 (Bates et al., 2015) and lmerTest (Kuznetsova et al., 2017) packages predicting
likeability ratings as a linear combination of raters’ gender, country, education, and age (binarised,
with the category 18-29 labeled as young and the other categories lumped together and labeled as
notYoung due to sparsity). We also included a random intercept for rater ID and nested random
intercepts for candidate ID within district ID to account for systematic variation attributable to
the speciﬁc rater and the speciﬁc candidate being rated. Finally, due to sparsity, we excluded
participants who self-identiﬁed as non-binary, and who did not disclose their education or country.
No predictor was signiﬁcant, conﬁrming that no systematic diﬀerences in ratings can be attributed
to demographic characteristics. We, therefore, included all likeability ratings in the analysis of rank
reported in the main analysis. This analysis is also available in the data package, with the output
included in the script.
−50 −25 0 25
Country France Greece Portugal Spain
Likability ~ rank (by Country)
−50 −25 0 25 50
−50 −25 0 25 50
Figure 6: Relation between likeability and rank, plotted separately for each district, with superim-
posed LOESS ﬁt. The black line shows the LOESS ﬁt disregarding country information. The ﬁt
over ratings provided by participants who did not disclose their country has been removed due to
the small sample size.
B GAMMs regression for rank prediction
We report here the full regression coeﬃcients and corresponding statistics that resulted from the
model visualised in Figure 2 in the main article. These results are visualised and further described
in Section 4.1 of the main article.
Table 1: Estimates of the GAMM ﬁtted to predict unit normalised rank as a linear combination of
gender, presence of a party logo in the picture, age (median centred per district), and screen position
(unit normalised). Parametric terms are presented ﬁrst, with the estimated bcoeﬃcient, corresponding
standard error (se), tstatistic and pvalue. For the non-linear smooths (age and screen position), the
table shows the estimated degrees of freedom (edf), the reference degrees of freedom (Ref.df ), F statistic,
Estimate se t p
(Intercept) 0.5254 0.0082
gender:woman -0.2642 0.0202 -13.106 < .001
logo:yes -0.0624 0.0234 -2.665 < .01
logo:no picture 0.3316 0.0314 10.571 < .001
edf Ref.df F p
age 1.4097 1.721 33.033 < .001
screen position 5.1438 6.253 9.767 < .001
We further provide here the table showing the regression coeﬃcients of the GAMM ﬁtted to
predict unit-normalised rank considering an interaction between number of candidates and screen
position, visualized in Figure 3 of the main article. See Section 4.1 of the main paper for a detailed
presentation and a visualization of these results.
Table 2: Estimates of the GAMM ﬁtted to predict unit normalised rank as a linear combination of gender,
presence of a party logo in the picture, and age (median centred per district, included a simple non-
linear smooth) and a non-linear interaction of unit normalised screen position and number of candidates
(n candidates, logged). The non-linear interaction was coded as a partial tensor product. The table
provides the estimated degrees of freedom (edf), the reference degrees of freedom (Ref.df), F statistic,
Estimate se t p
(Intercept) 0.5257 0.0082
gender:woman -0.2638 0.0201 -13.122 < .001
logo:yes -0.0592 0.0234 -2.532 < .05
logo:no picture 0.3275 0.0313 10.476 < .001
edf Ref.df F p
age 1.3318 1.598 37.723 < .001
screen position 4.92 6.004 10.056 < .001
screen position*n candidates 3.129 3.701 3.869 < .01
Finally, we provide the coeﬃcients and corresponding statistics for the analysis of the eﬀect of
likeability on rankings in a sample of districts. See section 4.2 for an elaboration on these results.
Table 3: Eﬀect of likeability on rank in the three target districts, extracted from district-speciﬁc models
also including gender (if applicable), age (centred), and screen position. The table provides the adjusted
r2of the full model, estimated degrees of freedom (edf ) and reference degrees of freedom (Ref.df ) of
the likeability eﬀect, with the corresponding Fstatistic and pvalue.
Estimate se t p
(Intercept) 0.4802 0.0120 40.176 < .001
gender:woman -0.2589 0.0224 -11.536 < .001
d11 0.0440 0.0136 3.245 < .01
d21 0.0296 0.0142 2.081 < .05
gender:women*d11 0.0394 0.0258 1.526 0.127
edf Ref.df F p
age*d5 1.000 1.000 63.548 < .001
age*d11 1.986 2.000 180.218 < .001
age*d21 1.217 1.386 73.514 < .001
screen position*d5 1.852 1.978 20.677 < .001
screen position*d11 1.954 1.998 22.926 < .001
screen position*d21 1.947 1.997 122.240 < .001
likeability*d5 1.100 1.189 23.107 < .001
likeability*d11 1.000 1.000 26.843 < .001
likeability*d21 1.277 1.478 8.752 < .001
C Ordinal regression models for rank prediction
Before carrying out any statistical analysis on rank, we ensured that no relation exists between our
control variables, gender and age, on the one side, and screen position on the other, which would
confound the interpretation of the target eﬀect of screen position on rank. A GAMM predicting
gender as a linear combination of simple smooths for age and screen position (unit-normalised per
district) with a random intercept for district identiﬁer, showed no reliable relation between these
variables. The same outcome was observed in a GAMM predicting age as a function of gender and
unit-normalised screen position, again including a random intercept for district identiﬁer.
To check that ballot order eﬀects do not depend on the normalisation of rank and the use
of GAMMs, we also ﬁtted Cumulative Link Mixed Models (CLMMs), using the ordinal package
Christensen (2019) in R to ﬁt CLMMs and a combination of the ggeﬀects (L¨udecke, 2018) ggplot2
(Wickham, 2016) packages to plot ordinal eﬀects. We again started by ﬁtting a baseline statistical
model predicting rank, this time untransformed, as a linear combination of age (centred), gender,
and presence of a party logo in the picture, adding a random intercept over district ID (AIC =
11,002.92). We tested the inclusion of random slopes over the diﬀerent predictors but never observed
an improvement in model ﬁt. All independent variables were signiﬁcant predictors of rank (p < 0.001)
with eﬀects consistent with what we reported in the main analysis. Predicted rank increases with
age, women tended to rank better than men, not having a picture was detrimental while showing a
party logo only marginally helped.
We then added screen position (unit-normalised) as a predictor variable and included a ran-
dom slope for unit-normalised screen position over district ID. Since in the GAMM we observed
a quadratic eﬀect of screen position on rank (unit-normalised), we also included a quadratic ef-
fect of screen position. Screen position was normalised to avoid the mechanistic relation between
screen position and rank, by which only larger districts had higher ranks and screen position val-
ues. By normalising screen position, we have the same range of values for all districts and can
estimate to what extent a candidate’s relative position on screen inﬂuenced the ﬁnal rank. Consis-
tent with the main analysis, screen position had a reliable quadratic eﬀect on rank and a reliable
linear eﬀect. The CLMM, which only included a linear eﬀect of screen position, had an AIC of
10,979.22 (∆AIC = 28.49 compared to the baseline CLMM), while the CLMM, which also included
a quadratic eﬀect of screen position, had an AIC of 10,957.68 (∆AI C = 50.02 compared to the
baseline CLMM), conﬁrming that the quadratic term improves the model ﬁt. Detailed statistics and
output are available in the R workspaces and can be replicated with the provided scripts. All other
predictors remained signiﬁcant. Figure 7 shows the eﬀect of screen position on rank, estimated using
CLMMs. The ﬁgure shows that, for candidates featuring at the top of the screen, it is likelier to rank
at the top; this probability decreases when the normalised screen position increases, with a small
increase for candidates towards the bottom of the page, in line with the quadratic eﬀect observed in
the GAMM. The probability of ranking lower follows the opposite pattern, although the magnitude
is lower - this follows from the fact that while every district has a candidate ranking at position
1, fewer and fewer districts have candidates with ranks over 50, which automatically decreases the
probability of this outcome. This is one of the main reasons why we presented the GAMM in the
We thus replicate the ballot order eﬀect on rank in the 2012 Parlamentarie using a CLMM which
construes the DV as a fully ordinal variable, complementing the analysis we present in the main
paper using GAMMs. Candidates appearing towards the bottom of the page were systematically
disadvantaged and ranked lower on average.
We now move to consider the relationship between the ballot order eﬀect and district size. To this
end, we ﬁtted separate CLMs, one per district (hence we did not have to consider random eﬀects),
predicting the candidate rank as a function of gender (if applicable), age, and screen position.
We did not consider the presence of a party logo in the candidate picture because the model was
unidentiﬁable in some districts due to sparsity. We then extracted the tstatistic of screen position
in each district-speciﬁc CLM and plotted it against the number of candidates in each district. The
result is shown in Figure 8.
0.00 0.25 0.50 0.75 1.00
unit normalised screen position
first mid last
Effect of screen position (unit normalised) on rank
Figure 7: Eﬀect of screen position (unit-normalized) on rank estimated via a Cumulative Link Mixed
Model (CLMM) also including age (centred), gender and presence of a party logo in the candidate
picture, including a random intercept for district ID and random slopes for screen position over
district ID. The plot shows the predicted probability (y-axis) that a candidate featuring in a given
screen position (x-axis) ended up at a given rank (colour legend).
r = 0.591 [ 0.286 , 0.787 ]r = 0.591 [ 0.286 , 0.787 ]r = 0.591 [ 0.286 , 0.787 ]r = 0.591 [ 0.286 , 0.787 ]r = 0.591 [ 0.286 , 0.787 ]r = 0.591 [ 0.286 , 0.787 ]r = 0.591 [ 0.286 , 0.787 ]r = 0.591 [ 0.286 , 0.787 ]r = 0.591 [ 0.286 , 0.787 ]r = 0.591 [ 0.286 , 0.787 ]r = 0.591 [ 0.286 , 0.787 ]r = 0.591 [ 0.286 , 0.787 ]r = 0.591 [ 0.286 , 0.787 ]r = 0.591 [ 0.286 , 0.787 ]r = 0.591 [ 0.286 , 0.787 ]r = 0.591 [ 0.286 , 0.787 ]r = 0.591 [ 0.286 , 0.787 ]r = 0.591 [ 0.286 , 0.787 ]r = 0.591 [ 0.286 , 0.787 ]r = 0.591 [ 0.286 , 0.787 ]r = 0.591 [ 0.286 , 0.787 ]r = 0.591 [ 0.286 , 0.787 ]r = 0.591 [ 0.286 , 0.787 ]r = 0.591 [ 0.286 , 0.787 ]r = 0.591 [ 0.286 , 0.787 ]r = 0.591 [ 0.286 , 0.787 ]r = 0.591 [ 0.286 , 0.787 ]r = 0.591 [ 0.286 , 0.787 ]r = 0.591 [ 0.286 , 0.787 ]
25 50 75 100
n candidates in district
screen position t stat
Screen position t stat on rank ~ n of candidates
Figure 8: Relation between district size (x-axis) and the tstatistic of the eﬀect of screen position on
rank (y-axis) in a series of CLMs ﬁtted for each district, with a linear ﬁt and the linear correlation
coeﬃcient with the corresponding CI.
In small districts, the eﬀect of screen position tends to be weak and does not have a consistent
direction. However, districts feature more than 35 candidates, and the eﬀect of screen position on
rank becomes stronger. The linear ﬁt and the correlation conﬁrm that the eﬀect of screen position
tends to get stronger and penalise candidates appearing at the bottom of the list more when more
These robustness checks conﬁrm a reliable eﬀect of a candidate’s position on screen in each
district and the ﬁnal ranking of the candidate in the district election during the 2012 Parlamentarie,
such that candidates appearing at the top of the page, due to the initials of their surname, had
a higher chance of ranking on top in the election outcome. This eﬀect is, however, moderated by
the number of candidates competing in a district, with candidates at the bottom of the page being
favoured in small districts and further penalised in larger districts. Figure 9 and Figure 10 show plots
of the eﬀect of screen position on rank, estimated using CLMs and GAMs respectively, separately for
each district to provide a more complete picture of the eﬀect under investigation. It is interesting to
note that darker lines in Figure 9, indicating the probability of ranking ﬁrst, have mixed directions
in smaller districts, appearing at the top, while dark blue lines consistently decrease while going to
the right on the x-axis in larger districts. This indicates that the probability of ranking at the top
decreases when the candidate is found towards the bottom of the page. A similar pattern is observed
with GAMs in Figure 10, where we see ﬁtted lines pointing in diﬀerent directions for smaller districts
while consistently pointing upward for larger districts, indicating that the predicted rank increases
when the candidate is found at the bottom of the screen.
D Ballot order eﬀect spillover to parliamentary elections
Finally, we analysed whether the eﬀect of screen position (unit-normalised) on the probability a
candidate had of being elected to parliament correlated with district size. Figure 11 shows a scatter-
plot with district size on the x-axis and the Z statistic of screen position in a GLMER model ﬁtted
separately for each district (controlling for age (global median centering here provided a better ﬁt
than median centering per district) and gender where applicable). A linear ﬁt is overlayed, and an
indication of Pearson’s correlation coeﬃcient with the corresponding 95% CI is provided.
The correlation is not reliable, suggesting that the number of candidates in a district did not mod-
ulate the eﬀect of screen position (unit-normalised) on the probability that a candidate competing
in the 2012 Parlamentarie entered the Italian parliament in 2013.
0 20 40 60 0 20 40 60 80 0 20 40 60 80 0 20 40 60 80 0 25 50 75 100
0 20 40 60 0 20 40 60 0 20 40 60 0 20 40 60 0 20 40 60 0 20 40 60
0 10 20 30 0 10 20 30 0 10 20 30 0 10 20 30 40 0 10 20 30 40 0 20 40
0 10 20 0 10 20 0 10 20 0 10 20 30 0 10 20 30 0 10 20 30
5 10 5 10 15 5 10 15 0 5 10 15 20 0 5 10 15 20 0 5 10 15 20 25
first mid last
Rank probability by screen position in every district
Figure 9: Eﬀect of screen position on rank estimated separately for every district using CLMs, controlling for age (centred) and gender (when
applicable). Districts are ordered from left to right, top to bottom, according to the number of candidates in the district (x-axis). The y-axis shows
the predicted probability that a candidate appearing on a given screen position ended up at a certain rank (colour legend) after the election.
0 25 50 75 100 0 25 50 75 100 0 25 50 75 100 0 25 50 75 100 0 25 50 75 100
0 25 50 75 100
position on screen
Rank by screen position (GAM)
Figure 10: Eﬀect of screen position on rank estimated separately for every district using GAMs, controlling for age (centred) and gender (when
applicable). Districts are ordered from left to right, top to bottom, according to the number of candidates in the district (x-axis). The y-axis shows
the predicted rank.
r = −0.292 [ −0.616 , 0.116 ]r = −0.292 [ −0.616 , 0.116 ]r = −0.292 [ −0.616 , 0.116 ]r = −0.292 [ −0.616 , 0.116 ]r = −0.292 [ −0.616 , 0.116 ]r = −0.292 [ −0.616 , 0.116 ]r = −0.292 [ −0.616 , 0.116 ]r = −0.292 [ −0.616 , 0.116 ]r = −0.292 [ −0.616 , 0.116 ]r = −0.292 [ −0.616 , 0.116 ]r = −0.292 [ −0.616 , 0.116 ]r = −0.292 [ −0.616 , 0.116 ]r = −0.292 [ −0.616 , 0.116 ]r = −0.292 [ −0.616 , 0.116 ]r = −0.292 [ −0.616 , 0.116 ]r = −0.292 [ −0.616 , 0.116 ]r = −0.292 [ −0.616 , 0.116 ]r = −0.292 [ −0.616 , 0.116 ]r = −0.292 [ −0.616 , 0.116 ]r = −0.292 [ −0.616 , 0.116 ]r = −0.292 [ −0.616 , 0.116 ]r = −0.292 [ −0.616 , 0.116 ]r = −0.292 [ −0.616 , 0.116 ]r = −0.292 [ −0.616 , 0.116 ]
25 50 75 100
Screen position Z statistic ~ n candidates
Figure 11: Relation between number of candidates in a district (x-axis) and the Zstatistic of the
eﬀect of screen position on rank (y-axis) in a series of CLMs ﬁtted for each district, with a linear ﬁt
and the linear correlation coeﬃcient with the corresponding CI.
Achen, C., L. Bartels, C. H. Achen, and L. M. Bartels (2017). Democracy for realists. Princeton
Baayen, R. H., S. Vasishth, R. Kliegl, and D. Bates (2017). The cave of shadows: Addressing the
human factor with generalized additive mixed models. Journal of Memory and Language 94,
Ballew, Charles C., I. and A. Todorov (2007). Predicting political elections from rapid and unre-
ﬂective face judgments. Proceedings of the National Academy of Sciences 104(46), 17948–53.
Bartels, L. M. (1996). Uninformed votes: Information eﬀects in presidential elections. American
Journal of Political Science 40, 194–230.
Bates, D., M. M¨achler, B. Bolker, and S. Walker (2015). Fitting linear mixed-eﬀects models using
lme4. Journal of Statistical Software 67 (1), 1–48.
Bickerton, C. J. and C. I. Accetti (2018, 5). ‘techno-populism’ as a new party family: the case of
the ﬁve star movement and podemos. Contemporary Italian Politics 10, 132–150.
Biorcio, R. and R. Sampugnaro (2019). Introduction: The ﬁve-star movement from the street to
local and national institutions. Contemporary Italian Politics 11 (1), 5–14.
Bordignon, F. and L. Ceccarini (2015, 12). Five stars and a cricket. beppe grillo shakes italian
politics. South European Society and Politics 18, 427–449.
Burghardt, K., E. F. Alsina, M. Girvan, W. Rand, and K. Lerman (2017, 3). The myopia of crowds:
Cognitive load and collective evaluation of answers on stack exchange. PLoS ONE 12.
Burghardt, K., T. Hogg, and K. Lerman (2018). Quantifying the impact of cognitive biases in
question-answering systems. In Twelfth International AAAI Conference on Web and Social Media.
Christensen, R. H. B. (2019). ordinal—regression models for ordinal data. R package version
Darcy, R. and M. Mackerras (1993). Rotation of ballots: Minimizing the number of rotations.
Electoral Studies 12 (1), 77–82.
D¨aubler, T. and L. Rudolph (2020). Cue-taking, satisﬁcing, or both? quasi-experimental evidence
for ballot position eﬀects. Political behavior 42 (2), 625–652.
Deseriis, M. (2021). Rethinking the digital democratic aﬀordance and its impact on political repre-
sentation: Toward a new framework. New Media & Society 23, 2452–2473.
Dev, H., K. Karahalios, and H. Sundaram (2019). Quantifying voter biases in online platforms: An
instrumental variable approach. Proceedings of the ACM on Human-Computer Interaction 3.
Estlund, D. and H. Landemore (2018). The epistemic value of democratic deliberation, pp. 113–131.
Oxford University Press New York.
Fiske, S. T. and S. E. Taylor (1991). Social cognition. From brains to culture. Mcgraw-Hill Book
Gerbaudo, P. (2018). The digital party: Political organisation and online democracy. Pluto Press.
Guglielmo, M. (2021). Anti-party digital parties between direct and reactive democracy. the case of
la france insoumise. In Digital Parties, pp. 127–148. Springer.
Ho, D. E. and K. Imai (2006, 9). Randomization inference with natural experiments: An anal-
ysis of ballot eﬀects in the 2003 california recall election. Journal of the American Statistical
Association 101, 888–900.
Kahneman, D. (2011). Thinking, fast and slow. Macmillan.
Krosnick, J. A. (1991). Response strategies for coping with the cognitive demands of attitude
measures in surveys. Applied Cognitive Psychology 5 (3), 213–236.
Kuznetsova, A., P. B. Brockhoﬀ, and R. H. B. Christensen (2017). lmerTest package: Tests in linear
mixed eﬀects models. Journal of Statistical Software 82 (13), 1–26.
Lau, R. and D. P. Redlawsk (2006). How voters decide: Information processing during electoral
campaigns. Cambridge University Press.
Lau, R. R. and D. P. Redlawsk (2001). Advantages and disadvantages of cognitive heuristics in
political decision making. American Journal of Political Science 45, 951.
Lerman, K. and T. Hogg (2014). Leveraging position bias to improve peer recommendation. PloS
one 9 (6), e98914.
Lutz, G. (2010). First come, ﬁrst served: The eﬀect of ballot position on electoral success in open
ballot pr elections. Representation 46, 167–181.
L¨udecke, D. (2018). ggeﬀects: Tidy data frames of marginal eﬀects from regression models. Journal
of Open Source Software 3 (26), 772.
Mair, P. and R. Katz (2002). The ascendancy of the party in public oﬃce: Party organizational
change in twentieth-century democracies. Political parties 24, 113–36.
Malhotra, N. (2008). Completion time and response order eﬀects in web surveys. Public Opinion
Quarterly 72, 914–934.
Marcinkiewicz, K. (2014, 3). Electoral contexts that assist voter coordination: Ballot position eﬀects
in poland. Electoral Studies 33, 322–334.
Meredith, M. and Y. Salant (2013, 3). On the causes and consequences of ballot order eﬀects.
Political Behavior 35, 175–197.
Miller, J. M. and J. A. Krosnick (1998). The impact of candidate name order on election outcomes.
The Public Opinion Quarterly 62, 291–330.
Mudde, C. (2004). The populist zeitgeist. Government and opposition 39 (4), 541–563.
Nairne, J. S. (1988). A framework for interpreting recency eﬀects in immediate serial recall. Memory
& Cognition 16 (4), 343–352.
Roberts, C., E. Gilbert, N. Allum, and L. Eisner (2019, 11). Research Synthesis: Satisﬁcing in
Surveys: A Systematic Review of the Literature. Public Opinion Quarterly 83 (3), 598–626.
Roßmann, J., T. Gummer, and H. Silber (2017, 10). Mitigating Satisﬁcing in Cognitively Demanding
Grid Questions: Evidence from Two Web-Based Experiments. Journal of Survey Statistics and
Methodology 6 (3), 376–400.
Simon, H. A. (1956). Rational choice and the structure of the environment. Psychological Re-
view 63 (2), 129–38.
S¨oderlund, P., ˚
A. von Schoultz, and A. Papageorgiou (2021). Coping with complexity: Ballot
position eﬀects in the ﬁnnish open-list proportional representation system. Electoral Studies 71,
Surowiecki, J. (2005). The wisdom of crowds. Anchor.
Tronconi, F. (2018, 1). The italian ﬁve star movement during the crisis: Towards normalisation?
South European Society and Politics 23, 163–180.
van Erkel, P. F. and P. Thijssen (2016). The ﬁrst one wins: Distilling the primacy eﬀect. Electoral
Studies 44, 245–254.
van Rij, J., M. Wieling, R. H. Baayen, and H. van Rijn (2020). itsadug: Interpreting time series
and autocorrelated data using gamms. R package version 2.4.
Vittori, D. (2017). Podemos and the ﬁve-star movement: Populist, nationalist or what? Contem-
porary Italian Politics 9 (2), 142–161.
Wickham, H. (2016). ggplot2: Elegant Graphics for Data Analysis. Springer-Verlag New York.
Wood, S. N. (2017). Generalized Additive Models.