Figure 1 - available via license: Creative Commons Attribution-NonCommercial 4.0 International
Content may be subject to copyright.
Distribution of estimated prevalence by hypothesized sensitivity bias from list experiments included in Blair et al.'s (2020) meta analysis. Recommendations are based on a preference for a conservative estimate (see Table 1).
Source publication
List experiments are widely used in the social sciences to elicit truthful responses to sensitive questions. Yet, the research design commonly suffers from the problem of measurement error in the form of non-strategic respondent error, where some inattentive participants might provide random responses. This type of error can result in severely bias...
Contexts in source publication
Context 1
... For 37 percent, the placebo control list would have been recommended. The remaining 27 percent of the list experiments would have benefited from our proposed mixed control list design (see Figure 1). In summary, assuming a non-negligible share of inattentive respondents, the vast majority of existing studies using list experiments have used a sub-optimal control list design, among which we are more likely to find type 1 errors. ...
Context 2
... contrast, studies of voter turnout (Holbrook and Krosnick, 2010) and support in authoritarian regimes ( Frye et al., 2017;Robinson and Tannenberg, 2019), where the hypothesized sensitivity bias concerns overreporting, are more often in the predicted prevalence range where the conventional control list is the best choice. It should be noted that several of the latter studies are in a range, ( 0.5 < 1 * τ ≤ ), where it is not possible to obtain conservative estimates (see bottom right of Figure 1). Table 1). ...
Citations
... Additionally, we addressed concerns about inattentive respondents in online surveys (Kees et al., 2017), which can be particularly problematic for list experiments. Inattentive respondents in list experiments can choose the middle number of the list (3 out of 5 list items), which differs between treatment and control groups, generating measurement errors (Ahlquist, 2018;Agerberg and Tannenberg, 2021). To mitigate this issue, we conducted pre-experiment attention checks to exclude inattentive respondents. ...
Discrimination is one of the largest barriers that immigrants and racial/ethnic minorities face in contemporary society. Social scientists have developed and applied field experimental methods to detect the existence and prevalence of discrimination in various domains. In addition, researchers have utilized questionnaires to directly ask discrimination victims about their experiences and the frequency of discrimination they encounter. However, self-reports of discrimination may be biased due to judgment errors in attributing mistreatment to discrimination and intentional overreporting (vigilance) or underreporting (minimization) of discrimination. In this study, we propose a two-stage model that distinguishes between these judgment and reporting biases. We argue that vigilance and minimization stem from sensitivity concerns. We conducted a list experiment with African American respondents who asked about their experiences of employment and everyday discrimination. Comparing the list experiment and direct question estimates, we find no evidence of systematic underreporting or overreporting of employment discrimination. For everyday discrimination, we find overreporting concentrated among ideologically liberal African Americans. These results provide new insights into biases in self-reported discrimination and suggest researchers should be attentive to the conditions under which these biases arise.
... The same analysis was performed after filtering the sample for attentive respondents, that is, those who correctly answered three attention questions (see Agerberg and Tannenberg, 2021, for a study on measurement errors and inattentive respondents). While this reduced the sample by nearly 40 %, there were no statistically significant differences in sociodemographic characteristics between both samples. ...
Scholarly research has consistently shown that teachers present negative assessments of and attitudes toward migrant students. However, previous studies have not clearly addressed the distinction between implicit and explicit prejudices, or identified their underlying sources. This study identifies the explicit and implicit prejudices held by elementary and middle school teachers regarding the learning abilities of an ethnic minority group: Haitian students within the Chilean educational system. We use a list experiment to assess how social desirability and intergroup attitudes toward minority students influence teachers' prejudices. The findings reveal that teachers harbor implicit prejudices towards Haitian students and are truthful in reporting their attitudes, thereby contradicting the desirability bias hypothesis. We suggest that teachers rely on stereotypes associated with the students' nationality when assessing Haitian students’ learning abilities. The implications of these results are discussed in relation to theories grounded in stereotypes and intergroup attitudes.
... Despite the scholarly belief that list experiments address the problem of strategic misreporting, some researchers have recently raised concerns that list experiments are vulnerable to non-strategic misreporting (Agerberg and Tannenberg 2021;Ahlquist 2018;Kuhn and Vivyan 2022;Riambau and Ostwald 2021). To borrow Riambau and Ostwald's (2021) explanation, while "[s]trategic errors arise when respondents lie to conceal their position on the sensitive issue" (172), "[n]on-strategic error includes such things as coding errors and poor quality responses that arise when respondents do not understand or rush through the list experiment" (173). ...
The incumbent-led subversion of democracy represents the most prevalent form of democratic backsliding in recent decades. A central puzzle in this mode of backsliding is why these incumbents enjoy popular support despite their actions against democracy. We address this puzzle using the case of Philippine President Rodrigo Duterte. Although some Philippine analysts have speculated that his popularity was inflated due to social desirability bias (SDB) among survey respondents, there has been limited empirical examination. Our pre-registered list experiment surveys conducted in February/March 2021 detected SBD-induced overreporting at about 39 percentage points in face-to-face surveys and 28 percentage points in online surveys. We also found that the poor Mindanaoans, and those who believed their neighbors supported Duterte, were more likely to respond according to SDB. These possibly counter-intuitive results should be interpreted with caution because the survey was conducted during the height of the COVID-19 lockdown, and the findings cannot necessarily be extrapolated to the other period of his presidency. Nevertheless, this study suggests that preference falsification could be an alternative explanation for the puzzle of popular incumbents in democratic backsliding.
... Finally, inattentive participants can lead to high levels of noise in estimates, which is particularly problematic for list experiments due to the nature of the analysis required to identify social desirability bias. These inattentive participants should be excluded where doing so does not affect the representativeness of the sample (Agerberg and Tannenberg, 2021). ...
Despite the right of disabled people to full social and economic inclusion, many
face multiple day-to-day and systemic challenges. These include but are not limited
to additional expenses, access to housing, and everyday accessibility difficulties.
Surveys show the general public hold positive attitudes towards policies that
seek to enable disabled people to overcome these challenges, but standard
survey methods are susceptible to response biases that may overestimate this
support. This study aimed to test whether two such biases influence support for
disability policy in Ireland: social desirability bias (i.e. the tendency for survey
respondents to alter their responses in order to present themselves in a
positive light); and inattention to the implications of policy support (e.g. that
welfare policies require funding). Together the survey experiments covered a
range of policy issues and types of disability, as identified in previous research
and in consultation with the disability advisory group for the project.
A nationally representative sample of 2,000 adults took part in the online study.
One stage of the study used list experiments to test for social desirability bias in
responses to three issues: (1) support for increased social welfare for disabled
people, (2) support for prioritising disabled people for social housing and (3) how
many people admit to parking in a disabled parking space without a permit. In each
list experiment, participants were assigned at random to one of two groups. One
(‘control’) group was presented with a list of items unrelated to the topic of interest
(in this case, disability policy) and asked how many they agree with. The other
(‘treatment’) group was presented with the same list but with the addition of an
item about the topic of interest. Any difference in the average response between
the groups can be attributed to the added item and gives an indication of support
for that item when participants are provided full anonymity (because they are
never asked directly about their support for that item). Allowing participants to
respond anonymously minimises the influence of the desire to be viewed positively
by others on responses.
Another stage of the study tested the influence of question detail on policy
support. The policies in this stage related to (1) increased cost of living support for
disabled people, (2) support for children with disabilities and (3) support for
building wheelchair accessible infrastructure. Participants were randomly
allocated to a group that was asked for support for a policy without any specified
funding mechanism, or to a group that was asked about support for the same policy
but with the funding mechanism specified, for example that the policy would be
funded through a budget reallocation or a tax increase.
The study shows that while the majority of people in Ireland support most policies
that aim to enable disabled people to participate fully in society, standard surveys
are likely to lead to inaccurate estimates of support. Approximately one-in-seven
people are estimated to express support for some policies when asked directly but
not when allowed to respond anonymously, with a similar change in support when
funding mechanisms or policy trade-offs are made explicit. Support is stronger
among those more familiar with disability issues, although further research is
required to understand why. If those familiar with disability simply better
understand the challenges associated with disability, this implies that enhancing
public understanding of the challenges and costs of disability would strengthen
support. If it is because they know someone who will directly benefit from the
policy, further research on how people understand and recognise disability among
people in their social networks may help. Complementing standard surveys with
reliable experimental methods is recommended to avoid misperceptions of
support for disabled people and to identify where potentially negative attitudes
may need to be challenged.
