Available via license: CC BY 4.0
Content may be subject to copyright.
False Consensus Biases AI Against Vulnerable
Stakeholders
Mengchen Dong1*, Jean-Franc¸ois Bonnefon2and Iyad Rahwan1
1*Center for Humans and Machines, Max Planck Institute for Human
Development, Lentzeallee 94, Berlin, 14195, Germany.
2Toulouse School of Economics, University of Toulouse 1 Capitole, 1 Esp. de
l’Universit´
e, Toulouse, 31000, France.
*Corresponding author(s). E-mail(s): dong@mpib-berlin.mpg.de;
Contributing authors: jean-francois.bonnefon@tse-fr.eu;
rahwan@mpib-berlin.mpg.de;
Abstract
The deployment of AI systems for welfare benefit allocation allows for accelerated decision-
making and faster provision of critical help, but has already led to an increase in unfair benefit
denials and false fraud accusations. Collecting data in the US and the UK (N= 2 449), we
explore the public acceptability of such speed-accuracy trade-offs in populations of claimants
and non-claimants. We observe a general willingness to trade off speed gains for modest
accuracy losses, but this aggregate view masks notable divergences between claimants and
non-claimants. Although welfare claimants comprise a relatively small proportion of the
general population (e.g., 20% in the US representative sample), this vulnerable group is much
less willing to accept AI deployed in welfare systems, raising concerns that solely using
aggregate data for calibration could lead to policies misaligned with stakeholder preferences.
Our study further uncovers asymmetric insights between claimants and non-claimants. The
latter consistently overestimate claimants’ willingness to accept speed-accuracy trade-offs,
even when financially incentivized for accurate perspective-taking. This suggests that policy
decisions influenced by the dominant voice of non-claimants, however well-intentioned, may
neglect the actual preferences of those directly affected by welfare AI systems. Our findings
underline the need for stakeholder engagement and transparent communication in the design
and deployment of these systems, particularly in contexts marked by power imbalances.
1
arXiv:2407.12143v1 [cs.CY] 17 May 2024
Significance Statement. Artificial Intelligence should optimally share the values and goals
of humans. But which humans should AI align with, given its potential for social good? In the
context of social welfare distribution, we analyze public preferences for the trade-offs between
speed gains and accuracy losses when AI systems are deployed. We show that welfare claimants
can be more averse to AI deployed in welfare systems, and less vulnerable non-claimants can
misunderstand their preferences for the trade-offs. These results constitute a strong call for
engaging specifically with vulnerable stakeholders when designing government AI systems,
rather than relying on aggregate data or assuming that other stakeholders with the dominant
voice in society can understand their preferences.
Keywords: artificial intelligence, social welfare, tradeoff, heterogeneity, the alignment problem
1 Introduction
The use of Artificial Intelligence (AI) is becoming commonplace in government operations [1–4].
In the United States alone, a 2020 survey of 142 federal agencies found that 45% were using or
planning to use machine learning algorithms to streamline their operations, increase their capaci-
ties, or improve the delivery of their public services [2]. In the specific context of providing welfare
benefits, the main promise of AI is to speed up decisions [4,5]. For many individuals and families,
welfare benefits provide critical assistance in times of financial hardship or emergency. Using AI
to speed up decisions can avoid delays that would exacerbate these hardships, and decrease the
period of uncertainty and anxiety during which applicants are waiting for a decision. There is,
however, a documented risk that since welfare AI systems often focus on fraud detection, their
speed gains come with a biased accuracy loss, increasing the acceptable trade-offs between the
speed and accuracy of welfare allocations rate at which people are unfairly denied the welfare
benefits they are entitled to [6–11].
As a result, government agencies that seek to deploy welfare AI systems must strike a careful
balance between speed gains and accuracy losses, and this balance must be informed by public
preferences [12,13], for at least two reasons. First, we know that people who lose trust in the AI
used by one government agency also lose trust in the AI used by other government agencies—
if welfare AI systems ignore public preferences when balancing speed and accuracy, they risk
creating distrust that can bleed into perceptions of other government services [14,15]. Second,
and more immediately, the wrong balance of speed gains and accuracy losses could erode the trust
of people who need welfare benefits, and make them less likely to apply, for fear of being wrongly
accused of fraudulent claims [14], especially when the AI system is labeled with foreboding names
like ‘FraudCaster’ [16] or described as a ‘suspicion machine’ in the media [8,17]. In sum, it is
important for welfare AI systems to trade off speed and accuracy in a way that is aligned with the
preferences of the general public as well as with the preferences of potential claimants.
Great efforts have been made to understand people’s attitudes toward and concerns about
welfare AI systems, often focusing on the opinions of the general public [14,18] or vulnerable
populations directly affected by welfare AI systems [11,19]. Qualitative evidence has also been
accumulated regarding the divergent preferences of different stakeholders involved in AI gov-
erning systems [8,20], contributing to long-lasting philosophical and regulatory discussions on
2
You are applying for a social
benefit, which can be about
social housing, unemployment
allowance, child support, etc.
To receive a decision from a
public servant, the average
waiting time is 8 weeks.
8 weeks
In the meantime, the welfare
department has developed an
AI program to automatically
process applications. On
average, the AI program makes
a welfare decision within
07 weeks,
01 weeks faster than a
public servant.
which means that you are
actually eligible for the benefit
but are declined because of
imperfect calculations or
predictions of the AI program.
05%
REJECTED
The AI program also helps
improve the detection of
welfare fraud. However, the AI
program features a
higher chance of
false rejection,
Fig. 1 An example of experimental stimuli, where the AI system is one week faster than humans but leads to a 5% accuracy loss. The
complete list of stimuli consisted of 36 such trade-offs, combining speed gains of 1 to 6 weeks (by the increment of 1) and accuracy
losses from 5% to 30% (by the increment of 5%).
fairness and alignment principles [21–24]. However, less is known about the extent of divergence
in AI design preferences and reconciliation between different perspectives and interests.
Here we present experimental evidence on two critical challenges for aligning AI with human
values in welfare AI systems. First, we identify heterogeneous preferences of welfare claimants
versus non-claimants, with claimants showing a stronger AI aversion irrespective of AI trades off
speed and accuracy. Second, we show that while welfare claimants show insights into the pref-
erences of non-claimants, non-claimants show no insights into the preferences of claimants. In
other words, the perspective of non-claimants is relatively easy to understand, but only claimants
understand their own perspective. These results hold in a representative US sample and in a tar-
geted UK sample balancing the number of claimants and non-claimants. The combination of
heterogeneous preferences and asymmetric insights creates the risk that welfare AI systems be
aligned with the position of the largest, best understood, least vulnerable group – silencing the
voice of the smallest, least understood, most vulnerable group, which nevertheless comprises the
primary stakeholders in the deployment of welfare AI.
2 Results in the US
Participants in the US (N= 987, representative on age, gender, and ethnicity, 20% self-declaring
as welfare claimants) indicated their preference between human and AI welfare decisions. We
varied the information about speed gains (1/2/3/4/5/6 weeks faster, as compared to a baseline
waiting time of 8 weeks if handled by public servants) and accuracy losses (5/10/15/20/25/30%
more false rejections than public servants) within a realistic range, based on governmental reports
and third-party investigations [9,12,25,26], yielding 36 trade-offs (as illustrated in Figure 1).
In each trade-off condition, participants indicated their preference on a scale ranging from 0 =
definitely a public servant to 100 = definitely the AI program. Participants were randomly assigned
3
to respond from their own perspective as claimants or non-claimants, or to adopt the oppo-
site perspective. The US study was not preregistered, hence all analyses should be considered
exploratory.
When participants responded from their own perspective (N= 506), their willingness to
let AI make decisions was influenced both by speed gains (β= 0.19,p<.001) and accuracy
losses (β= 0.40,p < .001). Overall (see Figure 2), they traded off a 1-week speed gain for a 2.4
percentage points loss of accuracy. Among these US participants, 21% self-declared as welfare
claimants. For all the 36 trade-offs, these claimants (vs. non-claimants) showed greater average
aversion to letting AI make welfare decisions (β=−0.19 p < .001). The average difference
between the responses of claimants and non-claimants was 5.9 points (range: 0.3 to 12.8, see
Figure 3A).
Fig. 2 Preferences for speed accuracy trade-offs from own perspective, in the representativeUS sample (N= 506; 21% as welfare
claimants).
Figure 3B displays the biases of claimants and non-claimants when trying to predict the
answers of the other group, across the 36 tradeoffs. Here we calculate the bias for each trade-off
condition by subtracting participants’ actual preference (e.g., claimants taking a claimant per-
spective) from the other groups’ insights through perspective taking (e.g., non-claimants taking
a claimant perspective). We then compare the bias scores with zero to determine their statistical
significance, using the formula below:
biasij =β0+µ0j+ϵij
4
(A) (B)
Perspective taking in the US sample
(A) Across all tradeoffs, non-claimants show greater willingness to let AI make decisions, up to a 13-point difference when
speed gains and accuracy losses are low. (B) When trying to predict the answers of the other group, non-claimants overestimate
willingness of claimants, and claimants underestimate the willingness of non-claimants.
claimants
Average gap in the willingness of claimants and non-claimants Prediction bias, where each dot is one of the 36 tradeoffs
non-claimants
Fig. 3 Perspective taking in the representative US sample (N = 987; 20% as welfare claimants). (A) The average gap between the
willingness of claimants and non-claimants to let AI make welfare decisions across the 36 tradeoffs. (B) Biases of claimants and
non-claimants trying to predict the answers of the other group.
where biasij represents the bias for the ith observation in the jth participant, β0represents the
fixed intercept, µ0jrepresents the random effect for the jth participant, and ϵij represents the
residual error for the ith observation in the jth participant.
Both groups fail to completely take the perspective of the other group. On average, claimants
underestimate the answers of non-claimants by 4.8 points, and non-claimants overestimate the
answers of claimants by 6.4 points. Both biases are significantly different from zero (p=.032
for claimants, and p<.001 for non-claimants): the 95% confidence interval is [-9.2, -0.5] for
claimants, and [4.4, 8.4] for non-claimants. Two issue when comparing the biases between the
two groups, though, are their unequal size in our sample (the standard error for claimants is twice
that for non-claimants), and the lack of financial incentives for responding correctly. These two
issues are addressed in our second study.
In sum, data from our representative US sample shows that US citizens, on average, were
willing to trade a 2.4 accuracy loss for a 1-week speed gain. However, welfare claimants are sys-
tematically more averse to AI than non-claimants, and we find evidence for a small asymmetry
in the insights that claimants and non-claimants have into each other’s answers, with claimants
being more calibrated when predicting the answers of non-claimants.
5
3 Results in the UK
To replicate and extend the results obtained from the US representative sample, we collected data
from N= 1 462 participants in the UK with a balanced composition of claimants and non-
claimants. Such a balanced sample can help consolidate our pre-registered test on the asymmetry
in perspective-taking. In addition, we implemented the following changes:
1. We examined preferences about a specific benefit in the UK (the Universal Credit) and tar-
geted a balanced sample between Universal Credit claimants (48%) and non-claimants (52%).
The UK government has recently announced the deployment of AI for the attribution of this
benefit, raising concerns that the AI system may be biased against some claimants [7].
2. We adopted a different range of speed (0/1/2/3 weeks faster, as compared to a baseline wait-
ing time of 4 weeks if handled by public servants) and accuracy (0/5/10/15/20% more false
rejections than public servants) parameters, resulting in 20 trade-offs. Notably, when wel-
fare AI demonstrates comparable performance (i.e., 0 week faster and 0% more error), people
were still in favor of humans making welfare decisions (M= 45.4,SD = 28.7;t= 4.36,
p<.001).
3. We added financial incentives for participants to correctly predict the preferences of the other
group, that is, when non-claimants predict claimants’ preference and claimants predict non-
claimants’ preference. We also asked non-claimants whether they had claimed welfare benefits
in the past, whether they thought they may claim benefits in the future, and whether they were
acquainted with people who were welfare claimants, to assess whether these circumstances
made it easier to adopt the perspective of claimants.
4. For each trade-off, we additionally asked participants whether their trust in the government
would decrease or increase (from 0 = decrease a lot decrease a lot to 100 = increase a lot) if the
government decided to replace public servants with the AI program they just considered.
5. Finally, we added a treatment that made explicit the existence of a procedure to ask for redress
in case a claimant felt their claim was unfairly rejected. Even though participants in the human
redress condition believed in the chance to appeal (β= 0.37,p<.001; vs. the redress condi-
tion), this clarification did not impact trade-off preferences (β= 0.03,p=.210). Therefore,
we pool the data from this treatment with that of the baseline treatment. Full analyses of this
treatment are presented in the Supplementary Information.
Again, when participants responded from their own perspective (N= 739), their willingness
to let AI make decisions was influenced both by speed gains (β= 0.34,p < .001) and accuracy
losses (β= 0.44,p < .001). Overall (see Figure 4), they traded off a 1-week speed gain for a
5 percentage point loss of accuracy. Among these UK participants, 47% self-declared as current
claimants of the Universal Credit. As in the US study, for all 20 trade-offs, welfare claimants
showed greater average aversion to letting AI make welfare decisions (β=−0.09,p=.008),
with an average difference of 5.7 points (range: 0.1 to 8.7, see Fig. 5A). In both groups, we observe
a strong correlation across trade-offs between the aversion to letting the AI make decisions, and
the loss of trust in the government that would deploy this AI (r = .77 for claimants, and r = .84 for
non-claimants).
Figure 5B displays the biases of claimants and non-claimants when trying to predict the
answers of the other group, across the 20 trade-offs. As in the US study, we calculated the
perspective-taking biases for claimants and non-claimants, respectively. On average, claimants
6
Fig. 4 Preferences for speed accuracy trade-offs from own perspective, in the balanced UK sample (N = 739; 47% as welfare
claimants)
provide an unbiased estimate of the answers of non-claimants (p=.323), with an underestima-
tion of 0.9 points and a 95%-confidence interval including zero, [-2.7, 0.9]. Non-claimants, how-
ever, overestimate the preferences of claimants by 4.2 points (p < .001), with a 95%-confidence
interval of [2.6, 5.7]. These asymmetrical insights between claimants and non-claimants are con-
sistent with our preregistered prediction. To explore whether some life experiences may reduce
bias in the predictions of non-claimants, we recorded whether they had past experience as
claimants of other benefits, whether they were acquainted with current claimants, and their per-
ceived likelihood of becoming claimants in the near future. We found no credible evidence for
any of these effects.
In sum, results from our targeted UK sample consolidate and extend results from our repre-
sentative sample. The average willingness to trade a 5-point accuracy loss for a 1-week speed gain
hides heterogeneity in responses, with welfare claimants being systematically more averse to AI
than non-claimants. We also find strong evidence for asymmetrical insights between claimants
and non-claimants: claimants are well-calibrated when predicting the answers of non-claimants,
but non-claimants overestimate the willingness of claimants to let AI make decisions. Finally,
lower acceptance of the AI system for welfare allocation is strongly linked to decreased trust in
the government among both welfare claimants and non-claimants.
4 Discussion
One primary advantage of using AI for welfare benefit allocation is quicker decision-making,
allowing claimants to receive support faster [4,5]. However, these systems often result in an accu-
racy loss, potentially leading to unfair denials or false fraud accusations [6–11]. Governments
must carefully balance these trade-offs to maintain public trust [14,27]. Indeed, we found that
7
(A) (B)
Perspective taking in the UK sample
(A) Across all tradeoffs, non-claimants show greater willingness to let AI make decisions, up to a 9-point difference. (B) When trying
to predict the answers of the other group, non-claimants overestimate willingness of claimants, whereas claimants are close to
providing unbiased estimates of the willingness of non-claimants.
claimants
Average gap in the willingness of claimants and non-claimants Prediction errors, where each dot is one of the 20 tradeoffs
non-claimants
Fig. 5 Perspective taking in the balanced UK sample (N = 1462; 48% as welfare claimants). (A) The average gap between the
willingness of claimants and non-claimants to let AI make welfare decisions across the 20 tradeoffs. (B) Biases of claimants and
non-claimants trying to predict the answers of the other group.
the acceptability of this balance to participants was closely tied to their resulting trust in the
government.
In both the US and UK, our data suggested that participants would trade a one-week speed
gain for a 2.5 to 5 percentage point accuracy loss. However, we also found that averaging across
participants masked strong divergences between claimants and non-claimants. Though the dif-
ference between the two groups varied across trade-offs, welfare claimants were systematically
less amenable to AI deployment than non-claimants. In parallel with the comparison between
welfare claimants and non-claimants, we also conducted latent profile analysis to explore the
underlying patterns in the preference data without relying on existing labels (e.g., claimant sta-
tus). Since we did not find strong support for a particular number of profiles, we report the results
in the Supplementary Information. In summary, using aggregate data to calibrate AI in welfare
systems could backfire, as average responses may not reflect the divergent preferences of stake-
holders. This finding aligns with recent calls in behavioral science to focus on heterogeneity when
informing policy [28], as well as to consider the positionality of AI models [29], that is, their social
and cultural position with regard to the stakeholders with which they interface.
Data revealed a further complication: asymmetric insight between claimants and non-
claimants. Claimants could provide unbiased estimates of the preferences of non-claimants, but
non-claimants failed to do the same, even in the presence of financial incentives. These find-
ings echo laboratory results suggesting that participants who are or feel more powerful struggle
8
to take the cognitive perspective of others [30–33], as well as sociological theories positing that
marginalized groups have greater opportunities and motivations to develop an understanding of
the thoughts and norms of dominant groups [34–36].
In the context of welfare AI, asymmetric insights create the risk that the perspective of
claimants may be silenced even when non-claimants seek to defend the interests of claimants.
These well-intentioned non-claimants may use their dominant voice to shape public opinion and
policy without realizing that they do not in fact understand the preferences of claimants, result-
ing in AI systems that are misaligned with the preferences of their primary stakeholders. Our
results thus underline the need to actively engage with claimants when building welfare AI sys-
tems, rather than to assume that their preferences are well-understood or can be understood
through empathetic perspective-taking.
Our results also highlight the need for transparent communication about welfare AI sys-
tems’ design choices. Given the significant heterogeneity in public preferences, and the close
link between meeting these preferences and trust in government, clear justifications are crucial.
Despite increasing technical attempts to align AI with pluralistic values and diverse perspec-
tives [37–39], there are inevitably situations where agreement or reconciliation cannot be easily
achieved (e.g., when non-claimants fail to estimate welfare claimants’ aversion to AI, but not vice
versa). In this case, our study sheds light on the possibility of evidence-based public communi-
cation when human-centered AI designs need to de-emphasize the preferences of the general
population but optimize toward a particular subgroup of people. Our core findings, heterogene-
ity and asymmetric insights, may also hold in other cases where AI is deployed in a context of
power imbalance – conducting behavioral research on these cases in advance of AI deployment
may help avoid the scandals that marred the deployment of welfare AI.
5 Methods
Both of the US and UK studies were approved by the ethics committee at the Max Planck Insti-
tute for Human Development, and obtained informed consent from all participants. Data were
collected in February 2022 and September 2022, respectively. All participants were recruited on
Prolific for a study named “Artificial Intelligence in Social Welfare”, and were paid £1.6 upon com-
pletion. Participants in the UK study who had to predict the answers of the other group received
an additional £0.03 for each response that fell within 5 points of this other group’s average.
Both studies were hosted on Qualtrics. After providing informed consent and basic demo-
graphic information (age, gender, education, income, and political ideology), participants were
instructed to take a claimant or non-claimant perspective. To familiarize themselves with the
stimuli and response scale, they were first shown two extreme trade-offs in the survey, as train-
ing examples. They answered these two examples, and had a chance to review and change their
answers. Then the survey started, and all targeted trade-offs were shown in random order, not
including the two trade-offs that were shown as examples during the training phase. Complete
descriptions of our design materials, and survey questions are included in the Supplementary
Information.
5.1 The US study
Participants. We had N= 987 participants from the United States, who were representative
on age (M= 45.3,SD = 16.3), gender (473 males and 514 females), and ethnicity (77.8%
9
White, 11.4% Black, 6.1% Asian, 2.5% Mixed, and 2.1% other), and 20.4% of them self-reported as
welfare claimants at the time of the study. The sample size was determined based on the recent
recommendation of around 500 people for latent profile analysis [40]. We aimed for an almost
doubled sample size given our two-condition perspective-taking manipulation.
Design and procedure. The US study employed a mixed design, with one between-subjects
and two within-subjects factors. First, participants were asked for their basic demographic infor-
mation, and were randomly assigned to take either a claimant (“You are applying for a social
benefit”) or a controlled taxpayer (“Someone else in your city is applying for a social bene-
fit”) perspective. We then manipulated the information about welfare AI’s speed (6 conditions:
1/2/3/4/5/6 weeks faster than a public servant) and accuracy (6 conditions: 5/10/15/20/25/30%
more false rejections than a public servant). The presented speed (an average of 8 weeks) and
accuracy (at most 40% errors) baselines referred to realistic information from some governmental
reports and third-party investigations [9,12,25,26].
After knowing the perspective they should take, participants went through two training
examples, reading two extreme cases of welfare AI (bad case: 0 week faster + 50% more false rejec-
tions; good case: 7 weeks faster + 1% more false rejections) and answering the same question “To
what extent do you prefer a public servant or the AI program to handle your/the person’s wel-
fare application?” on a 100-point scale (from 0 = definitely a public servant to 100 = definitely the
AI program). They then had a chance to review and calibrate their answers in the two cases before
moving to the 36 official test rounds – which did not allow revisions anymore. In each of the
36 test rounds, they read information about their perspective, AI speed, and AI accuracy in three
consecutive cards (see Fig. S1 in the Supplementary Information for an illustration of the cards
in different experiment conditions). After reading the three cards in each round, participants
answered the same question about their preference for welfare AI versus public servants.
5.2 The UK study
Participants. We performed a simulation-based power analysis for multilevel regression mod-
els, which suggested that a sample of N= 800 would allow us to detect the interaction effect of
AI performance, claimant status, and perspective-taking with higher than 80% power at an alpha
level of 0.05 (see the pre-registration at https://tinyurl.com/welfareAIregistration). We therefore
aimed for N= 1600 participants in the United Kingdom given our additional between-subjects
human redress manipulation. As pre-registered, we filtered out participants who provided differ-
ent answers to one identical welfare status question (“Are you a recipient of Universal Credit?”;
Answer: “Yes/No”), which was embedded both in the Prolific system screener and our own sur-
vey. After the screening, we eventually had N= 1 462 participants (age: M= 37.6,SD = 11.1;
ethnicity: 88.4% White, 3.0% Black, 5.6% Asian, 2.7% Mixed, and 0.3% other), with relatively bal-
anced compositions of males and females (42.7% male, 55.9% female, 1.4% other), and welfare
claimants (47.9%) versus non-claimants (52.1%).
Design and procedure. Study 2 examined a real-life social benefits scheme in the UK – Uni-
versal Credit (https://www.gov.uk/universal-credit). We employed a mixed design with three
between-subjects and two within-subjects factors. As between-subjects factors, we recruited both
Universal Credit claimants and non-claimants, and randomly assigned them to take a Univer-
sal Credit claimant or a controlled taxpayer perspective. They were then randomly assigned to a
no redress or a human redress condition, which differed on whether claimants could appeal to
10
public servants. As within-subject factors, we manipulated information about welfare AI’s speed
(0/1/2/3 weeks faster, as compared to a baseline waiting time of 4 weeks if handled by public
servants), and accuracy (0/5/10/15/20% more false rejection).
Before starting the 20 rounds of official tradeoff evaluations, as in the US study, participants
went through two training examples with a chance of revision, reading two extreme cases of
welfare AI (bad case: 0 week faster + 40% more false rejections; good case: 3 weeks faster + 1%
more false rejections). In each example, they answered two questions on a 100-point scale: “To
what extent do you prefer a public servant or the AI program to handle your/the person’s wel-
fare application?” (0 = definitely a public servant to 100 = definitely the AI program) and “If the UK
government decided to replace some public servants with the AI program in handling welfare
applications, would your trust in the government decrease or increase?” (0 = decrease a lot to 100 =
increase a lot). They then had a chance to review and change their answers in the two cases before
moving to the 20 official test rounds – which did not allow revisions anymore. In each of the
20 test rounds, they read information about their perspective, AI speed, AI accuracy, and human
redress condition in three consecutive cards (see Fig. S2 in the Supplementary Information for
an illustration of the cards in different experiment conditions). After reading the three cards in
each round, participants answered the same two question about their preference for welfare AI
versus public servants, and their trust in the government.
To increase the motivation of perspective taking, participants were informed and incen-
tivized to take the opposite perspective, for each accurate answer that fell within ±5 points of the
other group’s average. At the end of the 20 official rounds, as a manipulation check, participants
indicated the extent to which they believed that “you/the person can appeal to public servants if
you/they are not satisfied with the welfare decision made by the AI program?” (0 = not at all to
100 = very much).
5.3 Data availability
All anonymized data can be found on Open Science Framework (at https://tinyurl.com/
welfareAI; and will be made publicly accessible upon acceptance of the work).
5.4 Code availability
All code necessary to reproduce all analyses can be found on Open Science Framework (at https:
//tinyurl.com/welfareAI; and will be made publicly accessible upon acceptance of the work).
End notes
•Acknowledgements: Agence Nationale De La Recherche ANR-19-PI3A-0004 ( JFB); Agence
Nationale De La Recherche ANR-17-EURE-0010 (JFB); The research foundation TSE-
Partnership (JFB)
•Authors’ contributions: Conceptualization: MD, JFB, IR; Methodology: MD, JFB; Investiga-
tion: MD; Visualization: JF; Funding acquisition: IR; Project administration: MD; Supervision:
IR; Writing – original draft: MD, JFB; Writing – review & editing: MD, JFB, IR
•Materials & Correspondence: Correspondence and requests for materials should be addressed
to M.D.
•Additional information: Supplementary Information is available for this paper.
11
References
[1] Misuraca, G. & Van Noordt, C. Ai watch-artificial intelligence in public services: Overview
of the use and impact of ai in public services in the eu. JRC Research Reports (2020).
[2] Engstrom, D. F., Ho, D. E., Sharkey, C. M. & Cu´
ellar, M. F. Government by algorithm:
Artificial intelligence in federal administrative agencies. Tech. Rep., NYU School of Law
(2020).
[3] Coglianese, C. & Dor, L. M. B. Ai in adjudication and administration. Brook. L. Rev. 86, 791
(2020).
[4] de Sousa, W. G., de Melo, E. R. P., Bermejo, P. H. D. S., Farias, R. A. S. & Gomes, A. O. How
and where is artificial intelligence in the public sector going? a literature review and research
agenda. Government Information Quarterly 36, 101392 (2019).
[5] Bansak, K. et al. Improving refugee integration through data-driven algorithmic assignment.
Science 359, 325–329 (2018).
[6] Alston, P. Report of the special rapporteur on extreme poverty and human rights. Tech. Rep.
(2015).
[7] Booth, R. Benefits system automation could plunge claimants deeper into poverty. The
Guardian (2019).
[8] Constantaras, E., Geiger, G., Braun, J., Mehrotra, D. & Aung, H. Inside the suspicion machine.
Wired (2023).
[9] Muralidharan, K., Niehaus, P. & Sukhtankar, S. Identity verification standards in welfare
programs: Experimental evidence from india. Tech. Rep., National Bureau of Economic
Research (2020).
[10] Ryan-Mosley, T. An algorithm intended to reduce poverty might disqualify people in need.
MIT Technology Review (2023).
[11] Carney, T. Artificial intelligence in welfare: Striking the vulnerability balance? Monash
University Law Review 46, 23–51 (2020).
[12] Noriega, A., Garcia-Bulle, B., Tejerina, L. & Pentland, A. Algorithmic fairness and efficiency in
targeting social welfare programs at scale (2018).
[13] Bonnefon, J.-F., Shariff, A. & Rahwan, I. The moral psychology of ai and the ethical opt-out
problem. Ethics of artificial intelligence 109–126 (2020).
[14] Longoni, C., Cian, L. & Kyung, E. J. Algorithmic transference: People overgeneralize failures
of ai in the government. Journal of Marketing Research 60, 170–188 (2023).
12
[15] Dietvorst, B. J., Simmons, J. P. & Massey, C. Algorithm aversion: people erroneously avoid
algorithms after seeing them err. Journal of Experimental Psychology: General 144, 114 (2015).
[16] Solutions, P. Fraudcaster master design document. Tech.
Rep. EPIC-21-06-25-DC-DHS-FOIA-20220204, Pondera Solu-
tions (2022). URL https://epic.org/wp-content/uploads/2022/06/
EPIC-21-06-25-DC- DHS-FOIA-20220204-Pondera-FraudCaster-Master-Design-Document.
pdf.
[17] Johnson, K. Algorithms quietly run the city of dc, and maybe your hometown.
Ars Technica (2022). URL https://arstechnica.com/information-technology/2022/11/
algorithms-quietly-run-the-city-of-dc-and-maybe-your-hometown/.
[18] Jonsson, O. et al. European tech insights. IE Center for the Governance of Change, Madrid
(2021).
[19] Brown, A., Chouldechova, A., Putnam-Hornstein, E., Tobin, A. & Vaithianathan, R. Toward
algorithmic accountability in public services: A qualitative study of affected community perspectives
on algorithmic decision-making in child welfare services, 1–12 (2019).
[20] Lee, M. K., Kim, J. T. & Lizarondo, L. A human-centered approach to algorithmic services: Con-
siderations for fair and motivating smart community service management that allocates donations
to non-profit organizations, 3365–3376 (2017).
[21] Arnstein, S. R. A ladder of citizen participation. Journal of the American Institute of planners
35, 216–224 (1969).
[22] Birhane, A. et al. Power to the people? opportunities and challenges for participatory ai, 1–8 (2022).
[23] Reisman, D., Schultz, J., Crawford, K. & Whittaker, M. Algorithmic impact assessments: A
practical framework for public agency. AI Now 9(2018).
[24] Huang, K., Greene, J. D. & Bazerman, M. Veil-of-ignorance reasoning favors the greater
good. Proceedings of the national academy of sciences 116, 23989–23995 (2019).
[25] PLC, S. B. Annual report and accounts 2020. Disponibile su (2021).
[26] Benedikt, L. et al. Human-in-the-loop ai in government: a case study, 488–497 (2020).
[27] Fiske, S. T. & Dupree, C. Gaining trust as well as respect in communicating to motivated
audiences about science topics. Proceedings of the National Academy of Sciences 111, 13593–
13597 (2014).
[28] Bryan, C. J., Tipton, E. & Yeager, D. S. Behavioural science is unlikely to change the world
without a heterogeneity revolution. Nature Human Behaviour 5, 980–989 (2021).
[29] Cambo, S. A. & Gergle, D. Model positionality and computational reflexivity: Promoting
reflexivity in data science, 1–19 (2022).
13
[30] Blader, S. L., Shirako, A. & Chen, Y.-R. Looking out from the top: Differential effects of
status and power on perspective taking. Personality and Social Psychology Bulletin 42, 723–737
(2016).
[31] Galinsky, A. D., Magee, J. C., Inesi, M. E. & Gruenfeld, D. H. Power and perspectives not
taken. Psychological Science 17, 1068–1074 (2006).
[32] Guinote, A. How power affects people: Activating, wanting, and goal seeking. Annual Review
of Psychology 68, 353–381 (2017).
[33] Brown-Iannuzzi, J. L., Dotsch, R., Cooley, E. & Payne, B. K. The relationship between mental
representations of welfare recipients and attitudes toward welfare. Psychological Science 28,
92–103 (2017).
[34] Bourdieu, P. Distinction: A Social Critique of the Judgement of Taste (Harvard University Press,
1984).
[35] Du Bois, W. E. B. The souls of black folk (Oxford University Press, 2008).
[36] Haraway, D. Situated knowledges: The science question in feminism and the privilege of
partial perspective. Feminist Studies 14, 575–599 (1988).
[37] Sorensen, T. et al. A roadmap to pluralistic alignment. arXiv preprint arXiv:2402.05070 (2024).
[38] Bakker, M. et al. Fine-tuning language models to find agreement among humans with
diverse preferences. Advances in Neural Information Processing Systems 35, 38176–38189
(2022).
[39] Conitzer, V. et al. Social choice for ai alignment: Dealing with diverse human feedback. arXiv
preprint arXiv:2404.10271 (2024).
[40] Spurk, D., Hirschi, A., Wang, M., Valero, D. & Kauffeld, S. Latent profile analysis: A
review and “how to” guide of its application within vocational behavior research. Journal of
Vocational Behavior 120, 103445 (2020).
14
