No full-text available
To read the full-text of this research,
you can request a copy directly from the authors.
Blinding to Circumvent Human Biases: Deliberate Ignorance in Humans, Institutions, and Machines
Abstract
Inequalities and injustices are thorny issues in liberal societies, manifesting in forms such as the gender-pay gap; sentencing discrepancies among Black, Hispanic, and White defendants; and unequal medical-resource distribution across ethnicities. One cause of these inequalities is implicit social bias-unconsciously formed associations between social groups and attributions such as "nurturing," "lazy," or "uneducated." One strategy to counteract implicit and explicit human biases is delegating crucial decisions, such as how to allocate benefits, resources, or opportunities, to algorithms. Algorithms, however, are not necessarily impartial and objective. Although they can detect and mitigate human biases, they can also perpetuate and even amplify existing inequalities and injustices. We explore how a philosophical thought experiment, Rawls's "veil of ignorance," and a psychological phenomenon, deliberate ignorance, can help shield individuals, institutions, and algorithms from biases. We discuss the benefits and drawbacks of methods for shielding human and artificial decision makers from potentially biasing information. We then broaden our discussion beyond the issues of bias and fairness and turn to a research agenda aimed at improving human judgment accuracy with the assistance of algorithms that conceal information that has the potential to undermine performance. Finally, we propose interdisciplinary research questions.
... A key critical AI competence is therefore to assess whether a model is biased, for example, against marginalized social groups (e.g., racialized groups; Obermeyer et al., 2019). In the simplest case, this means being able to ascertain whether a model uses objectionable features (e.g., protected attributes including ethnicity or gender) or proxies thereof (Hertwig et al., 2023;Yeom et al., 2018). ...
Artificial intelligence (AI) is becoming integral to many areas of life, yet many—if not most—AI systems are opaque black boxes. This lack of transparency is a major source of concern, especially in high-stakes settings (e.g., medicine or criminal justice). The field of explainable AI (XAI) addresses this issue by explaining the decisions of opaque AI systems. However, such post hoc explanations are troubling because they cannot be faithful to what the original model computes—otherwise, there would be no need to use that black box model. A promising alternative is simple, inherently interpretable models (e.g., simple decision trees), which can match the performance of opaque AI systems. Because interpretable models represent—by design—faithful explanations of themselves, they empower informed decisions about whether to trust them. We connect research on XAI and inherently interpretable AI with that on behavioral science and boosts for competences. This perspective suggests that both interpretable AI and XAI could boost people’s competences to critically evaluate AI systems and their ability to make accurate judgments (e.g., medical diagnoses) in the absence of any AI support. Furthermore, we propose how to empirically assess whether and how AI support fosters such competences. Our theoretical analysis suggests that interpretable AI models are particularly promising and—because of XAI’s drawbacks—preferable. Finally, we argue that explaining large language models (LLMs) faces similar challenges as XAI for supervised machine learning and that the gist of our conjectures also holds for LLMs.
... One major source of worry is algorithmic bias, which can result in biased consequences. Biases in recruiting algorithms, for example, may perpetuate existing inequities, raising concerns about fairness and equal opportunity in the workplace [23,51]. Furthermore, the widespread use of AI raises the possibility of political manipulation. ...
This survey paper explores the transformative role of Artificial Intelligence (AI) in information security. Traditional methods, especially rule-based approaches, faced significant challenges in protecting sensitive data from ever-changing cyber threats, particularly with the rapid increase in data volume. This study thoroughly evaluates AI’s application in information security, discussing its strengths and weaknesses. It provides a detailed review of AI’s impact on information security, examining various AI algorithms used in this field, such as supervised, unsupervised, and reinforcement learning, and highlighting their respective strengths and limitations. The study identifies key areas for future AI research in information security, focusing on improving algorithms, strengthening information security, addressing ethical issues, and exploring safety and security-related concerns. It emphasizes significant security risks, including vulnerability to adversarial attacks, and aims to enhance the robustness and reliability of AI systems in protecting sensitive information by proposing solutions for potential threats. The findings aim to benefit cybersecurity professionals and researchers by offering insights into the intricate relationship between AI, information security, and emerging technologies.
Willful blindness, a legal doctrine in which individuals deliberately avoid knowledge to escape accountability, plays a significant role in the adjudication of crimes such as corporate fraud, financial misconduct, and sexual assault. However, current research has not fully examined its strategic misuse or provided a comprehensive analysis of its application across different legal contexts. This study bridges these gaps by introducing a Multi-Domain Analytical Framework to evaluate the concept's impact, strategic use, and broader implications in both criminal and civil cases. Empirical findings reveal that willful blindness is often employed to evade legal responsibility, undermining justice and compromising the fairness of legal outcomes. The study further demonstrates the inadequacy of existing legal standards and evidentiary requirements in addressing this issue. To counter these challenges, the research proposes targeted policy reforms designed to strengthen accountability, enhance evidentiary clarity, and improve the overall fairness and effectiveness of the legal system. In conclusion, willful blindness remains a critical and manipulative tool with profound implications for justice and accountability. This study highlights the necessity of reforming legal frameworks to mitigate the strategic exploitation of willful blindness. By setting the stage for further research and policy development, the findings contribute to the ongoing evolution of a more just and equitable legal system capable of addressing the complexities of modern legal challenges.
Digitally-enabled means for judgment aggregation have renewed interest in "wisdom of the crowd" effects and kick-started collective intelligence design as an emerging field in the cognitive and computational sciences. A keenly debated question here is whether social influence helps or hinders collective accuracy on estimation tasks, with recently introduced network theories offering a reconciliation of seemingly contradictory past results. Yet, despite a growing body of literature linking social network structure and the accuracy of collective beliefs, strategies for exploiting network structure to harness crowd wisdom are under-explored. In this paper, we introduce a potential new tool for collective intelligence design informed by such network theories: rewiring algorithms. We provide a proof of concept through agent-based modelling and simulation , showing that rewiring algorithms that dynamically manipulate the structure of communicating social networks can increase the accuracy of collective estimations in the absence of knowledge of the ground truth.
The opening of East Germany's Stasi archives in 1991 has often been lauded as a model of transparency in a transformative period. Yet many citizens have rejected the opportunity to read their files. To examine the reasons people invoke for this deliberate ignorance, we combined survey methods from psychology with historiographical methodologies. Our findings reveal a diverse range of reasons for deliberate ignorance, including regulation of negative emotions, avoidance of personal conflict, scepticism about the information compiled, and rejection of the victorious political system's authority over the files. Participants thus appear to prioritise cooperation and harmony over justice concerns—in stark contrast to the institutional norm of transparency and justice. Shining a light on the role of deliberate ignorance at the individual level—and specifically the convergence or divergence of individual and collective memory culture—may help explain the pace of societal change.
Accumulated findings from studies in which implicit-bias measures correlate with discriminatory judgment and behavior have led many social scientists to conclude that implicit biases play a causal role in racial and other discrimination. In turn, that belief has promoted and sustained two lines of work to develop remedies: (a) individual treatment interventions expected to weaken or eradicate implicit biases and (b) group-administered training programs to overcome biases generally, including implicit biases. Our review of research on these two types of sought remedies finds that they lack established methods that durably diminish implicit biases and have not reproducibly reduced discriminatory consequences of implicit (or other) biases. That disappointing conclusion prompted our turn to strategies based on methods that have been successful in the domain of public health. Preventive measures are designed to disable the path from implicit biases to discriminatory outcomes. Disparity-finding methods aim to discover disparities that sometimes have obvious fixes, or that at least suggest where responsibility should reside for developing a fix. Disparity-finding methods have the advantage of being useful in remediation not only for implicit biases but also systemic biases. For both of these categories of bias, causes of discriminatory outcomes are understood as residing in large part outside the conscious awareness of individual actors. We conclude with recommendations to guide organizations that wish to deal with biases for which they have not yet found solutions.
We highlight several sets of findings from the past decade elucidating the relationship between implicit social cognition and real-world inequality: Studies focusing on practical ramifications of implicit social cognition in applied contexts, the relationship between implicit social cognition and consequential real-world outcomes at the level of individuals and geographic units, and convergence between individual-level and corpus-based measures of implicit bias.
Improving the accuracy of medical image interpretation can improve the diagnosis of numerous diseases. We compared different approaches to aggregating repeated decisions about medical images to improve the accuracy of a single decision maker. We tested our algorithms on data from both novices (undergraduates) and experts (medical professionals). Participants viewed images of white blood cells and made decisions about whether the cells were cancerous or not. Each image was shown twice to the participants and their corresponding confidence judgments were collected. The maximum confidence slating (MCS) algorithm leverages metacognitive abilities to consider the more confident response in the pair of responses as the more accurate “final response” (Koriat, 2012), and it has previously been shown to improve accuracy on our task for both novices and experts (Hasan et al., 2021). We compared MCS to similarity‐based aggregation (SBA) algorithms where the responses made by the same participant on similar images are pooled together to generate the “final response.” We determined similarity by using two different neural networks where one of the networks had been trained on white blood cells and the other had not. We show that SBA improves performance for novices even when the neural network had no specific training on white blood cell images. Using an informative representation (i.e., network trained on white blood cells) allowed one to aggregate over more neighbors and further boosted the performance of novices. However, SBA failed to improve the performance for experts even with the informative representation. This difference in efficacy of the SBA suggests different decision mechanisms for novices and experts.
The EU General Data Protection Regulation (GDPR) and the California Privacy Rights Act (CPRA) mandate the principle of data minimization, which requires that only data necessary to fulfill a certain purpose be collected. However, it can often be difficult to determine the minimal amount of data required, especially in complex machine learning models such as deep neural networks. We present a first-of-a-kind method to reduce the amount of personal data needed to perform predictions with a machine learning model, by removing or generalizing some of the input features of the runtime data. Our method makes use of the knowledge encoded within the model to produce a generalization that has little to no impact on its accuracy, based on knowledge distillation approaches. We show that, in some cases, less data may be collected while preserving the exact same level of model accuracy as before, and if a small deviation in accuracy is allowed, even more generalizations of the input features may be performed. We also demonstrate that when collecting the features dynamically, the generalizations may be even further improved. This method enables organizations to truly minimize the amount of data collected, thus fulfilling the data minimization requirement set out in the regulations.
Online platforms’ data give advertisers the ability to “microtarget” recipients’ personal vulnerabilities by tailoring different messages for the same thing, such as a product or political candidate. One possible response is to raise awareness for and resilience against such manipulative strategies through psychological inoculation. Two online experiments (total N=828) demonstrated that a short, simple intervention prompting participants to reflect on an attribute of their own personality—by completing a short personality questionnaire—boosted their ability to accurately identify ads that were targeted at them by up to 26 percentage points. Accuracy increased even without personalized feedback, but merely providing a description of the targeted personality dimension did not improve accuracy. We argue that such a “boosting approach,” which here aims to improve people’s competence to detect manipulative strategies themselves, should be part of a policy mix aiming to increase platforms’ transparency and user autonomy.
There is growing concern that decision-making informed by machine learning (ML) algorithms may unfairly discriminate based on personal demographic attributes, such as race and gender. Scholars have responded by introducing numerous mathematical definitions of fairness to test the algorithm, many of which are in conflict with one another. However, these reductionist representations of fairness often bear little resemblance to real-life fairness considerations, which in practice are highly contextual. Moreover, fairness metrics tend to be implemented within narrow and targeted fairness toolkits for algorithm assessments that are difficult to integrate into an algorithm’s broader ethical assessment. In this paper, we derive lessons from ethical philosophy and welfare economics as they relate to the contextual factors relevant for fairness. In particular we highlight the debate around the acceptability of particular inequalities and the inextricable links between fairness, welfare and autonomy. We propose Key Ethics Indicators (KEIs) as a way towards providing a more holistic understanding of whether or not an algorithm is aligned to the decision-maker’s ethical values.
People rely on data-driven AI technologies nearly every time they go online, whether they are shopping, scrolling through news feeds, or looking for entertainment. Yet despite their ubiquity, personalization algorithms and the associated large-scale collection of personal data have largely escaped public scrutiny. Policy makers who wish to introduce regulations that respect people’s attitudes towards privacy and algorithmic personalization on the Internet would greatly benefit from knowing how people perceive personalization and personal data collection. To contribute to an empirical foundation for this knowledge, we surveyed public attitudes towards key aspects of algorithmic personalization and people’s data privacy concerns and behavior using representative online samples in Germany ( N = 1065), Great Britain ( N = 1092), and the United States ( N = 1059). Our findings show that people object to the collection and use of sensitive personal information and to the personalization of political campaigning and, in Germany and Great Britain, to the personalization of news sources. Encouragingly, attitudes are independent of political preferences: People across the political spectrum share the same concerns about their data privacy and show similar levels of acceptance regarding personalized digital services and the use of private data for personalization. We also found an acceptability gap: People are more accepting of personalized services than of the collection of personal data and information required for these services. A large majority of respondents rated, on average, personalized services as more acceptable than the collection of personal information or data. The acceptability gap can be observed at both the aggregate and the individual level. Across countries, between 64% and 75% of respondents showed an acceptability gap. Our findings suggest a need for transparent algorithmic personalization that minimizes use of personal data, respects people’s preferences on personalization, is easy to adjust, and does not extend to political advertising.
Individuals and institutions in societies in transition face diffi cult questions: whether or not to seek, explore, and produce public knowledge about their harrowing past. Not disclosing painful truths can be a conduit to reconciliation, as in premodern memory
politics, but it can also mask the past regime’s perpetrators, benefactors, and its victims, highlighted in modern memory politics. Using the transformations of twentieth-century Germany as a case study, this chapter argues that deliberate ignorance has always been
an element of memory politics, even in the twentieth-century approach to Vergangenheitsbewältigung (coming to terms with the past), with its emphasis on knowledge, remembrance, and disclosure. Profoundly dialectic in nature, deliberate ignorance can modulate the pace of change in periods of transition and preserve social cohesion, while
simultaneously undermining personal trust and institutional confi dence. Turning to individuals’ decisions to read or not read the fi les compiled on them by the East German’s Ministry for State Security, it is argued that offi cial memory politics and individuals’ knowledge preferences need not concur. In the public records and in initial results of an empirical analysis of individuals’ choice not to read their fi les, highly diverse and distinct reasons for deliberate ignorance have been observed.
The Internet has evolved into a ubiquitous and indispensable digital environment in which people communicate, seek information, and make decisions. Despite offering various benefits, online environments are also replete with smart, highly adaptive choice architectures designed primarily to maximize commercial interests, capture and sustain users’ attention, monetize user data, and predict and influence future behavior. This online landscape holds multiple negative consequences for society, such as a decline in human autonomy, rising incivility in online conversation, the facilitation of political extremism, and the spread of disinformation. Benevolent choice architects working with regulators may curb the worst excesses of manipulative choice architectures, yet the strategic advantages, resources, and data remain with commercial players. One way to address some of this imbalance is with interventions that empower Internet users to gain some control over their digital environments, in part by boosting their information literacy and their cognitive resistance to manipulation. Our goal is to present a conceptual map of interventions that are based on insights from psychological science. We begin by systematically outlining how online and offline environments differ despite being increasingly inextricable. We then identify four major types of challenges that users encounter in online environments: persuasive and manipulative choice architectures, AI-assisted information architectures, false and misleading information, and distracting environments. Next, we turn to how psychological science can inform interventions to counteract these challenges of the digital world. After distinguishing among three types of behavioral and cognitive interventions—nudges, technocognition, and boosts—we focus on boosts, of which we identify two main groups: (a) those aimed at enhancing people’s agency in their digital environments (e.g., self-nudging, deliberate ignorance) and (b) those aimed at boosting competencies of reasoning and resilience to manipulation (e.g., simple decision aids, inoculation). These cognitive tools are designed to foster the civility of online discourse and protect reason and human autonomy against manipulative choice architectures, attention-grabbing techniques, and the spread of false information.
Psychologists, economists, historians, computer scientists, sociologists, philosophers, and legal scholars explore the conscious choice not to seek information.
The history of intellectual thought abounds with claims that knowledge is valued and sought, yet individuals and groups often choose not to know. We call the conscious choice not to seek or use knowledge (or information) deliberate ignorance. When is this a virtue, when is it a vice, and what can be learned from formally modeling the underlying motives? On which normative grounds can it be judged? Which institutional interventions can promote or prevent it? In this book, psychologists, economists, historians, computer scientists, sociologists, philosophers, and legal scholars explore the scope of deliberate ignorance.
Drawing from multiple examples, including the right not to know in genetic testing, collective amnesia in transformational societies, blind orchestral auditions, and “don't ask don't tell” policies), the contributors offer novel insights and outline avenues for future research into this elusive yet fascinating aspect of human nature.
Contributors
Sarah Auster, Benjamin E. Berkman, Felix Bierbrauer, Gordon D. A. Brown, Jason Dana, Stefanie Egidy, Dagmar Ellerbrock, Christoph Engel, Jens Frankenreiter, Simon Gächter, Gerd Gigerenzer, Russell Golman, Krishna P. Gummadi, Kristin Hagel, David Hagmann, Ulrike Hahn, Ralph Hertwig, Christian Hilbe, Derek M. Isaacowitz, Anne Kandler, Yaakov Kareev, Lewis A. Kornhauser, Joachim I. Krueger, Christina Leuker, Stephan Lewandowsky, Robert J. MacCoun, Richard McElreath, Thorsten Pachur, Peter J. Richerson, Lael J. Schooler, Laura Schmid, Barry Schwartz, Nora Szech, Eric Talley, Doron Teichman, Pete C. Trimmer, Sonja Utz, Lukasz Walasek, Michael R. Waldmann, Peter Wehling, Roi Yair, Eyal Zamir
Machines powered by artificial intelligence increasingly mediate our social, cultural, economic and political interactions. Understanding the behaviour of artificial intelligence systems is essential to our ability to control their actions, reap their benefits and minimize their harms. Here we argue that this necessitates a broad scientific research agenda to study machine behaviour that incorporates and expands upon the discipline of computer science and includes insights from across the sciences. We first outline a set of questions that are fundamental to this emerging field and then explore the technical, legal and institutional constraints on the study of machine behaviour. Understanding the behaviour of the machines powered by artificial intelligence that increasingly mediate our social, cultural, economic and political interactions is essential to our ability to control the actions of these intelligent machines, reap their benefits and minimize their harms.
Is it possible to judge someone accurately from his or her online activity? The Internet provides vast opportunities for individuals to present themselves in different ways, from simple self-enhancement to malicious identity fraud. We often rely on our Internet-based judgments of others to make decisions, such as whom to socialize with, date, or employ. Recently, personality-perception researchers have turned to studying social media and digital devices in order to ask whether a person’s digital traces can reveal aspects of his or her identity. Simultaneously, advances in “big data” analytics have demonstrated that computer algorithms can predict individuals’ traits from their digital traces. In this article, we address three questions: What do we currently know about human- and computer-based personality assessments? How accurate are these assessments? Where are these fields heading? We discuss trends in the current findings, provide an overview of methodological approaches, and recommend directions for future research.
To what extent does our online activity reveal who we are? Recent research has demonstrated that the digital traces left by individuals as they browse and interact with others online may reveal who they are and what their interests may be. In the present paper we report a systematic review that synthesises current evidence on predicting demographic attributes from online digital traces. Studies were included if they met the following criteria: (i) they reported findings where at least one demographic attribute was predicted/inferred from at least one form of digital footprint, (ii) the method of prediction was automated, and (iii) the traces were either visible (e.g. tweets) or non-visible (e.g. clickstreams). We identified 327 studies published up until October 2018. Across these articles, 14 demographic attributes were successfully inferred from digital traces; the most studied included gender, age, location, and political orientation. For each of the demographic attributes identified, we provide a database containing the platforms and digital traces examined, sample sizes, accuracy measures and the classification methods applied. Finally, we discuss the main research trends/findings, methodological approaches and recommend directions for future research.
Background
There is interest in using convolutional neural networks (CNNs) to analyze medical imaging to provide computer-aided diagnosis (CAD). Recent work has suggested that image classification CNNs may not generalize to new data as well as previously believed. We assessed how well CNNs generalized across three hospital systems for a simulated pneumonia screening task.
Methods and findings
A cross-sectional design with multiple model training cohorts was used to evaluate model generalizability to external sites using split-sample validation. A total of 158,323 chest radiographs were drawn from three institutions: National Institutes of Health Clinical Center (NIH; 112,120 from 30,805 patients), Mount Sinai Hospital (MSH; 42,396 from 12,904 patients), and Indiana University Network for Patient Care (IU; 3,807 from 3,683 patients). These patient populations had an age mean (SD) of 46.9 years (16.6), 63.2 years (16.5), and 49.6 years (17) with a female percentage of 43.5%, 44.8%, and 57.3%, respectively. We assessed individual models using the area under the receiver operating characteristic curve (AUC) for radiographic findings consistent with pneumonia and compared performance on different test sets with DeLong’s test. The prevalence of pneumonia was high enough at MSH (34.2%) relative to NIH and IU (1.2% and 1.0%) that merely sorting by hospital system achieved an AUC of 0.861 (95% CI 0.855–0.866) on the joint MSH–NIH dataset. Models trained on data from either NIH or MSH had equivalent performance on IU (P values 0.580 and 0.273, respectively) and inferior performance on data from each other relative to an internal test set (i.e., new data from within the hospital system used for training data; P values both <0.001). The highest internal performance was achieved by combining training and test data from MSH and NIH (AUC 0.931, 95% CI 0.927–0.936), but this model demonstrated significantly lower external performance at IU (AUC 0.815, 95% CI 0.745–0.885, P = 0.001). To test the effect of pooling data from sites with disparate pneumonia prevalence, we used stratified subsampling to generate MSH–NIH cohorts that only differed in disease prevalence between training data sites. When both training data sites had the same pneumonia prevalence, the model performed consistently on external IU data (P = 0.88). When a 10-fold difference in pneumonia rate was introduced between sites, internal test performance improved compared to the balanced model (10× MSH risk P < 0.001; 10× NIH P = 0.002), but this outperformance failed to generalize to IU (MSH 10× P < 0.001; NIH 10× P = 0.027). CNNs were able to directly detect hospital system of a radiograph for 99.95% NIH (22,050/22,062) and 99.98% MSH (8,386/8,388) radiographs. The primary limitation of our approach and the available public data is that we cannot fully assess what other factors might be contributing to hospital system–specific biases.
Conclusion
Pneumonia-screening CNNs achieved better internal than external performance in 3 out of 5 natural comparisons. When models were trained on pooled data from sites with different pneumonia prevalence, they performed better on new pooled data from these sites but not on external data. CNNs robustly identified hospital system and department within a hospital, which can have large differences in disease burden and may confound predictions.
Big Data analytics and artificial intelligence (AI) draw non-intuitive and unverifiable inferences and predictions about the behaviors, preferences, and private lives of individuals. These inferences draw on highly diverse and feature-rich data of unpredictable value, and create new opportunities for discriminatory, biased, and invasive decision-making. Concerns about algorithmic accountability are often actually concerns about the way in which these technologies draw privacy invasive and non-verifiable inferences about us that we cannot predict, understand, or refute.Data protection law is meant to protect people’s privacy, identity, reputation, and autonomy, but is currently failing to protect data subjects from the novel risks of inferential analytics. The broad concept of personal data in Europe could be interpreted to include inferences, predictions, and assumptions that refer to or impact on an individual. If seen as personal data, individuals are granted numerous rights under data protection law. However, the legal status of inferences is heavily disputed in legal scholarship, and marked by inconsistencies and contradictions within and between the views of the Article 29 Working Party and the European Court of Justice.As we show in this paper, individuals are granted little control and oversight over how their personal data is used to draw inferences about them. Compared to other types of personal data, inferences are effectively ‘economy class’ personal data in the General Data Protection Regulation (GDPR). Data subjects’ rights to know about (Art 13-15), rectify (Art 16), delete (Art 17), object to (Art 21), or port (Art 20) personal data are significantly curtailed when it comes to inferences, often requiring a greater balance with controller’s interests (e.g. trade secrets, intellectual property) than would otherwise be the case. Similarly, the GDPR provides insufficient protection against sensitive inferences (Art 9) or remedies to challenge inferences or important decisions based on them (Art 22(3)).This situation is not accidental. In standing jurisprudence the European Court of Justice (ECJ; Bavarian Lager, YS. and M. and S., and Nowak) and the Advocate General (AG; YS. and M. and S. and Nowak) have consistently restricted the remit of data protection law to assessing the legitimacy of input personal data undergoing processing, and to rectify, block, or erase it. Critically, the ECJ has likewise made clear that data protection law is not intended to ensure the accuracy of decisions and decision-making processes involving personal data, or to make these processes fully transparent.Conflict looms on the horizon in Europe that will further weaken the protection afforded to data subjects against inferences. Current policy proposals addressing privacy protection (the ePrivacy Regulation and the EU Digital Content Directive) fail to close the GDPR’s accountability gaps concerning inferences. At the same time, the GDPR and Europe’s new Copyright Directive aim to facilitate data mining, knowledge discovery, and Big Data analytics by limiting data subjects’ rights over personal data. And lastly, the new Trades Secrets Directive provides extensive protection of commercial interests attached to the outputs of these processes (e.g. models, algorithms and inferences).In this paper we argue that a new data protection right, the ‘right to reasonable inferences’, is needed to help close the accountability gap currently posed ‘high risk inferences’ , meaning inferences that are privacy invasive or reputation damaging and have low verifiability in the sense of being predictive or opinion-based. In cases where algorithms draw ‘high risk inferences’ about individuals, this right would require ex-ante justification to be given by the data controller to establish whether an inference is reasonable. This disclosure would address (1) why certain data is a relevant basis to draw inferences; (2) why these inferences are relevant for the chosen processing purpose or type of automated decision; and (3) whether the data and methods used to draw the inferences are accurate and statistically reliable. The ex-ante justification is bolstered by an additional ex-post mechanism enabling unreasonable inferences to be challenged. A right to reasonable inferences must, however, be reconciled with EU jurisprudence and counterbalanced with IP and trade secrets law as well as freedom of expression and Article 16 of the EU Charter of Fundamental Rights: the freedom to conduct a business.
Columbia Business Law Review, 2019(2).
Big Data analytics and artificial intelligence (AI) draw non-intuitive and unverifiable inferences and predictions about the behaviors, preferences, and private lives of individuals. These inferences draw on highly diverse and feature-rich data of unpredictable value, and create new opportunities for discriminatory, biased, and invasive decision-making. Data protection law is meant to protect people’s privacy, identity, reputation, and autonomy, but is currently failing to protect data subjects from the novel risks of inferential analytics. The legal status of inferences is heavily disputed in legal scholarship, and marked by inconsistencies and contradictions within and between the views of the Article 29 Working Party and the European Court of Justice (ECJ).
This Article shows that individuals are granted little control and oversight over how their personal data is used to draw inferences about them. Compared to other types of personal data, inferences are effectively ‘economy class’ personal data in the General Data Protection Regulation (GDPR). Data subjects’ rights to know about (Art 13-15), rectify (Art 16), delete (Art 17), object to (Art 21), or port (Art 20) personal data are significantly curtailed for inferences. The GDPR also provides insufficient protection against sensitive inferences (Art 9) or remedies to challenge inferences or important decisions based on them (Art 22(3)).
This situation is not accidental. In standing jurisprudence the ECJ has consistently restricted the remit of data protection law to assessing the legitimacy of input personal data undergoing processing, and to rectify, block, or erase it. Critically, the ECJ has likewise made clear that data protection law is not intended to ensure the accuracy of decisions and decision-making processes involving personal data, or to make these processes fully transparent. Current policy proposals addressing privacy protection (the ePrivacy Regulation and the EU Digital Content Directive) and Europe’s new Copyright Directive and Trade Secrets Directive also fail to close the GDPR’s accountability gaps concerning inferences.
This Article argues that a new data protection right, the ‘right to reasonable inferences’, is needed to help close the accountability gap currently posed by ‘high risk inferences’ , meaning inferences drawn from Big Data analytics that damage privacy or reputation, or have low verifiability in the sense of being predictive or opinion-based while being used in important decisions. This right would require ex-ante justification to be given by the data controller to establish whether an inference is reasonable. This disclosure would address (1) why certain data form a normatively acceptable basis from which to draw inferences; (2) why these inferences are relevant and normatively acceptable for the chosen processing purpose or type of automated decision; and (3) whether the data and methods used to draw the inferences are accurate and statistically reliable. The ex-ante justification is bolstered by an additional ex-post mechanism enabling unreasonable inferences to be challenged.
Significance
Many human endeavors—from teams and organizations to crowds and democracies—rely on solving problems collectively. Prior research has shown that when people interact and influence each other while solving complex problems, the average problem-solving performance of the group increases, but the best solution of the group actually decreases in quality. We find that when such influence is intermittent it improves the average while maintaining a high maximum performance. We also show that storing solutions for quick recall is similar to constant social influence. Instead of supporting more transparency, the results imply that technologies and organizations should be redesigned to intermittently isolate people from each other’s work for best collective performance in solving complex problems.
In one broad class of supervised machine learning problems, researchers and practitioners seek to estimate the likelihood of a given event in a manner that is fair. For example, when estimating a loan applicant's risk of default, one might aim to create a statistical model that is free of race or gender bias. Over the last several years, three formal definitions of fairness have gained popularity in the computer science literature: (1) anti-classification, meaning that protected attributes---like race, gender, and their proxies---are not used to form predictions; (2) classification parity, meaning that common measures of predictive performance (e.g., false positive and false negative rates) are equal across groups defined by the protected attributes; and (3) calibration, meaning that conditional on predictions, outcomes are independent of protected attributes. Here we show that all three of these fairness definitions suffer from significant statistical limitations. In particular, requiring anti-classification or classification parity can, perversely, harm the historically disadvantaged groups they were designed to protect, while also violating common legal, economic, and social understandings of fairness; and calibration, while generally desirable, provides little guarantee that decisions are equitable. By highlighting these challenges in the foundation of fair machine learning, we hope to help researchers and practitioners productively advance the area.
An examination of the cognitive tools that the mind uses to grapple with uncertainty in the real world.
How do humans navigate uncertainty, continuously making near-effortless decisions and predictions even under conditions of imperfect knowledge, high complexity, and extreme time pressure? Taming Uncertainty argues that the human mind has developed tools to grapple with uncertainty. Unlike much previous scholarship in psychology and economics, this approach is rooted in what is known about what real minds can do. Rather than reducing the human response to uncertainty to an act of juggling probabilities, the authors propose that the human cognitive system has specific tools for dealing with different forms of uncertainty. They identify three types of tools: simple heuristics, tools for information search, and tools for harnessing the wisdom of others. This set of strategies for making predictions, inferences, and decisions constitute the mind's adaptive toolbox.
The authors show how these three dimensions of human decision making are integrated and they argue that the toolbox, its cognitive foundation, and the environment are in constant flux and subject to developmental change. They demonstrate that each cognitive tool can be analyzed through the concept of ecological rationality—that is, the fit between specific tools and specific environments. Chapters deal with such specific instances of decision making as food choice architecture, intertemporal choice, financial uncertainty, pedestrian navigation, and adolescent behavior.
Drawing from many disciplines, the report adopts a behavioural psychology perspective to argue that “social media changes people’s political behaviour”. Four pressure points are identified and analysed in detail: the attention economy; choice architectures; algorithmic content curation; and mis/disinformation. Policy implications are outlined in detail.
Objective:
Despite abundant information about negative consequences of consuming meat, consumption in many Western countries is many times higher than recommended. One possible explanation for this discrepancy is that people consciously decide to ignore such information-a phenomenon called deliberate ignorance. We investigated this potential barrier for information interventions aiming to reduce meat consumption.
Methods:
In three studies, a total of 1133 participants had the opportunity to see 18 information chunks on negative consequences of meat consumption or to ignore part of the information. Deliberate ignorance was measured as the number of ignored information chunks. We assessed potential predictors and outcomes of deliberate ignorance. Interventions to reduce deliberate ignorance (i.e., self-affirmation, contemplation, and self-efficacy) were experimentally tested.
Results:
The more information participants ignored, the less they changed their intention to reduce their meat consumption (r = -.124). This effect was partially explained by cognitive dissonance induced by the presented information. While neither self-affirmation nor contemplation exercises reduced deliberate ignorance, self-efficacy exercises did.
Conclusion:
Deliberate ignorance is a potential barrier for information interventions aiming to reduce meat consumption and needs to be considered in future interventions and research. Self-efficacy exercises are a promising approach to reduce deliberate ignorance and should be further explored.
In online content moderation, two key values may come into conflict: protecting freedom of expression and preventing harm. Robust rules based in part on how citizens think about these moral dilemmas are necessary to deal with this conflict in a principled way, yet little is known about people's judgments and preferences around content moderation. We examined such moral dilemmas in a conjoint survey experiment where US respondents (N = 2, 564) indicated whether they would remove problematic social media posts on election denial, antivaccination, Holocaust denial, and climate change denial and whether they would take punitive action against the accounts. Respondents were shown key information about the user and their post as well as the consequences of the misinformation. The majority preferred quashing harmful misinformation over protecting free speech. Respondents were more reluctant to suspend accounts than to remove posts and more likely to do either if the harmful consequences of the misinformation were severe or if sharing it was a repeated offense. Features related to the account itself (the person behind the account, their partisanship, and number of followers) had little to no effect on respondents' decisions. Content moderation of harmful misinformation was a partisan issue: Across all four scenarios, Republicans were consistently less willing than Democrats or independents to remove posts or penalize the accounts that posted them. Our results can inform the design of transparent rules for content moderation of harmful misinformation.
As our dependence on intelligent machines continues to grow, so does the demand for more transparent and interpretable models. In addition, the ability to explain the model generally is now the gold standard for building trust and deployment of Artificial Intelligence (AI) systems in critical domains. Explainable Artificial Intelligence (XAI) aims to provide a suite of machine learning (ML) techniques that enable human users to understand, appropriately trust, and produce more explainable models. Selecting an appropriate approach for building an XAI-enabled application requires a clear understanding of the core ideas within XAI and the associated programming frameworks. We survey state-of-the-art programming techniques for XAI and present the different phases of XAI in a typical ML development process. We classify the various XAI approaches and using this taxonomy, discuss the key differences among the existing XAI techniques. Furthermore, concrete examples are used to describe these techniques that are mapped to programming frameworks and software toolkits. It is the intention that this survey will help stakeholders in selecting the appropriate approaches, programming frameworks, and software toolkits by comparing them through the lens of the presented taxonomy.
Medical systems in general, and patient treatment decisions and outcomes in particular, can be affected by bias based on gender and other demographic elements. As language models are increasingly applied to medicine, there is a growing interest in building algorithmic fairness into processes impacting patient care. Much of the work addressing this question has focused on biases encoded in language models—statistical estimates of the relationships between concepts derived from distant reading of corpora. Building on this work, we investigate how differences in gender-specific word frequency distributions and language models interact with regards to bias. We identify and remove gendered language from two clinical-note datasets and describe a new debiasing procedure using BERT-based gender classifiers. We show minimal degradation in health condition classification tasks for low- to medium-levels of dataset bias removal via data augmentation. Finally, we compare the bias semantically encoded in the language models with the bias empirically observed in health records. This work outlines an interpretable approach for using data augmentation to identify and reduce biases in natural language processing pipelines.
Over the last decade, the importance of machine learning increased dramatically in business and marketing. However, when machine learning is used for decision-making, bias rooted in unrepresentative datasets, inadequate models, weak algorithm designs, or human stereotypes can lead to low performance and unfair decisions, resulting in financial, social, and reputational losses. This paper offers a systematic, interdisciplinary literature review of machine learning biases as well as methods to avoid and mitigate these biases. We identified eight distinct machine learning biases, summarized these biases in the cross-industry standard process for data mining to account for all phases of machine learning projects, and outline twenty-four mitigation methods. We further contextualize these biases in a real-world case study and illustrate adequate mitigation strategies. These insights synthesize the literature on machine learning biases in a concise manner and point to the importance of human judgment for machine learning algorithms.
Motivated by the recent rise of populism in western democracies, we develop a tractable equilibrium model in which a populist backlash emerges endogenously in a strong economy. In the model, voters dislike inequality, especially the high consumption of “elites.” Economic growth exacerbates inequality due to heterogeneity in preferences, which leads to heterogeneity in returns on capital. In response to rising inequality, voters optimally elect a populist promising to end globalization. Equality is a luxury good. Countries with more inequality, higher financial development, and trade deficits are more vulnerable to populism, both in the model and in the data.
This article is protected by copyright. All rights reserved
This article surveys the use of algorithmic systems to support decision-making in the public sector. Governments adopt, procure, and use algorithmic systems to support their functions within several contexts—including criminal justice, education, and benefits provision—with important consequences for accountability, privacy, social inequity, and public participation in decision-making. We explore the social implications of municipal algorithmic systems across a variety of stages, including problem formulation, technology acquisition, deployment, and evaluation. We highlight several open questions that require further empirical research.
Expected final online publication date for the Annual Review of Law and Social Science, Volume 17 is October 2021. Please see http://www.annualreviews.org/page/journal/pubdates for revised estimates.
With the widespread use of artificial intelligence (AI) systems and applications in our everyday lives, accounting for fairness has gained significant importance in designing and engineering of such systems. AI systems can be used in many sensitive environments to make important and life-changing decisions; thus, it is crucial to ensure that these decisions do not reflect discriminatory behavior toward certain groups or populations. More recently some work has been developed in traditional machine learning and deep learning that address such challenges in different subdomains. With the commercialization of these systems, researchers are becoming more aware of the biases that these applications can contain and are attempting to address them. In this survey, we investigated different real-world applications that have shown biases in various ways, and we listed different sources of biases that can affect AI applications. We then created a taxonomy for fairness definitions that machine learning researchers have defined to avoid the existing bias in AI systems. In addition to that, we examined different domains and subdomains in AI showing what researchers have observed with regard to unfair outcomes in the state-of-the-art methods and ways they have tried to address them. There are still many future directions and solutions that can be taken to mitigate the problem of bias in AI systems. We are hoping that this survey will motivate researchers to tackle these issues in the near future by observing existing work in their respective fields.
Psychologists, economists, historians, computer scientists, sociologists, philosophers, and legal scholars explore the conscious choice not to seek information.
The history of intellectual thought abounds with claims that knowledge is valued and sought, yet individuals and groups often choose not to know. We call the conscious choice not to seek or use knowledge (or information) deliberate ignorance. When is this a virtue, when is it a vice, and what can be learned from formally modeling the underlying motives? On which normative grounds can it be judged? Which institutional interventions can promote or prevent it? In this book, psychologists, economists, historians, computer scientists, sociologists, philosophers, and legal scholars explore the scope of deliberate ignorance.
Drawing from multiple examples, including the right not to know in genetic testing, collective amnesia in transformational societies, blind orchestral auditions, and “don't ask don't tell” policies), the contributors offer novel insights and outline avenues for future research into this elusive yet fascinating aspect of human nature.
Contributors
Sarah Auster, Benjamin E. Berkman, Felix Bierbrauer, Gordon D. A. Brown, Jason Dana, Stefanie Egidy, Dagmar Ellerbrock, Christoph Engel, Jens Frankenreiter, Simon Gächter, Gerd Gigerenzer, Russell Golman, Krishna P. Gummadi, Kristin Hagel, David Hagmann, Ulrike Hahn, Ralph Hertwig, Christian Hilbe, Derek M. Isaacowitz, Anne Kandler, Yaakov Kareev, Lewis A. Kornhauser, Joachim I. Krueger, Christina Leuker, Stephan Lewandowsky, Robert J. MacCoun, Richard McElreath, Thorsten Pachur, Peter J. Richerson, Lael J. Schooler, Laura Schmid, Barry Schwartz, Nora Szech, Eric Talley, Doron Teichman, Pete C. Trimmer, Sonja Utz, Lukasz Walasek, Michael R. Waldmann, Peter Wehling, Roi Yair, Eyal Zamir
Rules for building formal models that use fast-and-frugal heuristics, extending the psychological study of classification to the real world of uncertainty.
This book focuses on classification—allocating objects into categories—“in the wild,” in real-world situations and far from the certainty of the lab. In the wild, unlike in typical psychological experiments, the future is not knowable and uncertainty cannot be meaningfully reduced to probability. Connecting the science of heuristics with machine learning, the book shows how to create formal models using classification rules that are simple, fast, and transparent and that can be as accurate as mathematically sophisticated algorithms developed for machine learning.
The authors—whose individual expertise ranges from empirical psychology to mathematical modeling to artificial intelligence and data science—offer real-world examples, including voting, HIV screening, and magistrate decision making; present an accessible guide to inducing the models statistically; compare the performance of such models to machine learning algorithms when applied to problems that include predicting diabetes or bank failure; and discuss conceptual and historical connections to cognitive psychology. Finally, they analyze such challenging safety-related applications as decreasing civilian casualties in checkpoints and regulating investment banks.
Deep learning has triggered the current rise of artificial intelligence and is the workhorse of today’s machine intelligence. Numerous success stories have rapidly spread all over science, industry and society, but its limitations have only recently come into focus. In this Perspective we seek to distil how many of deep learning’s failures can be seen as different symptoms of the same underlying problem: shortcut learning. Shortcuts are decision rules that perform well on standard benchmarks but fail to transfer to more challenging testing conditions, such as real-world scenarios. Related issues are known in comparative psychology, education and linguistics, suggesting that shortcut learning may be a common characteristic of learning systems, biological and artificial alike. Based on these observations, we develop a set of recommendations for model interpretation and benchmarking, highlighting recent advances in machine learning to improve robustness and transferability from the lab to real-world applications.
A recent wave of research has attempted to define fairness quantitatively. In particular, this work has explored what fairness might mean in the context of decisions based on the predictions of statistical and machine learning models. The rapid growth of this new field has led to wildly inconsistent motivations, terminology, and notation, presenting a serious challenge for cataloging and comparing definitions. This article attempts to bring much-needed order. First, we explicate the various choices and assumptions made—often implicitly—to justify the use of prediction-based decision-making. Next, we show how such choices and assumptions can raise fairness concerns and we present a notationally consistent catalog of fairness definitions from the literature. In doing so, we offer a concise reference for thinking through the choices, assumptions, and fairness considerations of prediction-based decision-making.
Expected final online publication date for the Annual Review of Statistics, Volume 8 is March 8, 2021. Please see http://www.annualreviews.org/page/journal/pubdates for revised estimates.
Advances in machine learning, specifically the subfield of deep learning, have produced algorithms that perform image-based diagnostic tasks with accuracy approaching or exceeding that of trained physicians. Despite their well-documented successes, these machine learning algorithms are vulnerable to cognitive and technical bias,¹ including bias introduced when an insufficient quantity or diversity of data is used to train an algorithm.²,3 We investigated an understudied source of systemic bias in clinical applications of deep learning—the geographic distribution of patient cohorts used to train algorithms.
Public opinion is shaped in significant part by online content, spread via social media and curated algorithmically. The current online ecosystem has been designed predominantly to capture user attention rather than to promote deliberate cognition and autonomous choice; information overload, finely tuned personalization and distorted social cues, in turn, pave the way for manipulation and the spread of false information. How can transparency and autonomy be promoted instead, thus fostering the positive potential of the web? Effective web governance informed by behavioural research is critically needed to empower individuals online. We identify technologically available yet largely untapped cues that can be harnessed to indicate the epistemic quality of online content, the factors underlying algorithmic decisions and the degree of consensus in online debates. We then map out two classes of behavioural interventions—nudging and boosting— that enlist these cues to redesign online environments for informed and autonomous choice.
The enormous financial success of online advertising platforms is partially due to the precise targeting features they offer. Although researchers and journalists have found many ways that advertisers can target---or exclude---particular groups of users seeing their ads, comparatively little attention has been paid to the implications of the platform's ad delivery process, comprised of the platform's choices about which users see which ads. It has been hypothesized that this process can "skew" ad delivery in ways that the advertisers do not intend, making some users less likely than others to see particular ads based on their demographic characteristics. In this paper, we demonstrate that such skewed delivery occurs on Facebook, due to market and financial optimization effects as well as the platform's own predictions about the "relevance" of ads to different groups of users. We find that both the advertiser's budget and the content of the ad each significantly contribute to the skew of Facebook's ad delivery. Critically, we observe significant skew in delivery along gender and racial lines for "real" ads for employment and housing opportunities despite neutral targeting parameters. Our results demonstrate previously unknown mechanisms that can lead to potentially discriminatory ad delivery, even when advertisers set their targeting parameters to be highly inclusive. This underscores the need for policymakers and platforms to carefully consider the role of the ad delivery optimization run by ad platforms themselves---and not just the targeting choices of advertisers---in preventing discrimination in digital advertising.
Racial bias in health algorithms
The U.S. health care system uses commercial algorithms to guide health decisions. Obermeyer et al. find evidence of racial bias in one widely used algorithm, such that Black patients assigned the same level of risk by the algorithm are sicker than White patients (see the Perspective by Benjamin). The authors estimated that this racial bias reduces the number of Black patients identified for extra care by more than half. Bias occurs because the algorithm uses health costs as a proxy for health needs. Less money is spent on Black patients who have the same level of need, and the algorithm thus falsely concludes that Black patients are healthier than equally sick White patients. Reformulating the algorithm so that it no longer uses costs as a proxy for needs eliminates the racial bias in predicting who needs extra care.
Science , this issue p. 447 ; see also p. 421
In the last 20 years, research on implicit social cognition has established that social judgments and behavior are guided by attitudes and stereotypes of which the actor may lack awareness. Research using the methods of implicit social cognition has produced the concept of implicit bias, which has generated wide attention not only in social, clinical, and developmental psychology, but also in disciplines outside of psychology, including business, law, criminal justice, medicine, education, and political science. Although this rapidly growing body of research offers prospects of useful societal applications, the theory needed to confidently guide those applications remains insufficiently developed. This article describes the methods that have been developed, the findings that have been obtained, and the theoretical questions that remain to be answered.
Expected final online publication date for the Annual Review of Psychology, Volume 71 is January 4, 2020. Please see http://www.annualreviews.org/page/journal/pubdates for revised estimates.
Black box machine learning models are currently being used for high-stakes decision making throughout society, causing problems in healthcare, criminal justice and other domains. Some people hope that creating methods for explaining these black box models will alleviate some of the problems, but trying to explain black box models, rather than creating models that are interpretable in the first place, is likely to perpetuate bad practice and can potentially cause great harm to society. The way forward is to design models that are inherently interpretable. This Perspective clarifies the chasm between explaining black boxes and using inherently interpretable models, outlines several key reasons why explainable black boxes should be avoided in high-stakes decisions, identifies challenges to interpretable machine learning, and provides several example applications where interpretable models could potentially replace black box models in criminal justice, healthcare and computer vision. There has been a recent rise of interest in developing methods for ‘explainable AI’, where models are created to explain how a first ‘black box’ machine learning model arrives at a specific decision. It can be argued that instead efforts should be directed at building inherently interpretable models in the first place, in particular where they are applied in applications that directly affect human lives, such as in healthcare and criminal justice.