No file available
To read the file of this research,
you can request a copy directly from the authors.
Analyzing Fairness of Computer Vision and Natural Language Processing Models
Preprints and early-stage research may not have been peer reviewed yet.
Abstract
Machine learning (ML) algorithms play a crucial role in decision making across diverse fields such as healthcare, finance, education, and law enforcement. Despite their widespread adoption, these systems raise ethical and social concerns due to potential biases and fairness issues. This study focuses on evaluating and improving the fairness of Computer Vision and Natural Language Processing (NLP) models applied to unstructured datasets, emphasizing how biased predictions can reinforce existing systemic inequalities. A publicly available dataset from Kaggle was utilized to simulate a practical scenario for examining fairness in ML workflows. To address and mitigate biases, the study employed two leading fairness libraries: Fairlearn by Microsoft, and AIF360 by IBM. These tools offer comprehensive frameworks for fairness analysis, including metrics evaluation, result visualization, and bias mitigation techniques. The research aims to measure bias levels in ML models, compare the effectiveness of these fairness libraries, and provide actionable recommendations for practitioners. The results demonstrate that each library possesses distinct strengths and limitations in evaluating and mitigating fairness. By systematically analyzing these tools, the study contributes valuable insights to the growing field of ML fairness, offering practical guidance for integrating fairness solutions into real world applications. This research underscores the importance of building more equitable and responsible machine learning systems.
ResearchGate has not been able to resolve any citations for this publication.
Understanding why a model makes a certain prediction can be as crucial as the prediction's accuracy in many applications. However, the highest accuracy for large modern datasets is often achieved by complex models that even experts struggle to interpret, such as ensemble or deep learning models, creating a tension between accuracy and interpretability. In response, various methods have recently been proposed to help users interpret the predictions of complex models, but it is often unclear how these methods are related and when one method is preferable over another. To address this problem, we present a unified framework for interpreting predictions, SHAP (SHapley Additive exPlanations). SHAP assigns each feature an importance value for a particular prediction. Its novel components include: (1) the identification of a new class of additive feature importance measures, and (2) theoretical results showing there is a unique solution in this class with a set of desirable properties. The new class unifies six existing methods, notable because several recent methods in the class lack the proposed desirable properties. Based on insights from this unification, we present new methods that show improved computational performance and/or better consistency with human intuition than previous approaches.
Are judicial rulings based solely on laws and facts? Legal formalism holds that judges apply legal reasons to the facts of a case in a rational, mechanical, and deliberative manner. In contrast, legal realists argue that the rational application of legal reasons does not sufficiently explain the decisions of judges and that psychological, political, and social factors influence judicial rulings. We test the common caricature of realism that justice is "what the judge ate for breakfast" in sequential parole decisions made by experienced judges. We record the judges' two daily food breaks, which result in segmenting the deliberations of the day into three distinct "decision sessions." We find that the percentage of favorable rulings drops gradually from ≈ 65% to nearly zero within each decision session and returns abruptly to ≈ 65% after a break. Our findings suggest that judicial rulings can be swayed by extraneous variables that should have no bearing on legal decisions.
In recent years, fairness in machine learning (ML), artificial intelligence (AI), and algorithmic decision-making systems has emerged as a highly active area of research and development. To date, most measures and methods to mitigate bias and improve fairness in algorithmic systems have been built in isolation from policymaking and civil societal contexts and lack serious engagement with philosophical, political, legal, and economic theories of equality and distributive justice. Many current measures define “fairness” in simple terms to mean narrowing gaps in performance or outcomes between demographic groups while preserving as much of the original system’s accuracy as possible. This oversimplified translation of the complex socio-legal concept of equality into fairness measures is troubling. Many current fairness measures suffer from both fairness and performance degradation—or “leveling down”—where fairness is achieved by making every group worse off or by bringing better-performing groups down to the level of the worst off. Leveling down is a symptom of the decision to measure fairness solely in terms of equality, or disparity between groups in performance and outcomes, that ignores other relevant concerns of distributive justice (e.g., welfare or priority), which are more difficult to quantify and measure. When fairness can only be measured in terms of distribution of performance or outcomes, corrective actions can likewise only target how these goods are distributed between groups. We refer to this trend as “strict egalitarianism by default.” Strict egalitarianism by default runs counter to both the stated objectives of fairness measures and the presumptive aim of the field: to improve outcomes for historically disadvantaged or marginalized groups. When fairness can only be achieved by making everyone worse off in material or relational terms–through injuries of stigma, loss of solidarity, unequal concern, and missed opportunities for substantive equality—something has gone wrong in translating the vague concept of “fairness” into practice. Leveling down should be rejected in fairML because it (1) unnecessarily and arbitrarily harms advantaged groups in cases where performance is intrinsically valuable, such as medical applications of AI; (2) demonstrates a lack of equal concern for affected groups, undermines social solidarity, and contributes to stigmatization; (3) fails to live up to the substantive aims of equality law and fairML and squanders the opportunity afforded by interest in algorithmic fairness to substantively address longstanding social inequalities; and (4) fails to meet the aims of many viable theories of distributive justice including pluralist egalitarian approaches, prioritarianism, sufficientarianism, and others. This paper critically scrutinizes these initial observations to determine how fairML can move beyond mere leveling down and strict egalitarianism by default. We examine the causes and prevalence of leveling down across fairML and explore possible justifications and criticisms based on philosophical and legal theories of equality and distributive justice as well as equality-law jurisprudence. We find that fairML does not currently engage in the type of measurement, reporting, or analysis necessary to justify leveling down in practice. The types of decisions for which ML and AI are currently used, as well as inherent limitations on data collection and measurement, suggest leveling down is rarely justified in practice. We propose a first step toward substantive equality in fairML: “leveling up” systems by enforcing minimum acceptable harm thresholds, or “minimum rate constraints,” as fairness constraints at the design stage. We likewise propose an alternative harms-based framework to counter the oversimplified egalitarian framing currently dominant in the field and push future discussion more towards substantive equality opportunities and away from strict egalitarianism by default.
With the widespread use of artificial intelligence (AI) systems and applications in our everyday lives, accounting for fairness has gained significant importance in designing and engineering of such systems. AI systems can be used in many sensitive environments to make important and life-changing decisions; thus, it is crucial to ensure that these decisions do not reflect discriminatory behavior toward certain groups or populations. More recently some work has been developed in traditional machine learning and deep learning that address such challenges in different subdomains. With the commercialization of these systems, researchers are becoming more aware of the biases that these applications can contain and are attempting to address them. In this survey, we investigated different real-world applications that have shown biases in various ways, and we listed different sources of biases that can affect AI applications. We then created a taxonomy for fairness definitions that machine learning researchers have defined to avoid the existing bias in AI systems. In addition to that, we examined different domains and subdomains in AI showing what researchers have observed with regard to unfair outcomes in the state-of-the-art methods and ways they have tried to address them. There are still many future directions and solutions that can be taken to mitigate the problem of bias in AI systems. We are hoping that this survey will motivate researchers to tackle these issues in the near future by observing existing work in their respective fields.
AI technologies have the potential to dramatically impact the lives of people with disabilities (PWD). Indeed, improving the lives of PWD is a motivator for many state-of-the-art AI systems, such as automated speech recognition tools that can caption videos for people who are deaf and hard of hearing, or language prediction algorithms that can augment communication for people with speech or cognitive disabilities. However, widely deployed AI systems may not work properly for PWD, or worse, may actively discriminate against them. These considerations regarding fairness in AI for PWD have thus far received little attention. In this position paper, we identify potential areas of concern regarding how several AI technology categories may impact particular disability constituencies if care is not taken in their design, development, and testing. We intend for this risk assessment of how various classes of AI might interact with various classes of disability to provide a roadmap for future research that is needed to gather data, test these hypotheses, and build more inclusive algorithms.
Machine learning is a tool for building models that accurately represent input training data. When undesired biases concerning demographic groups are in the training data, well-trained models will reflect those biases. We present a framework for mitigating such biases by including a variable for the group of interest and simultaneously learning a predictor and an adversary. The input to the network X, here text or census data, produces a prediction Y, such as an analogy completion or income bracket, while the adversary tries to model a protected variable Z, here gender or zip code. The objective is to maximize the predictor's ability to predict Y while minimizing the adversary's ability to predict Z. Applied to analogy completion, this method results in accurate predictions that exhibit less evidence of stereotyping Z. When applied to a classification task using the UCI Adult (Census) Dataset, it results in a predictive model that does not lose much accuracy while achieving very close to equality of odds (Hardt, et al., 2016). The method is flexible and applicable to multiple definitions of fairness as well as a wide range of gradient-based learning models, including both regression and classification tasks.
The most prevalent notions of fairness in machine learning are statistical definitions: they fix a small collection of pre-defined groups, and then ask for parity of some statistic of the classifier across these groups. Constraints of this form are susceptible to (intentional or inadvertent) "fairness gerrymandering", in which a classifier appears to be fair on each individual group, but badly violates the fairness constraint on one or more structured subgroups defined over the protected attributes.. We propose instead to demand statistical notions of fairness across exponentially (or infinitely) many subgroups, defined by a structured class of functions over the protected attributes. This interpolates between statistical definitions of fairness, and recently proposed individual notions of fairness, but it raises several computational challenges. It is no longer clear how to even audit a fixed classifier to see if it satisfies such a strong definition of fairness. We prove that the computational problem of auditing subgroup fairness for both equality of false positive rates and statistical parity is equivalent to the problem of weak agnostic learning --- which means it is computationally hard in the worst case, even for simple structured subclasses. However, it also suggests that common heuristics for learning can be applied to successfully solve the auditing problem in practice. We then derive an algorithm that provably converges to the best fair distribution over classifiers in a given class, given access to oracles which can solve the agnostic learning and auditing problems. The algorithm is based on a formulation of subgroup fairness as fictitious play in a two-player zero-sum game between a Learner and an Auditor. We implement our algorithm using linear regression as a heuristic oracle, and show that we can effectively both audit and learn fair classifiers on real datasets.
Recidivism prediction instruments provide decision makers with an assessment of the likelihood that a criminal defendant will reoffend at a future point in time. While such instruments are gaining increasing popularity across the country, their use is attracting tremendous controversy. Much of the controversy concerns potential discriminatory bias in the risk assessments that are produced. This paper discusses a fairness criterion originating in the field of educational and psychological testing that has recently been applied to assess the fairness of recidivism prediction instruments. We demonstrate how adherence to the criterion may lead to considerable disparate impact when recidivism prevalence differs across groups.
We propose a criterion for discrimination against a specified sensitive attribute in supervised learning, where the goal is to predict some target based on available features. Assuming data about the predictor, target, and membership in the protected group are available, we show how to optimally adjust any learned predictor so as to remove discrimination according to our definition. Our framework also improves incentives by shifting the cost of poor classification from disadvantaged groups to the decision maker, who can respond by improving the classification accuracy. In line with other studies, our notion is oblivious: it depends only on the joint statistics of the predictor, the target and the protected attribute, but not on interpretation of individualfeatures. We study the inherent limits of defining and identifying biases based on such oblivious measures, outlining what can and cannot be inferred from different oblivious tests. We illustrate our notion using a case study of FICO credit scores.
Despite widespread adoption, machine learning models remain mostly black boxes. Understanding the reasons behind predictions is, however, quite important in assessing trust, which is fundamental if one plans to take action based on a prediction, or when choosing whether to deploy a new model. Such understanding also provides insights into the model, which can be used to transform an untrustworthy model or prediction into a trustworthy one.
In this work, we propose LIME, a novel explanation technique that explains the predictions of any classifier in an interpretable and faithful manner, by learning an interpretable model locally varound the prediction. We also propose a method to explain models by presenting representative individual predictions and their explanations in a non-redundant way, framing the task as a submodular optimization problem. We demonstrate the flexibility of these methods by explaining different models for text (e.g. random forests) and image classification (e.g. neural networks). We show the utility of explanations via novel experiments, both simulated and with human subjects, on various scenarios that require trust: deciding if one should trust a prediction, choosing between models, improving an untrustworthy classifier, and identifying why a classifier should not be trusted.
Multi-Criteria Decision Making is an increasingly accepted tool for decision making in management. In this work, we highlight the application of a novel Multi-Objective Evolutionary Algorithm, NSGA-II to the risk-return trade-off for a bank loan portfolio manager. The manager of a bank operating in a competitive environment faces the stan-dard goal of maximizing shareholder wealth. Specifically, this attempts to maxi-mize the net worth of the bank, which in turn involves maximizing the net interest margin of the bank (among other factors, such as non-interest income). At the same time, there are significant regulatory constraints placed on the bank, such as the maintenance of adequate capital, interest-rate risk exposure, etc. The Genetic Algorithm based technique used here obtains an approximation to the set of Pareto-optimal solutions which increases the decision flexibility avail-able to the bank manager and provides a visualization tool for one of the tradeoffs involved. The algorithm is also computationally efficient and is contrasted with a traditional multi-objective function -the epsilon-constraint method.
HelpWanted: An Examination of Hiring Algorithms, Equity and Bias
- Miranda Bogen
- Aaron Rieke
Miranda Bogen and Aaron Rieke. 2018. HelpWanted: An Examination of Hiring Algorithms, Equity
and Bias. Technical Report. Upturn.
Efficient candidate screening under multiple tests and implications for fairness
- Lee Cohen
- Zachary C Lipton
- Yishay Mansour
Lee Cohen, Zachary C. Lipton, and Yishay Mansour. 2019. Efficient candidate screening under
multiple tests and implications for fairness. arXiv:cs.LG/1905.11361 (2019).
Machine Bias: There's Software Used Across the Country to Predict Future Criminals. And it's Biased Against Blacks
- J Angwin
- J Larson
- S Mattu
- L Kirchner
Angwin, J., Larson, J., Mattu, S., & Kirchner, L. (2016). Machine Bias: There's Software Used
Across the Country to Predict Future Criminals. And it's Biased Against Blacks. ProPublica.
Weapons of Math Destruction: How Big Data Increases Inequality and Threatens Democracy
- O' Cathy
- Neil
Cathy O'Neil. 2016.Weapons of Math Destruction: How Big Data Increases Inequality and
Threatens Democracy. Crown Publishing Group, New York, NY.
Gender Shades: Intersectional Accuracy Disparities in Commercial Gender Classification
- J Buolamwini
- T Gebru
Buolamwini, J., & Gebru, T. (2018). Gender Shades: Intersectional Accuracy Disparities in
Commercial Gender Classification. Proceedings of the 2018 Conference on Fairness,
Accountability, and Transparency (FAT).
A Framework for Understanding Unintended Consequences of Machine Learning
- H Suresh
- J V Guttag
Suresh, H., & Guttag, J. V. (2021). A Framework for Understanding Unintended Consequences
of Machine Learning. Communications of the ACM, 64(8), 62-71.
Certifying and Removing Disparate Impact
- M Feldman
- S A Friedler
- J Moeller
- C Scheidegger
- S Venkatasubramanian
Feldman, M., Friedler, S. A., Moeller, J., Scheidegger, C., & Venkatasubramanian, S. (2015).
Certifying and Removing Disparate Impact. Proceedings of the 21st ACM SIGKDD International
Conference on Knowledge Discovery and Data Mining (KDD).
Fairness in Machine Learning: Lessons from Political Philosophy
- R Binns
Binns, R. (2018). Fairness in Machine Learning: Lessons from Political Philosophy. Proceedings
of the 2018 Conference on Fairness, Accountability, and Transparency (FAT).
Data Preprocessing Techniques for Classification without Discrimination
- F Kamiran
- T Calders
Kamiran, F., & Calders, T. (2012). Data Preprocessing Techniques for Classification without
Discrimination. Knowledge and Information Systems, 33(1), 1-33.
Fairness and Accountability in Machine Learning
- S Barocas
- M Hardt
- A Narayanan
Barocas, S., Hardt, M., & Narayanan, A. (2019). Fairness and Accountability in Machine
Learning. Proceedings of the 2019 Conference on Fairness, Accountability, and Transparency.
Does the end justify the means? on the moral justification of fairness-aware machine learning
- Hilde Weerts
- Lambèr Royakkers
- Mykola Pechenizkiy
Hilde Weerts, Lambèr Royakkers, and Mykola Pechenizkiy. Does the end justify the means? on
the moral justification of fairness-aware machine learning. arXiv preprint arXiv:2202.08536, 2022.