AI can be sexist and racist — it’s time to make it fair

James Zou,Londa Schiebinger

DOI: https://doi.org/10.1038/d41586-018-05707-8

IF: 64.8

2018-07-01

Nature

Abstract:Computer scientists must identify sources of bias, de-bias training data and develop artificial-intelligence algorithms that are robust to skews in the data, argue James Zou and Londa Schiebinger.

multidisciplinary sciences

What problem does this paper attempt to address?

Feminist AI: Can We Expect Our AI Systems to Become Feminist?

Galit Wellner,Tiran Rothman

DOI: https://doi.org/10.1007/s13347-019-00352-z

2019-05-18

Abstract:The rise of AI-based systems has been accompanied by the belief that these systems are impartial and do not suffer from the biases that humans and older technologies express. It becomes evident, however, that gender and racial biases exist in some AI algorithms. The question is where the bias is rooted—in the training dataset or in the algorithm? Is it a linguistic issue or a broader sociological current? Works in feminist philosophy of technology and behavioral economics reveal the gender bias in AI technologies as a multi-faceted phenomenon, and the linguistic explanation as too narrow. The next step moves from the linguistic aspects to the relational ones, with postphenomenology. One of the analytical tools of this theory is the "I-technology-world" formula that models our relations with technologies, and through them—with the world. Realizing that AI technologies give rise to new types of relations in which the technology has an "enhanced technological intentionality", a new formula is suggested: "I-algorithm-dataset." In the third part of the article, four types of solutions to the gender bias in AI are reviewed: ignoring any reference to gender, revealing the considerations that led the algorithm to decide, designing algorithms that are not biased, or lastly, involving humans in the process. In order to avoid gender bias, we can recall a feminist basic understanding—visibility matters. Users and developers should be aware of the possibility of gender and racial biases, and try to avoid them, bypass them, or exterminates them altogether.
Addressing Bias and Fairness Issues in Artificial Intelligence

Pragya Gupta

Abstract:-This paper explores the multifaceted aspects of bias in AI, encompassing its types, sources, implications, and mitigation strategies. Drawing on real-world examples, it delves into reporting bias, selection bias, group attribution bias, and implicit bias, highlighting their impact on societal inequalities and marginalized groups. The reinforcement of historical biases in AI training data perpetuates discrimination and hampers progress towards equality. The paper discusses quantitative measures like disparate impact and demographic parity, while emphasizing the vital role of qualitative assessments and human evaluators in identifying bias. Furthermore, it explores strategies for addressing bias, such as diverse training data, in-processing and post-processing models, and algorithmic debiasing techniques. The article also underscores interdisciplinary collaboration, ethical considerations, and regulatory standards as essential components of building fair and accountable AI systems. Lastly, it outlines future directions for research, including adaptive algorithms, intersectional fairness, inclusive development, and robust ethical frameworks, aiming to guide AI towards equitable and responsible advancement.

Political Science,Sociology,Computer Science
Bias, Fairness and Accountability with Artificial Intelligence and Machine Learning Algorithms

Nengfeng Zhou,Zach Zhang,Vijayan N. Nair,Harsh Singhal,Jie Chen

DOI: https://doi.org/10.1111/insr.12492

2022-04-10

International Statistical Review

Abstract:SummaryThe advent of artificial intelligence (AI) and machine learning algorithms has led to opportunities as well as challenges in their use. In this overview paper, we begin with a discussion of bias and fairness issues that arise with the use of AI techniques, with a focus on supervised machine learning algorithms. We then describe the types and sources of data bias and discuss the nature of algorithmic unfairness. In addition, we provide a review of fairness metrics in the literature, discuss their limitations, and describe de‐biasing (or mitigation) techniques in the model life cycle.

statistics & probability
AI Algorithmic Bias: Understanding its Causes, Ethical and Social Implications

Lakshitha R Jain,Vineetha Menon

DOI: https://doi.org/10.1109/ICTAI59109.2023.00073

2023-11-06

Abstract:The escalating usage of artificial intelligence (AI) and machine learning algorithms across diverse fields has prompted apprehension regarding the propagation of algorithmic bias, which may exacerbate instances of discrimination and inequality. Algorithmic bias in AI and machine learning (ML) techniques manifests in real-world applications as a result of either insufficient data variation or augmentation availability in the AI/ML training data, or a flawed learning policy. This leads to the accidental propagation of AI bias as an unjust treatment of particular groups of individuals, owing to their race, gender [1], age, or other distinguishing attributes in practical applications. This paper offers a comprehensive analysis of algorithmic bias, encompassing its origins, ethical and social ramifications, and possible remediations. In addition, this paper introduces an innovative methodology for identifying and measuring algorithmic bias that integrates statistical analysis with input from users and domain specialists. This exposition examines distinct forms of algorithmic biases, such as selection bias, confirmation bias, and measurement bias, and examines underlying catalysts for algorithmic bias, encompassing data integrity concerns, decisions regarding algorithmic design, and institutional prejudgments. The adverse ramifications of algorithmic bias, including the perpetuation of social inequality and the impeding of societal advancement, are the focus of our examination. The present study seeks to make a contribution to the advancement of impartial [2] and equitable AI systems with the potential to foster societal progress and benefit individuals across diverse demographics by identifying the sources and repercussions of algorithmic bias and recommending efficacious interventions.

Sociology,Computer Science
Does Gender Matter? Towards Fairness in Dialogue Systems

Haochen Liu,Jamell Dacon,Wenqi Fan,Hui Liu,Zitao Liu,Jiliang Tang

DOI: https://doi.org/10.18653/v1/2020.coling-main.390

2019-01-01

Abstract:Recently there are increasing concerns about the fairness of Artificial Intelligence (AI) in real-world applications such as computer vision and recommendations. For example, recognition algorithms in computer vision are unfair to black people such as poorly detecting their faces and inappropriately identifying them as "gorillas". As one crucial application of AI, dialogue systems have been extensively applied in our society. They are usually built with real human conversational data; thus they could inherit some fairness issues which are held in the real world. However, the fairness of dialogue systems has not been well investigated. In this paper, we perform a pioneering study about the fairness issues in dialogue systems. In particular, we construct a benchmark dataset and propose quantitative measures to understand fairness in dialogue models. Our studies demonstrate that popular dialogue models show significant prejudice towards different genders and races. Besides, to mitigate the bias in dialogue systems, we propose two simple but effective debiasing methods. Experiments show that our methods can reduce the bias in dialogue systems significantly. The dataset and the implementation are released to foster fairness research in dialogue systems.
AI’s fairness problem: understanding wrongful discrimination in the context of automated decision-making

Hugo Cossette-Lefebvre,Jocelyn Maclure

DOI: https://doi.org/10.1007/s43681-022-00233-w

2022-11-16

AI and Ethics

Abstract:Abstract The use of predictive machine learning algorithms is increasingly common to guide or even take decisions in both public and private settings. Their use is touted by some as a potentially useful method to avoid discriminatory decisions since they are, allegedly, neutral, objective, and can be evaluated in ways no human decisions can. By (fully or partly) outsourcing a decision process to an algorithm, it should allow human organizations to clearly define the parameters of the decision and to, in principle, remove human biases. Yet, in practice, the use of algorithms can still be the source of wrongful discriminatory decisions based on at least three of their features: the data-mining process and the categorizations they rely on can reconduct human biases, their automaticity and predictive design can lead them to rely on wrongful generalizations, and their opaque nature is at odds with democratic requirements. We highlight that the two latter aspects of algorithms and their significance for discrimination are too often overlooked in contemporary literature. Though these problems are not all insurmountable, we argue that it is necessary to clearly define the conditions under which a machine learning decision tool can be used. We identify and propose three main guidelines to properly constrain the deployment of machine learning algorithms in society: algorithms should be vetted to ensure that they do not unduly affect historically marginalized groups; they should not systematically override or replace human decision-making processes; and the decision reached using an algorithm should always be explainable and justifiable.
Oxford Handbook on AI Ethics Book Chapter on Race and Gender

Timnit Gebru

DOI: https://doi.org/10.48550/arXiv.1908.06165

2019-08-08

Abstract:From massive face-recognition-based surveillance and machine-learning-based decision systems predicting crime recidivism rates, to the move towards automated health diagnostic systems, artificial intelligence (AI) is being used in scenarios that have serious consequences in people's lives. However, this rapid permeation of AI into society has not been accompanied by a thorough investigation of the sociopolitical issues that cause certain groups of people to be harmed rather than advantaged by it. For instance, recent studies have shown that commercial face recognition systems have much higher error rates for dark skinned women while having minimal errors on light skinned men. A 2016 ProPublica investigation uncovered that machine learning based tools that assess crime recidivism rates in the US are biased against African Americans. Other studies show that natural language processing tools trained on newspapers exhibit societal biases (e.g. finishing the analogy "Man is to computer programmer as woman is to X" by homemaker). At the same time, books such as Weapons of Math Destruction and Automated Inequality detail how people in lower socioeconomic classes in the US are subjected to more automated decision making tools than those who are in the upper class. Thus, these tools are most often used on people towards whom they exhibit the most bias. While many technical solutions have been proposed to alleviate bias in machine learning systems, we have to take a holistic and multifaceted approach. This includes standardization bodies determining what types of systems can be used in which scenarios, making sure that automated decision tools are created by people from diverse backgrounds, and understanding the historical and political factors that disadvantage certain groups who are subjected to these tools.

Computers and Society,Artificial Intelligence
An Introduction to Artificial Intelligence and Solutions to the Problems of Algorithmic Discrimination

Nicholas Schmidt,Bryce Stephens

DOI: https://doi.org/10.48550/arXiv.1911.05755

2019-11-09

Abstract:There is substantial evidence that Artificial Intelligence (AI) and Machine Learning (ML) algorithms can generate bias against minorities, women, and other protected classes. Federal and state laws have been enacted to protect consumers from discrimination in credit, housing, and employment, where regulators and agencies are tasked with enforcing these laws. Additionally, there are laws in place to ensure that consumers understand why they are denied access to services and products, such as consumer loans. In this article, we provide an overview of the potential benefits and risks associated with the use of algorithms and data, and focus specifically on fairness. While our observations generalize to many contexts, we focus on the fairness concerns raised in consumer credit and the legal requirements of the Equal Credit and Opportunity Act. We propose a methodology for evaluating algorithmic fairness and minimizing algorithmic bias that aligns with the provisions of federal and state anti-discrimination statutes that outlaw overt, disparate treatment, and, specifically, disparate impact discrimination. We argue that while the use of AI and ML algorithms heighten potential discrimination risks, these risks can be evaluated and mitigated, but doing so requires a deep understanding of these algorithms and the contexts and domains in which they are being used.

Computers and Society,Machine Learning
Bias in Artificial Intelligence Systems

Rafał Rejmaniak

DOI: https://doi.org/10.15290/bsp.2021.26.03.02

2021-09-01

Białostockie Studia Prawnicze

Abstract:Abstract Artificial intelligence systems are currently deployed in many areas of human activity. Such systems are increasingly assigned tasks that involve taking decisions about people or predicting future behaviours. These decisions are commonly regarded as fairer and more objective than those taken by humans, as AI systems are thought to be resistant to such influences as emotions or subjective beliefs. In reality, using such a system does not guarantee either objectivity or fairness. This article describes the phenomenon of bias in AI systems and the role of humans in creating it. The analysis shows that AI systems, even if operating correctly from a technical standpoint, are not guaranteed to take decisions that are more objective than those of a human, but those systems can still be used to reduce social inequalities.
Does AI Debias Recruitment? Race, Gender, and AI's "Eradication of Difference"

Eleanor Drage,Kerry Mackereth

DOI: https://doi.org/10.1007/s13347-022-00543-1

Abstract:In this paper, we analyze two key claims offered by recruitment AI companies in relation to the development and deployment of AI-powered HR tools: (1) recruitment AI can objectively assess candidates by removing gender and race from their systems, and (2) this removal of gender and race will make recruitment fairer, help customers attain their DEI goals, and lay the foundations for a truly meritocratic culture to thrive within an organization. We argue that these claims are misleading for four reasons: First, attempts to "strip" gender and race from AI systems often misunderstand what gender and race are, casting them as isolatable attributes rather than broader systems of power. Second, the attempted outsourcing of "diversity work" to AI-powered hiring tools may unintentionally entrench cultures of inequality and discrimination by failing to address the systemic problems within organizations. Third, AI hiring tools' supposedly neutral assessment of candidates' traits belie the power relationship between the observer and the observed. Specifically, the racialized history of character analysis and its associated processes of classification and categorization play into longer histories of taxonomical sorting and reflect the current demands and desires of the job market, even when not explicitly conducted along the lines of gender and race. Fourth, recruitment AI tools help produce the "ideal candidate" that they supposedly identify through by constructing associations between words and people's bodies. From these four conclusions outlined above, we offer three key recommendations to AI HR firms, their customers, and policy makers going forward.
Addressing Bias in Artificial Intelligence in Health Care

Ravi B. Parikh,Stephanie Teeple,Amol S. Navathe

DOI: https://doi.org/10.1001/jama.2019.18058

IF: 11.816

2019-12-24

JAMA

Abstract:Recent scrutiny of artificial intelligence (AI)–based facial recognition software has renewed concerns about the unintended effects of AI on social bias and inequity. Academic and government officials have raised concerns over racial and gender bias in several AI-based technologies, including internet search engines and algorithms to predict risk of criminal behavior. Companies like IBM and Microsoft have made public commitments to "de-bias" their technologies, whereas Amazon mounted a public campaign criticizing such research. As AI applications gain traction in medicine, clinicians and health system leaders have raised similar concerns over automating and propagating existing biases.<sup><a class="ref-link section-jump-link" href="#jvp190155r1">1</a></sup>

medicine, general & internal
Ethics and discrimination in artificial intelligence-enabled recruitment practices

Zhisheng Chen

DOI: https://doi.org/10.1057/s41599-023-02079-x

2023-09-14

Humanities and Social Sciences Communications

Abstract:This study aims to address the research gap on algorithmic discrimination caused by AI-enabled recruitment and explore technical and managerial solutions. The primary research approach used is a literature review. The findings suggest that AI-enabled recruitment has the potential to enhance recruitment quality, increase efficiency, and reduce transactional work. However, algorithmic bias results in discriminatory hiring practices based on gender, race, color, and personality traits. The study indicates that algorithmic bias stems from limited raw data sets and biased algorithm designers. To mitigate this issue, it is recommended to implement technical measures, such as unbiased dataset frameworks and improved algorithmic transparency, as well as management measures like internal corporate ethical governance and external oversight. Employing Grounded Theory, the study conducted survey analysis to collect firsthand data on respondents' experiences and perceptions of AI-driven recruitment applications and discrimination.
AI and Social Impact

Sandra Moore,Sheena Brown,William Butler

DOI: https://doi.org/10.4018/978-1-7998-8693-8.ch008

2022-01-01

Abstract:Just because everything can be automated does not mean everything should be. As machine learning and artificial intelligence become intertwined within the global fabric of society, potential societal impacts must be considered. Is it necessary to know someone's sexual orientation? Will that help sell products or pose a threat to that individual? Countries call for such algorithms; however, the literature has shown that current attempts are neither significant nor correct most of the time. Is it important to know one's race? What happens when a person of color is targeted based on biased algorithms by police? Or denied a loan based on biased resourcing that indicates low-income individuals are more likely to offend? These algorithms contain a multitude of bias based on the datasets used. The use of inclusive datasets is necessary to get accurate, unbiased, and therefore viable data to ensure that AI technologies function correctly.
Bias Amplification in Artificial Intelligence Systems

Kirsten Lloyd

DOI: https://doi.org/10.48550/arXiv.1809.07842

IF: 14.4

2018-09-20

Artificial Intelligence

Abstract:As Artificial Intelligence (AI) technologies proliferate, concern has centered around the long-term dangers of job loss or threats of machines causing harm to humans. All of this concern, however, detracts from the more pertinent and already existing threats posed by AI today: its ability to amplify bias found in training datasets, and swiftly impact marginalized populations at scale. Government and public sector institutions have a responsibility to citizens to establish a dialogue with technology developers and release thoughtful policy around data standards to ensure diverse representation in datasets to prevent bias amplification and ensure that AI systems are built with inclusion in mind.
Gender Bias in AI Recruitment Systems: A Sociological-and Data Science-based Case Study

Anthony Wirth,Sheilla Njoto,Reeva M. Lederman,M. Cheong,A. McLoughney,Leah Ruppanner

DOI: https://doi.org/10.1109/ISTAS55053.2022.10227106

2022-11-10

Abstract:This paper explores the extent to which gender bias is introduced in the deployment of automation for hiring practices. We use an interdisciplinary methodology to test our hypotheses: observing a human-led recruitment panel and building an explainable algorithmic prototype from the ground up, to quantify gender bias. The key findings of this study are threefold: identifying potential sources of human bias from a recruitment panel’s ranking of CVs; identifying sources of bias from a potential algorithmic pipeline which simulates human decision making; and recommending ways to mitigate bias from both aspects. Our research has provided an innovative research design that combines social science and data science to theorise how automation may introduce bias in hiring practices, and also pinpoint where it is introduced. It also furthers the current scholarship on gender bias in hiring practices by providing key empirical inferences on the factors contributing to bias.

Sociology,Computer Science
AI Gender Bias, Disparities, and Fairness: Does Training Data Matter?

Ehsan Latif,Xiaoming Zhai,Lei Liu

2023-12-26

Abstract:This study delves into the pervasive issue of gender issues in artificial intelligence (AI), specifically within automatic scoring systems for student-written responses. The primary objective is to investigate the presence of gender biases, disparities, and fairness in generally targeted training samples with mixed-gender datasets in AI scoring outcomes. Utilizing a fine-tuned version of BERT and GPT-3.5, this research analyzes more than 1000 human-graded student responses from male and female participants across six assessment items. The study employs three distinct techniques for bias analysis: Scoring accuracy difference to evaluate bias, mean score gaps by gender (MSG) to evaluate disparity, and Equalized Odds (EO) to evaluate fairness. The results indicate that scoring accuracy for mixed-trained models shows an insignificant difference from either male- or female-trained models, suggesting no significant scoring bias. Consistently with both BERT and GPT-3.5, we found that mixed-trained models generated fewer MSG and non-disparate predictions compared to humans. In contrast, compared to humans, gender-specifically trained models yielded larger MSG, indicating that unbalanced training data may create algorithmic models to enlarge gender disparities. The EO analysis suggests that mixed-trained models generated more fairness outcomes compared with gender-specifically trained models. Collectively, the findings suggest that gender-unbalanced data do not necessarily generate scoring bias but can enlarge gender disparities and reduce scoring fairness.

Computers and Society
Data quality dimensions for fair AI

Camilla Quaresmini,Giuseppe Primiero

2024-12-05

Abstract:Artificial Intelligence (AI) systems are not intrinsically neutral and biases trickle in any type of technological tool. In particular when dealing with people, the impact of AI algorithms' technical errors originating with mislabeled data is undeniable. As they feed wrong and discriminatory classifications, these systems are not systematically guarded against bias. In this article we consider the problem of bias in AI systems from the point of view of data quality dimensions. We highlight the limited model construction of bias mitigation tools based on accuracy strategy, illustrating potential improvements of a specific tool in gender classification errors occurring in two typically difficult contexts: the classification of non-binary individuals, for which the label set becomes incomplete with respect to the dataset; and the classification of transgender individuals, for which the dataset becomes inconsistent with respect to the label set. Using formal methods for reasoning about the behavior of the classification system in presence of a changing world, we propose to reconsider the fairness of the classification task in terms of completeness, consistency, timeliness and reliability, and offer some theoretical results.

Artificial Intelligence,Logic in Computer Science
Bias and Fairness in Artificial Intelligence: Methods and Mitigation Strategies

Kabir Singh Chadha

DOI: https://doi.org/10.36676/jrps.v15.i3.1425

2024-07-13

Abstract:Artificial intelligence (AI) has quickly evolved from a sci-fi idea to a crucial part of modern technology, impacting a number of industries like healthcare, banking, education, and law enforcement. Fairness and bias issues with AI systems have drawn a lot of attention as they grow increasingly prevalent in everyday life. In artificial intelligence, "bias" refers to the systematic and unjust discrimination against particular groups of individuals. Prejudices in training data or those unintentionally introduced during algorithm development are common examples of bias. Contrarily, fairness is the idea that every person should have equal access to opportunities and treatment regardless of society or personal traits.
Bias detectives: the researchers striving to make algorithms fair

Rachel Courtland

DOI: https://doi.org/10.1038/d41586-018-05469-3

IF: 64.8

2018-06-01

Nature

Abstract:As machine learning infiltrates society, scientists are trying to help ward off injustice.

multidisciplinary sciences
Bias, Fairness, and Accountability with AI and ML Algorithms

Nengfeng Zhou,Zach Zhang,Vijayan N. Nair,Harsh Singhal,Jie Chen,Agus Sudjianto

DOI: https://doi.org/10.48550/arXiv.2105.06558

2021-05-14

Abstract:The advent of AI and ML algorithms has led to opportunities as well as challenges. In this paper, we provide an overview of bias and fairness issues that arise with the use of ML algorithms. We describe the types and sources of data bias, and discuss the nature of algorithmic unfairness. This is followed by a review of fairness metrics in the literature, discussion of their limitations, and a description of de-biasing (or mitigation) techniques in the model life cycle.

Machine Learning

AI can be sexist and racist — it’s time to make it fair

Feminist AI: Can We Expect Our AI Systems to Become Feminist?

Addressing Bias and Fairness Issues in Artificial Intelligence

Bias, Fairness and Accountability with Artificial Intelligence and Machine Learning Algorithms

AI Algorithmic Bias: Understanding its Causes, Ethical and Social Implications

Does Gender Matter? Towards Fairness in Dialogue Systems

AI’s fairness problem: understanding wrongful discrimination in the context of automated decision-making

Oxford Handbook on AI Ethics Book Chapter on Race and Gender

An Introduction to Artificial Intelligence and Solutions to the Problems of Algorithmic Discrimination

Bias in Artificial Intelligence Systems

Does AI Debias Recruitment? Race, Gender, and AI's "Eradication of Difference"

Addressing Bias in Artificial Intelligence in Health Care

Ethics and discrimination in artificial intelligence-enabled recruitment practices

AI and Social Impact

Bias Amplification in Artificial Intelligence Systems

Gender Bias in AI Recruitment Systems: A Sociological-and Data Science-based Case Study

AI Gender Bias, Disparities, and Fairness: Does Training Data Matter?

Data quality dimensions for fair AI

Bias and Fairness in Artificial Intelligence: Methods and Mitigation Strategies

Bias detectives: the researchers striving to make algorithms fair

Bias, Fairness, and Accountability with AI and ML Algorithms