Bias detectives: the researchers striving to make algorithms fair

Rachel Courtland

DOI: https://doi.org/10.1038/d41586-018-05469-3

IF: 64.8

2018-06-01

Nature

Abstract:As machine learning infiltrates society, scientists are trying to help ward off injustice.

multidisciplinary sciences

What problem does this paper attempt to address?

Fairness Deconstructed: A Sociotechnical View of 'Fair' Algorithms in Criminal Justice

Rajiv Movva

DOI: https://doi.org/10.48550/arXiv.2106.13455

2021-06-25

Computers and Society

Abstract:Early studies of risk assessment algorithms used in criminal justice revealed widespread racial biases. In response, machine learning researchers have developed methods for fairness, many of which rely on equalizing empirical metrics across protected attributes. Here, I recall sociotechnical perspectives to delineate the significant gap between fairness in theory and practice, focusing on criminal justice. I (1) illustrate how social context can undermine analyses that are restricted to an AI system's outputs, and (2) argue that much of the fair ML literature fails to account for epistemological issues with underlying crime data. Instead of building AI that reifies power imbalances, like risk assessment algorithms, I ask whether data science can be used to understand the root causes of structural marginalization.
Fairness-aware machine learning: a perspective

Indre Zliobaite

DOI: https://doi.org/10.48550/arXiv.1708.00754

2017-08-02

Abstract:Algorithms learned from data are increasingly used for deciding many aspects in our life: from movies we see, to prices we pay, or medicine we get. Yet there is growing evidence that decision making by inappropriately trained algorithms may unintentionally discriminate people. For example, in automated matching of candidate CVs with job descriptions, algorithms may capture and propagate ethnicity related biases. Several repairs for selected algorithms have already been proposed, but the underlying mechanisms how such discrimination happens from the computational perspective are not yet scientifically understood. We need to develop theoretical understanding how algorithms may become discriminatory, and establish fundamental machine learning principles for prevention. We need to analyze machine learning process as a whole to systematically explain the roots of discrimination occurrence, which will allow to devise global machine learning optimization criteria for guaranteed prevention, as opposed to pushing empirical constraints into existing algorithms case-by-case. As a result, the state-of-the-art will advance from heuristic repairing, to proactive and theoretically supported prevention. This is needed not only because law requires to protect vulnerable people. Penetration of big data initiatives will only increase, and computer science needs to provide solid explanations and accountability to the public, before public concerns lead to unnecessarily restrictive regulations against machine learning.

Artificial Intelligence,Computers and Society,Machine Learning
Bias and Fairness in Computer Vision Applications of the Criminal Justice System

Sophie Noiret,Jennifer Lumetzberger,Martin Kampel

DOI: https://doi.org/10.48550/arXiv.2208.03209

2022-08-05

Computer Vision and Pattern Recognition

Abstract:Discriminatory practices involving AI-driven police work have been the subject of much controversies in the past few years, with algorithms such as COMPAS, PredPol and ShotSpotter being accused of unfairly impacting minority groups. At the same time, the issues of fairness in machine learning, and in particular in computer vision, have been the subject of a growing number of academic works. In this paper, we examine how these area intersect. We provide information on how these practices have come to exist and the difficulties in alleviating them. We then examine three applications currently in development to understand what risks they pose to fairness and how those risks can be mitigated.
Fairway: SE Principles for Building Fairer Software

Joymallya Chakraborty,Suvodeep Majumder,Zhe Wu,Tim Menzies

2020-01-01

Abstract:Machine learning software is increasingly being used to make decisions that affect people's lives. But sometimes, the core part of this software (the learned model), behaves in a biased manner that gives undue advantages to a specific group of people (where those groups are determined by sex, race, etc.). This"algorithmic discrimination"in the AI software systems has become a matter of serious concern in the machine learning and software engineering community. There have been works done to find"algorithmic bias"or"ethical bias"in the software system. Once the bias is detected in the AI software system, the mitigation of bias is extremely important. In this work, we a)explain how ground-truth bias in training data affects machine learning model fairness and how to find that bias in AI software,b)propose a methodFairwaywhich combines pre-processing and in-processing approach to remove ethical bias from training data and trained model. Our results show that we can find bias and mitigate bias in a learned model, without much damaging the predictive performance of that model. We propose that (1) test-ing for bias and (2) bias mitigation should be a routine part of the machine learning software development life cycle. Fairway offers much support for these two purposes.
The Technology of Outrage: Bias in Artificial Intelligence

Will Bridewell,Paul F. Bello,Selmer Bringsjord

2024-09-26

Abstract:Artificial intelligence and machine learning are increasingly used to offload decision making from people. In the past, one of the rationales for this replacement was that machines, unlike people, can be fair and unbiased. Evidence suggests otherwise. We begin by entertaining the ideas that algorithms can replace people and that algorithms cannot be biased. Taken as axioms, these statements quickly lead to absurdity. Spurred on by this result, we investigate the slogans more closely and identify equivocation surrounding the word 'bias.' We diagnose three forms of outrage-intellectual, moral, and political-that are at play when people react emotionally to algorithmic bias. Then we suggest three practical approaches to addressing bias that the AI community could take, which include clarifying the language around bias, developing new auditing methods for intelligent systems, and building certain capabilities into these systems. We conclude by offering a moral regarding the conversations about algorithmic bias that may transfer to other areas of artificial intelligence.

Computers and Society,Artificial Intelligence
Designing Equitable Algorithms

Alex Chohlas-Wood,Madison Coots,Sharad Goel,Julian Nyarko

DOI: https://doi.org/10.48550/arXiv.2302.09157

2023-02-18

Abstract:Predictive algorithms are now used to help distribute a large share of our society's resources and sanctions, such as healthcare, loans, criminal detentions, and tax audits. Under the right circumstances, these algorithms can improve the efficiency and equity of decision-making. At the same time, there is a danger that the algorithms themselves could entrench and exacerbate disparities, particularly along racial, ethnic, and gender lines. To help ensure their fairness, many researchers suggest that algorithms be subject to at least one of three constraints: (1) no use of legally protected features, such as race, ethnicity, and gender; (2) equal rates of "positive" decisions across groups; and (3) equal error rates across groups. Here we show that these constraints, while intuitively appealing, often worsen outcomes for individuals in marginalized groups, and can even leave all groups worse off. The inherent trade-off we identify between formal fairness constraints and welfare improvements -- particularly for the marginalized -- highlights the need for a more robust discussion on what it means for an algorithm to be "fair". We illustrate these ideas with examples from healthcare and the criminal-legal system, and make several proposals to help practitioners design more equitable algorithms.

Machine Learning,Computers and Society
AI’s fairness problem: understanding wrongful discrimination in the context of automated decision-making

Hugo Cossette-Lefebvre,Jocelyn Maclure

DOI: https://doi.org/10.1007/s43681-022-00233-w

2022-11-16

AI and Ethics

Abstract:Abstract The use of predictive machine learning algorithms is increasingly common to guide or even take decisions in both public and private settings. Their use is touted by some as a potentially useful method to avoid discriminatory decisions since they are, allegedly, neutral, objective, and can be evaluated in ways no human decisions can. By (fully or partly) outsourcing a decision process to an algorithm, it should allow human organizations to clearly define the parameters of the decision and to, in principle, remove human biases. Yet, in practice, the use of algorithms can still be the source of wrongful discriminatory decisions based on at least three of their features: the data-mining process and the categorizations they rely on can reconduct human biases, their automaticity and predictive design can lead them to rely on wrongful generalizations, and their opaque nature is at odds with democratic requirements. We highlight that the two latter aspects of algorithms and their significance for discrimination are too often overlooked in contemporary literature. Though these problems are not all insurmountable, we argue that it is necessary to clearly define the conditions under which a machine learning decision tool can be used. We identify and propose three main guidelines to properly constrain the deployment of machine learning algorithms in society: algorithms should be vetted to ensure that they do not unduly affect historically marginalized groups; they should not systematically override or replace human decision-making processes; and the decision reached using an algorithm should always be explainable and justifiable.
Algorithmic bias: on the implicit biases of social technology

Gabbrielle M. Johnson

DOI: https://doi.org/10.1007/s11229-020-02696-y

IF: 1.595

2020-06-20

Synthese

Abstract:Often machine learning programs inherit social patterns reflected in their training data without any directed effort by programmers to include such biases. Computer scientists call this algorithmic bias. This paper explores the relationship between machine bias and human cognitive bias. In it, I argue similarities between algorithmic and cognitive biases indicate a disconcerting sense in which sources of bias emerge out of seemingly innocuous patterns of information processing. The emergent nature of this bias obscures the existence of the bias itself, making it difficult to identify, mitigate, or evaluate using standard resources in epistemology and ethics. I demonstrate these points in the case of mitigation techniques by presenting what I call 'the Proxy Problem'. One reason biases resist revision is that they rely on proxy attributes, seemingly innocuous attributes that correlate with socially-sensitive attributes, serving as proxies for the socially-sensitive attributes themselves. I argue that in both human and algorithmic domains, this problem presents a common dilemma for mitigation: attempts to discourage reliance on proxy attributes risk a tradeoff with judgement accuracy. This problem, I contend, admits of no purely algorithmic solution.

history & philosophy of science
Fairness and Bias in Algorithmic Hiring: a Multidisciplinary Survey

Alessandro Fabris,Nina Baranowska,Matthew J. Dennis,David Graus,Philipp Hacker,Jorge Saldivar,Frederik Zuiderveen Borgesius,Asia J. Biega

DOI: https://doi.org/10.1145/3696457

2024-09-25

Abstract:Employers are adopting algorithmic hiring technology throughout the recruitment pipeline. Algorithmic fairness is especially applicable in this domain due to its high stakes and structural inequalities. Unfortunately, most work in this space provides partial treatment, often constrained by two competing narratives, optimistically focused on replacing biased recruiter decisions or pessimistically pointing to the automation of discrimination. Whether, and more importantly what types of, algorithmic hiring can be less biased and more beneficial to society than low-tech alternatives currently remains unanswered, to the detriment of trustworthiness. This multidisciplinary survey caters to practitioners and researchers with a balanced and integrated coverage of systems, biases, measures, mitigation strategies, datasets, and legal aspects of algorithmic hiring and fairness. Our work supports a contextualized understanding and governance of this technology by highlighting current opportunities and limitations, providing recommendations for future work to ensure shared benefits for all stakeholders.

Computers and Society,Artificial Intelligence
Bias, Fairness and Accountability with Artificial Intelligence and Machine Learning Algorithms

Nengfeng Zhou,Zach Zhang,Vijayan N. Nair,Harsh Singhal,Jie Chen

DOI: https://doi.org/10.1111/insr.12492

2022-04-10

International Statistical Review

Abstract:SummaryThe advent of artificial intelligence (AI) and machine learning algorithms has led to opportunities as well as challenges in their use. In this overview paper, we begin with a discussion of bias and fairness issues that arise with the use of AI techniques, with a focus on supervised machine learning algorithms. We then describe the types and sources of data bias and discuss the nature of algorithmic unfairness. In addition, we provide a review of fairness metrics in the literature, discuss their limitations, and describe de‐biasing (or mitigation) techniques in the model life cycle.

statistics & probability
Human Perceptions of Fairness in Algorithmic Decision Making: A Case Study of Criminal Risk Prediction

Nina Grgić-Hlača,Elissa M. Redmiles,Krishna P. Gummadi,Adrian Weller

DOI: https://doi.org/10.48550/arXiv.1802.09548

2018-02-27

Abstract:As algorithms are increasingly used to make important decisions that affect human lives, ranging from social benefit assignment to predicting risk of criminal recidivism, concerns have been raised about the fairness of algorithmic decision making. Most prior works on algorithmic fairness normatively prescribe how fair decisions ought to be made. In contrast, here, we descriptively survey users for how they perceive and reason about fairness in algorithmic decision making. A key contribution of this work is the framework we propose to understand why people perceive certain features as fair or unfair to be used in algorithms. Our framework identifies eight properties of features, such as relevance, volitionality and reliability, as latent considerations that inform people's moral judgments about the fairness of feature use in decision-making algorithms. We validate our framework through a series of scenario-based surveys with 576 people. We find that, based on a person's assessment of the eight latent properties of a feature in our exemplar scenario, we can accurately (> 85%) predict if the person will judge the use of the feature as fair. Our findings have important implications. At a high-level, we show that people's unfairness concerns are multi-dimensional and argue that future studies need to address unfairness concerns beyond discrimination. At a low-level, we find considerable disagreements in people's fairness judgments. We identify root causes of the disagreements, and note possible pathways to resolve them.

Machine Learning,Computers and Society
AI Algorithmic Bias: Understanding its Causes, Ethical and Social Implications

Lakshitha R Jain,Vineetha Menon

DOI: https://doi.org/10.1109/ICTAI59109.2023.00073

2023-11-06

Abstract:The escalating usage of artificial intelligence (AI) and machine learning algorithms across diverse fields has prompted apprehension regarding the propagation of algorithmic bias, which may exacerbate instances of discrimination and inequality. Algorithmic bias in AI and machine learning (ML) techniques manifests in real-world applications as a result of either insufficient data variation or augmentation availability in the AI/ML training data, or a flawed learning policy. This leads to the accidental propagation of AI bias as an unjust treatment of particular groups of individuals, owing to their race, gender [1], age, or other distinguishing attributes in practical applications. This paper offers a comprehensive analysis of algorithmic bias, encompassing its origins, ethical and social ramifications, and possible remediations. In addition, this paper introduces an innovative methodology for identifying and measuring algorithmic bias that integrates statistical analysis with input from users and domain specialists. This exposition examines distinct forms of algorithmic biases, such as selection bias, confirmation bias, and measurement bias, and examines underlying catalysts for algorithmic bias, encompassing data integrity concerns, decisions regarding algorithmic design, and institutional prejudgments. The adverse ramifications of algorithmic bias, including the perpetuation of social inequality and the impeding of societal advancement, are the focus of our examination. The present study seeks to make a contribution to the advancement of impartial [2] and equitable AI systems with the potential to foster societal progress and benefit individuals across diverse demographics by identifying the sources and repercussions of algorithmic bias and recommending efficacious interventions.

Sociology,Computer Science
A Survey on Bias and Fairness in Machine Learning

Ninareh Mehrabi,Fred Morstatter,Nripsuta Saxena,Kristina Lerman,Aram Galstyan

DOI: https://doi.org/10.1145/3457607

IF: 16.6

2021-07-01

ACM Computing Surveys

Abstract:With the widespread use of artificial intelligence (AI) systems and applications in our everyday lives, accounting for fairness has gained significant importance in designing and engineering of such systems. AI systems can be used in many sensitive environments to make important and life-changing decisions; thus, it is crucial to ensure that these decisions do not reflect discriminatory behavior toward certain groups or populations. More recently some work has been developed in traditional machine learning and deep learning that address such challenges in different subdomains. With the commercialization of these systems, researchers are becoming more aware of the biases that these applications can contain and are attempting to address them. In this survey, we investigated different real-world applications that have shown biases in various ways, and we listed different sources of biases that can affect AI applications. We then created a taxonomy for fairness definitions that machine learning researchers have defined to avoid the existing bias in AI systems. In addition to that, we examined different domains and subdomains in AI showing what researchers have observed with regard to unfair outcomes in the state-of-the-art methods and ways they have tried to address them. There are still many future directions and solutions that can be taken to mitigate the problem of bias in AI systems. We are hoping that this survey will motivate researchers to tackle these issues in the near future by observing existing work in their respective fields.

computer science, theory & methods
AI can be sexist and racist — it’s time to make it fair

James Zou,Londa Schiebinger

DOI: https://doi.org/10.1038/d41586-018-05707-8

IF: 64.8

2018-07-01

Nature

Abstract:Computer scientists must identify sources of bias, de-bias training data and develop artificial-intelligence algorithms that are robust to skews in the data, argue James Zou and Londa Schiebinger.

multidisciplinary sciences
Discrimination in the Age of Algorithms

Jon Kleinberg,Jens Ludwig,Sendhil Mullainathan,Cass R. Sunstein

DOI: https://doi.org/10.48550/arXiv.1902.03731

2019-02-11

Abstract:The law forbids discrimination. But the ambiguity of human decision-making often makes it extraordinarily hard for the legal system to know whether anyone has actually discriminated. To understand how algorithms affect discrimination, we must therefore also understand how they affect the problem of detecting discrimination. By one measure, algorithms are fundamentally opaque, not just cognitively but even mathematically. Yet for the task of proving discrimination, processes involving algorithms can provide crucial forms of transparency that are otherwise unavailable. These benefits do not happen automatically. But with appropriate requirements in place, the use of algorithms will make it possible to more easily examine and interrogate the entire decision process, thereby making it far easier to know whether discrimination has occurred. By forcing a new level of specificity, the use of algorithms also highlights, and makes transparent, central tradeoffs among competing values. Algorithms are not only a threat to be regulated; with the right safeguards in place, they have the potential to be a positive force for equity.

Computers and Society,Artificial Intelligence,Machine Learning
Addressing Bias in Artificial Intelligence in Health Care

Ravi B. Parikh,Stephanie Teeple,Amol S. Navathe

DOI: https://doi.org/10.1001/jama.2019.18058

IF: 11.816

2019-12-24

JAMA

Abstract:Recent scrutiny of artificial intelligence (AI)–based facial recognition software has renewed concerns about the unintended effects of AI on social bias and inequity. Academic and government officials have raised concerns over racial and gender bias in several AI-based technologies, including internet search engines and algorithms to predict risk of criminal behavior. Companies like IBM and Microsoft have made public commitments to "de-bias" their technologies, whereas Amazon mounted a public campaign criticizing such research. As AI applications gain traction in medicine, clinicians and health system leaders have raised similar concerns over automating and propagating existing biases.<sup><a class="ref-link section-jump-link" href="#jvp190155r1">1</a></sup>

medicine, general & internal
Bias, Fairness, and Accountability with AI and ML Algorithms

Nengfeng Zhou,Zach Zhang,Vijayan N. Nair,Harsh Singhal,Jie Chen,Agus Sudjianto

DOI: https://doi.org/10.48550/arXiv.2105.06558

2021-05-14

Abstract:The advent of AI and ML algorithms has led to opportunities as well as challenges. In this paper, we provide an overview of bias and fairness issues that arise with the use of ML algorithms. We describe the types and sources of data bias, and discuss the nature of algorithmic unfairness. This is followed by a review of fairness metrics in the literature, discussion of their limitations, and a description of de-biasing (or mitigation) techniques in the model life cycle.

Machine Learning
Bias and Fairness in Artificial Intelligence: Methods and Mitigation Strategies

Kabir Singh Chadha

DOI: https://doi.org/10.36676/jrps.v15.i3.1425

2024-07-13

Abstract:Artificial intelligence (AI) has quickly evolved from a sci-fi idea to a crucial part of modern technology, impacting a number of industries like healthcare, banking, education, and law enforcement. Fairness and bias issues with AI systems have drawn a lot of attention as they grow increasingly prevalent in everyday life. In artificial intelligence, "bias" refers to the systematic and unjust discrimination against particular groups of individuals. Prejudices in training data or those unintentionally introduced during algorithm development are common examples of bias. Contrarily, fairness is the idea that every person should have equal access to opportunities and treatment regardless of society or personal traits.
A Review on Fairness in Machine Learning

Dana Pessach,Erez Shmueli

DOI: https://doi.org/10.1145/3494672

IF: 16.6

2023-04-30

ACM Computing Surveys

Abstract:An increasing number of decisions regarding the daily lives of human beings are being controlled by artificial intelligence and machine learning (ML) algorithms in spheres ranging from healthcare, transportation, and education to college admissions, recruitment, provision of loans, and many more realms. Since they now touch on many aspects of our lives, it is crucial to develop ML algorithms that are not only accurate but also objective and fair. Recent studies have shown that algorithmic decision making may be inherently prone to unfairness, even when there is no intention for it. This article presents an overview of the main concepts of identifying, measuring, and improving algorithmic fairness when using ML algorithms, focusing primarily on classification tasks. The article begins by discussing the causes of algorithmic bias and unfairness and the common definitions and measures for fairness. Fairness-enhancing mechanisms are then reviewed and divided into pre-process, in-process, and post-process mechanisms. A comprehensive comparison of the mechanisms is then conducted, toward a better understanding of which mechanisms should be used in different scenarios. The article ends by reviewing several emerging research sub-fields of algorithmic fairness, beyond classification.

computer science, theory & methods
Algorithmic bias: Looking beyond data bias to ensure algorithmic accountability and equity

Savannah Thais,Hannah Shumway,Austin Saragih

DOI: https://doi.org/10.38105/spr.5lwvw66ssy

2023-08-31

MIT Science Policy Review

Abstract:As algorithms increasingly aid public sector decision making in the United States, it becomes important to understand how to effectively tackle algorithmic bias in systems that local, state, and federal government entities use and procure, including what kinds of policies are currently in place or proposed. There is a prevalent belief that algorithmic bias arises primarily from statistical biases present in the data used to train or develop the algorithm, but bias may also arise during data collection, problem specification, how and where algorithms are deployed, and within the broader societal contextualization of algorithms. So far, enacted policies in in the US center on temporary bans of particular types of algorithms, transparency, and post-hoc bias audits, as well as more wide-ranging (but non-binding) policy frameworks; all largely focus on quantitative notions of fairness when they assess bias, leaving room for more comprehensive legislation to meaningfully address this issue going forward.

Bias detectives: the researchers striving to make algorithms fair

Fairness Deconstructed: A Sociotechnical View of 'Fair' Algorithms in Criminal Justice

Fairness-aware machine learning: a perspective

Bias and Fairness in Computer Vision Applications of the Criminal Justice System

Fairway: SE Principles for Building Fairer Software

The Technology of Outrage: Bias in Artificial Intelligence

Designing Equitable Algorithms

AI’s fairness problem: understanding wrongful discrimination in the context of automated decision-making

Algorithmic bias: on the implicit biases of social technology

Fairness and Bias in Algorithmic Hiring: a Multidisciplinary Survey

Bias, Fairness and Accountability with Artificial Intelligence and Machine Learning Algorithms

Human Perceptions of Fairness in Algorithmic Decision Making: A Case Study of Criminal Risk Prediction

AI Algorithmic Bias: Understanding its Causes, Ethical and Social Implications

A Survey on Bias and Fairness in Machine Learning

AI can be sexist and racist — it’s time to make it fair

Discrimination in the Age of Algorithms

Addressing Bias in Artificial Intelligence in Health Care

Bias, Fairness, and Accountability with AI and ML Algorithms

Bias and Fairness in Artificial Intelligence: Methods and Mitigation Strategies

A Review on Fairness in Machine Learning

Algorithmic bias: Looking beyond data bias to ensure algorithmic accountability and equity