Data Bias Management

Gianluca Demartini,Kevin Roitero,Stefano Mizzaro

DOI: https://doi.org/10.1145/3611641

IF: 22.7

2023-12-21

Communications of the ACM

Abstract:Envisioning a unique approach toward bias and fairness research.

computer science, theory & methods, software engineering, hardware & architecture

What problem does this paper attempt to address?

Managing Bias in Human-Annotated Data: Moving Beyond Bias Removal

Gianluca Demartini,Kevin Roitero,Stefano Mizzaro

DOI: https://doi.org/10.48550/arXiv.2110.13504

2021-10-26

Information Retrieval

Abstract:Due to the widespread use of data-powered systems in our everyday lives, the notions of bias and fairness gained significant attention among researchers and practitioners, in both industry and academia. Such issues typically emerge from the data, which comes with varying levels of quality, used to train systems. With the commercialization and employment of such systems that are sometimes delegated to make life-changing decisions, a significant effort is being made towards the identification and removal of possible sources of bias that may surface to the final end-user. In this position paper, we instead argue that bias is not something that should necessarily be removed in all cases, and the attention and effort should shift from bias removal to the identification, measurement, indexing, surfacing, and adjustment of bias, which we name bias management. We argue that if correctly managed, bias can be a resource that can be made transparent to the the users and empower them to make informed choices about their experience with the system.
Managing bias and unfairness in data for decision support: a survey of machine learning and data engineering approaches to identify and mitigate bias and unfairness within data management and analytics systems

Agathe Balayn,Christoph Lofi,Geert-Jan Houben

DOI: https://doi.org/10.1007/s00778-021-00671-8

2021-05-05

The VLDB Journal

Abstract:Abstract The increasing use of data-driven decision support systems in industry and governments is accompanied by the discovery of a plethora of bias and unfairness issues in the outputs of these systems. Multiple computer science communities, and especially machine learning, have started to tackle this problem, often developing algorithmic solutions to mitigate biases to obtain fairer outputs. However, one of the core underlying causes for unfairness is bias in training data which is not fully covered by such approaches. Especially, bias in data is not yet a central topic in data engineering and management research. We survey research on bias and unfairness in several computer science domains, distinguishing between data management publications and other domains. This covers the creation of fairness metrics, fairness identification, and mitigation methods, software engineering approaches and biases in crowdsourcing activities. We identify relevant research gaps and show which data management activities could be repurposed to handle biases and which ones might reinforce such biases. In the second part, we argue for a novel data-centered approach overcoming the limitations of current algorithmic-centered methods. This approach focuses on eliciting and enforcing fairness requirements and constraints on data that systems are trained, validated, and used on. We argue for the need to extend database management systems to handle such constraints and mitigation methods. We discuss the associated future research directions regarding algorithms, formalization, modelling, users, and systems.

computer science, information systems, hardware & architecture
A Principled Approach for a New Bias Measure

Bruno Scarone,Alfredo Viola,Renée J. Miller,Ricardo Baeza-Yates

2024-09-11

Abstract:The widespread use of machine learning and data-driven algorithms for decision making has been steadily increasing over many years. The areas in which this is happening are diverse: healthcare, employment, finance, education, the legal system to name a few; and the associated negative side effects are being increasingly harmful for society. Negative data \emph{bias} is one of those, which tends to result in harmful consequences for specific groups of people. Any mitigation strategy or effective policy that addresses the negative consequences of bias must start with awareness that bias exists, together with a way to understand and quantify it. However, there is a lack of consensus on how to measure data bias and oftentimes the intended meaning is context dependent and not uniform within the research community. The main contributions of our work are: (1) The definition of Uniform Bias (UB), the first bias measure with a clear and simple interpretation in the full range of bias values. (2) A systematic study to characterize the flaws of existing measures in the context of anti employment discrimination rules used by the Office of Federal Contract Compliance Programs, additionally showing how UB solves open problems in this domain. (3) A framework that provides an efficient way to derive a mathematical formula for a bias measure based on an algorithmic specification of bias addition. Our results are experimentally validated using nine publicly available datasets and theoretically analyzed, which provide novel insights about the problem. Based on our approach, we also design a bias mitigation model that might be useful to policymakers.

Machine Learning,Computers and Society
Bias in data‐driven artificial intelligence systems—An introductory survey

Eirini Ntoutsi,Pavlos Fafalios,Ujwal Gadiraju,Vasileios Iosifidis,Wolfgang Nejdl,Maria‐Esther Vidal,Salvatore Ruggieri,Franco Turini,Symeon Papadopoulos,Emmanouil Krasanakis,Ioannis Kompatsiaris,Katharina Kinder‐Kurlanda,Claudia Wagner,Fariba Karimi,Miriam Fernandez,Harith Alani,Bettina Berendt,Tina Kruegel,Christian Heinze,Klaus Broelemann,Gjergji Kasneci,Thanassis Tiropanis,Steffen Staab

DOI: https://doi.org/10.1002/widm.1356

2020-02-03

WIREs Data Mining and Knowledge Discovery

Abstract:<p>Artificial Intelligence (AI)‐based systems are widely employed nowadays to make decisions that have far‐reaching impact on individuals and society. Their decisions might affect everyone, everywhere, and anytime, entailing concerns about potential human rights issues. Therefore, it is necessary to move beyond traditional AI algorithms optimized for predictive performance and embed ethical and legal principles in their design, training, and deployment to ensure social good while still benefiting from the huge potential of the AI technology. The goal of this survey is to provide a broad multidisciplinary overview of the area of bias in AI systems, focusing on technical challenges and solutions as well as to suggest new research directions towards approaches well‐grounded in a legal frame. In this survey, we focus on data‐driven AI, as a large part of AI is powered nowadays by (big) data and powerful machine learning algorithms. If otherwise not specified, we use the general term bias to describe problems related to the gathering or processing of data that might result in prejudiced decisions on the bases of demographic features such as race, sex, and so forth.</p><p>This article is categorized under: </p><ul class="plain-list"><li>Commercial, Legal, and Ethical Issues > Fairness in Data Mining</li><li>Commercial, Legal, and Ethical Issues > Ethical Considerations</li><li>Commercial, Legal, and Ethical Issues > Legal Issues</li></ul>
Data Management for Causal Algorithmic Fairness

Babak Salimi,Bill Howe,Dan Suciu

DOI: https://doi.org/10.48550/arXiv.1908.07924

2019-08-20

Databases

Abstract:Fairness is increasingly recognized as a critical component of machine learning systems. However, it is the underlying data on which these systems are trained that often reflects discrimination, suggesting a data management problem. In this paper, we first make a distinction between associational and causal definitions of fairness in the literature and argue that the concept of fairness requires causal reasoning. We then review existing works and identify future opportunities for applying data management techniques to causal algorithmic fairness.
Data quality dimensions for fair AI

Camilla Quaresmini,Giuseppe Primiero

2024-12-05

Abstract:Artificial Intelligence (AI) systems are not intrinsically neutral and biases trickle in any type of technological tool. In particular when dealing with people, the impact of AI algorithms' technical errors originating with mislabeled data is undeniable. As they feed wrong and discriminatory classifications, these systems are not systematically guarded against bias. In this article we consider the problem of bias in AI systems from the point of view of data quality dimensions. We highlight the limited model construction of bias mitigation tools based on accuracy strategy, illustrating potential improvements of a specific tool in gender classification errors occurring in two typically difficult contexts: the classification of non-binary individuals, for which the label set becomes incomplete with respect to the dataset; and the classification of transgender individuals, for which the dataset becomes inconsistent with respect to the label set. Using formal methods for reasoning about the behavior of the classification system in presence of a changing world, we propose to reconsider the fairness of the classification task in terms of completeness, consistency, timeliness and reliability, and offer some theoretical results.

Artificial Intelligence,Logic in Computer Science
Towards Assessing Data Bias in Clinical Trials

Chiara Criscuolo,Tommaso Dolci,Mattia Salnitri

DOI: https://doi.org/10.1007/978-3-031-23905-2_5

2022-12-20

Abstract:Algorithms and technologies are essential tools that pervade all aspects of our daily lives. In the last decades, health care research benefited from new computer-based recruiting methods, the use of federated architectures for data storage, the introduction of innovative analyses of datasets, and so on. Nevertheless, health care datasets can still be affected by data bias. Due to data bias, they provide a distorted view of reality, leading to wrong analysis results and, consequently, decisions. For example, in a clinical trial that studied the risk of cardiovascular diseases, predictions were wrong due to the lack of data on ethnic minorities. It is, therefore, of paramount importance for researchers to acknowledge data bias that may be present in the datasets they use, eventually adopt techniques to mitigate them and control if and how analyses results are impacted. This paper proposes a method to address bias in datasets that: (i) defines the types of data bias that may be present in the dataset, (ii) characterizes and quantifies data bias with adequate metrics, (iii) provides guidelines to identify, measure, and mitigate data bias for different data sources. The method we propose is applicable both for prospective and retrospective clinical trials. We evaluate our proposal both through theoretical considerations and through interviews with researchers in the health care environment.

Computers and Society,Artificial Intelligence,Machine Learning,Methodology
AI bias: exploring discriminatory algorithmic decision-making models and the application of possible machine-centric solutions adapted from the pharmaceutical industry

Lorenzo Belenguer

DOI: https://doi.org/10.1007/s43681-022-00138-8

2022-02-10

AI and Ethics

Abstract:A new and unorthodox approach to deal with discriminatory bias in Artificial Intelligence is needed. As it is explored in detail, the current literature is a dichotomy with studies originating from the contrasting fields of study of either philosophy and sociology or data science and programming. It is suggested that there is a need instead for an integration of both academic approaches, and needs to be machine-centric rather than human-centric applied with a deep understanding of societal and individual prejudices. This article is a novel approach developed into a framework of action: a bias impact assessment to raise awareness of bias and why, a clear set of methodologies as shown in a table comparing with the four stages of pharmaceutical trials, and a summary flowchart. Finally, this study concludes the need for a transnational independent body with enough power to guarantee the implementation of those solutions.
AIM: Attributing, Interpreting, Mitigating Data Unfairness

Zhining Liu,Ruizhong Qiu,Zhichen Zeng,Yada Zhu,Hendrik Hamann,Hanghang Tong

DOI: https://doi.org/10.1145/3637528.3671797

2024-06-18

Abstract:Data collected in the real world often encapsulates historical discrimination against disadvantaged groups and individuals. Existing fair machine learning (FairML) research has predominantly focused on mitigating discriminative bias in the model prediction, with far less effort dedicated towards exploring how to trace biases present in the data, despite its importance for the transparency and interpretability of FairML. To fill this gap, we investigate a novel research problem: discovering samples that reflect biases/prejudices from the training data. Grounding on the existing fairness notions, we lay out a sample bias criterion and propose practical algorithms for measuring and countering sample bias. The derived bias score provides intuitive sample-level attribution and explanation of historical bias in data. On this basis, we further design two FairML strategies via sample-bias-informed minimal data editing. They can mitigate both group and individual unfairness at the cost of minimal or zero predictive utility loss. Extensive experiments and analyses on multiple real-world datasets demonstrate the effectiveness of our methods in explaining and mitigating unfairness. Code is available at <a class="link-external link-https" href="https://github.com/ZhiningLiu1998/AIM" rel="external noopener nofollow">this https URL</a>.

Machine Learning,Artificial Intelligence
A Survey on Bias and Fairness in Machine Learning

Ninareh Mehrabi,Fred Morstatter,Nripsuta Saxena,Kristina Lerman,Aram Galstyan

DOI: https://doi.org/10.1145/3457607

IF: 16.6

2021-07-01

ACM Computing Surveys

Abstract:With the widespread use of artificial intelligence (AI) systems and applications in our everyday lives, accounting for fairness has gained significant importance in designing and engineering of such systems. AI systems can be used in many sensitive environments to make important and life-changing decisions; thus, it is crucial to ensure that these decisions do not reflect discriminatory behavior toward certain groups or populations. More recently some work has been developed in traditional machine learning and deep learning that address such challenges in different subdomains. With the commercialization of these systems, researchers are becoming more aware of the biases that these applications can contain and are attempting to address them. In this survey, we investigated different real-world applications that have shown biases in various ways, and we listed different sources of biases that can affect AI applications. We then created a taxonomy for fairness definitions that machine learning researchers have defined to avoid the existing bias in AI systems. In addition to that, we examined different domains and subdomains in AI showing what researchers have observed with regard to unfair outcomes in the state-of-the-art methods and ways they have tried to address them. There are still many future directions and solutions that can be taken to mitigate the problem of bias in AI systems. We are hoping that this survey will motivate researchers to tackle these issues in the near future by observing existing work in their respective fields.

computer science, theory & methods
Understanding Bias in Machine Learning

Jindong Gu,Daniela Oelke

DOI: https://doi.org/10.48550/arXiv.1909.01866

IF: 5.414

2019-09-02

Machine Learning

Abstract:Bias is known to be an impediment to fair decisions in many domains such as human resources, the public sector, health care etc. Recently, hope has been expressed that the use of machine learning methods for taking such decisions would diminish or even resolve the problem. At the same time, machine learning experts warn that machine learning models can be biased as well. In this article, our goal is to explain the issue of bias in machine learning from a technical perspective and to illustrate the impact that biased data can have on a machine learning model. To reach such a goal, we develop interactive plots to visualizing the bias learned from synthetic data.
Hands on Data and Algorithmic Bias in Recommender Systems

Ludovico Boratto,Mirko Marras

DOI: https://doi.org/10.1145/3340631.3398669

2020-07-07

Abstract:This tutorial provides a common ground for both researchers and practitioners interested in data and algorithmic bias in recommender systems. Guided by real-world examples in various domains, we introduce problem space and concepts underlying bias investigation in recommendation. Then, we practically show two use cases, addressing biases that lead to disparate exposure of items based on their popularity and to systematically discriminate against a legally-protected class of users. Finally, we cover a range of techniques for evaluating and mitigating the impact of these biases on the recommended lists, including pre-, in-, and post-processing procedures. This tutorial is accompanied by Jupyter notebooks putting into practice core concepts in data from real-world platforms.
Big Data, Data Science, and Civil Rights

Solon Barocas,Elizabeth Bradley,Vasant Honavar,Foster Provost

DOI: https://doi.org/10.48550/arXiv.1706.03102

2017-06-10

Abstract:Advances in data analytics bring with them civil rights implications. Data-driven and algorithmic decision making increasingly determine how businesses target advertisements to consumers, how police departments monitor individuals or groups, how banks decide who gets a loan and who does not, how employers hire, how colleges and universities make admissions and financial aid decisions, and much more. As data-driven decisions increasingly affect every corner of our lives, there is an urgent need to ensure they do not become instruments of discrimination, barriers to equality, threats to social justice, and sources of unfairness. In this paper, we argue for a concrete research agenda aimed at addressing these concerns, comprising five areas of emphasis: (i) Determining if models and modeling procedures exhibit objectionable bias; (ii) Building awareness of fairness into machine learning methods; (iii) Improving the transparency and control of data- and model-driven decision making; (iv) Looking beyond the algorithm(s) for sources of bias and unfairness-in the myriad human decisions made during the problem formulation and modeling process; and (v) Supporting the cross-disciplinary scholarship necessary to do all of that well.

Computers and Society
Studying Up Machine Learning Data: Why Talk About Bias When We Mean Power?

Milagros Miceli,Julian Posada,Tianling Yang

DOI: https://doi.org/10.48550/arXiv.2109.08131

2021-09-17

Abstract:Research in machine learning (ML) has primarily argued that models trained on incomplete or biased datasets can lead to discriminatory outputs. In this commentary, we propose moving the research focus beyond bias-oriented framings by adopting a power-aware perspective to "study up" ML datasets. This means accounting for historical inequities, labor conditions, and epistemological standpoints inscribed in data. We draw on HCI and CSCW work to support our argument, critically analyze previous research, and point at two co-existing lines of work within our community -- one bias-oriented, the other power-aware. This way, we highlight the need for dialogue and cooperation in three areas: data quality, data work, and data documentation. In the first area, we argue that reducing societal problems to "bias" misses the context-based nature of data. In the second one, we highlight the corporate forces and market imperatives involved in the labor of data workers that subsequently shape ML datasets. Finally, we propose expanding current transparency-oriented efforts in dataset documentation to reflect the social contexts of data design and production.

Human-Computer Interaction,Computers and Society,Machine Learning
Bias-inducing geometries: an exactly solvable data model with fairness implications

Stefano Sarao Mannelli,Federica Gerace,Negar Rostamzadeh,Luca Saglietti

2023-11-13

Abstract:Machine learning (ML) may be oblivious to human bias but it is not immune to its perpetuation. Marginalisation and iniquitous group representation are often traceable in the very data used for training, and may be reflected or even enhanced by the learning models. In the present work, we aim at clarifying the role played by data geometry in the emergence of ML bias. We introduce an exactly solvable high-dimensional model of data imbalance, where parametric control over the many bias-inducing factors allows for an extensive exploration of the bias inheritance mechanism. Through the tools of statistical physics, we analytically characterise the typical properties of learning models trained in this synthetic framework and obtain exact predictions for the observables that are commonly employed for fairness assessment. Despite the simplicity of the data model, we retrace and unpack typical unfairness behaviour observed on real-world datasets. We also obtain a detailed analytical characterisation of a class of bias mitigation strategies. We first consider a basic loss-reweighing scheme, which allows for an implicit minimisation of different unfairness metrics, and quantify the incompatibilities between some existing fairness criteria. Then, we consider a novel mitigation strategy based on a matched inference approach, consisting in the introduction of coupled learning models. Our theoretical analysis of this approach shows that the coupled strategy can strike superior fairness-accuracy trade-offs.

Machine Learning,Disordered Systems and Neural Networks
Addressing Bias and Fairness Issues in Artificial Intelligence

Pragya Gupta

Abstract:-This paper explores the multifaceted aspects of bias in AI, encompassing its types, sources, implications, and mitigation strategies. Drawing on real-world examples, it delves into reporting bias, selection bias, group attribution bias, and implicit bias, highlighting their impact on societal inequalities and marginalized groups. The reinforcement of historical biases in AI training data perpetuates discrimination and hampers progress towards equality. The paper discusses quantitative measures like disparate impact and demographic parity, while emphasizing the vital role of qualitative assessments and human evaluators in identifying bias. Furthermore, it explores strategies for addressing bias, such as diverse training data, in-processing and post-processing models, and algorithmic debiasing techniques. The article also underscores interdisciplinary collaboration, ethical considerations, and regulatory standards as essential components of building fair and accountable AI systems. Lastly, it outlines future directions for research, including adaptive algorithms, intersectional fairness, inclusive development, and robust ethical frameworks, aiming to guide AI towards equitable and responsible advancement.

Political Science,Sociology,Computer Science
Leveraging Ontologies to Document Bias in Data

Mayra Russo,Maria-Esther Vidal

2024-06-30

Abstract:Machine Learning (ML) systems are capable of reproducing and often amplifying undesired biases. This puts emphasis on the importance of operating under practices that enable the study and understanding of the intrinsic characteristics of ML pipelines, prompting the emergence of documentation frameworks with the idea that ``any remedy for bias starts with awareness of its existence''. However, a resource that can formally describe these pipelines in terms of biases detected is still amiss. To fill this gap, we present the Doc-BiasO ontology, a resource that aims to create an integrated vocabulary of biases defined in the \textit{fair-ML} literature and their measures, as well as to incorporate relevant terminology and the relationships between them. Overseeing ontology engineering best practices, we re-use existing vocabulary on machine learning and AI, to foster knowledge sharing and interoperability between the actors concerned with its research, development, regulation, among others. Overall, our main objective is to contribute towards clarifying existing terminology on bias research as it rapidly expands to all areas of AI and to improve the interpretation of bias in data and downstream impact.

Artificial Intelligence
Preliminary Insights on Industry Practices for Addressing Fairness Debt

Ronnie de Souza Santos,Luiz Fernando de Lima,Maria Teresa Baldassarre,Rodrigo Spinola

2024-09-04

Abstract:Context: This study explores how software professionals identify and address biases in AI systems within the software industry, focusing on practical knowledge and real-world applications. Goal: We aimed to understand the strategies employed by practitioners to manage bias and their implications for fairness debt. Method: We used a qualitative research method, gathering insights from industry professionals through interviews and employing thematic analysis to explore the collected data. Findings: Professionals identify biases through discrepancies in model outputs, demographic inconsistencies, and issues with training data. They address these biases using strategies such as enhanced data management, model adjustments, crisis management, improving team diversity, and ethical analysis. Conclusion: Our paper presents initial evidence on addressing fairness debt and provides a foundation for developing structured guidelines to manage fairness-related issues in AI systems.

Software Engineering
A Comprehensive Empirical Study of Bias Mitigation Methods for Machine Learning Classifiers

Zhenpeng Chen,Jie M. Zhang,Federica Sarro,Mark Harman

DOI: https://doi.org/10.48550/arXiv.2207.03277

2023-02-11

Abstract:Software bias is an increasingly important operational concern for software engineers. We present a large-scale, comprehensive empirical study of 17 representative bias mitigation methods for Machine Learning (ML) classifiers, evaluated with 11 ML performance metrics (e.g., accuracy), 4 fairness metrics, and 20 types of fairness-performance trade-off assessment, applied to 8 widely-adopted software decision tasks. The empirical coverage is much more comprehensive, covering the largest numbers of bias mitigation methods, evaluation metrics, and fairness-performance trade-off measures compared to previous work on this important software property. We find that (1) the bias mitigation methods significantly decrease ML performance in 53% of the studied scenarios (ranging between 42%~66% according to different ML performance metrics); (2) the bias mitigation methods significantly improve fairness measured by the 4 used metrics in 46% of all the scenarios (ranging between 24%~59% according to different fairness metrics); (3) the bias mitigation methods even lead to decrease in both fairness and ML performance in 25% of the scenarios; (4) the effectiveness of the bias mitigation methods depends on tasks, models, the choice of protected attributes, and the set of metrics used to assess fairness and ML performance; (5) there is no bias mitigation method that can achieve the best trade-off in all the scenarios. The best method that we find outperforms other methods in 30% of the scenarios. Researchers and practitioners need to choose the bias mitigation method best suited to their intended application scenario(s).

Software Engineering,Artificial Intelligence

Data Bias Management

Managing Bias in Human-Annotated Data: Moving Beyond Bias Removal

Managing bias and unfairness in data for decision support: a survey of machine learning and data engineering approaches to identify and mitigate bias and unfairness within data management and analytics systems

A Principled Approach for a New Bias Measure

Bias in data‐driven artificial intelligence systems—An introductory survey

Data Management for Causal Algorithmic Fairness

Data quality dimensions for fair AI

Towards Assessing Data Bias in Clinical Trials

AI bias: exploring discriminatory algorithmic decision-making models and the application of possible machine-centric solutions adapted from the pharmaceutical industry

AIM: Attributing, Interpreting, Mitigating Data Unfairness

A Survey on Bias and Fairness in Machine Learning

Understanding Bias in Machine Learning

Hands on Data and Algorithmic Bias in Recommender Systems

Big Data, Data Science, and Civil Rights

Studying Up Machine Learning Data: Why Talk About Bias When We Mean Power?

Bias-inducing geometries: an exactly solvable data model with fairness implications

Addressing Bias and Fairness Issues in Artificial Intelligence

Leveraging Ontologies to Document Bias in Data

Preliminary Insights on Industry Practices for Addressing Fairness Debt

A Comprehensive Empirical Study of Bias Mitigation Methods for Machine Learning Classifiers