Fluorosteroids. V. Preparation and biological activity of 6α-fluoro-17β-carboxylic acids of the androstane series

T. Gusarova,G. S. Grinenko,M. L. Gerasimova,A. Terekhina,G. Gritsina,O. S. Anisimova,G. A. Klubnichkina,V. M. Ryzhkova

DOI: https://doi.org/10.1007/BF00773106

1980-07-01

Pharmaceutical Chemistry Journal

Abstract:

What problem does this paper attempt to address?

Data vs. Model Machine Learning Fairness Testing: An Empirical Study

Arumoy Shome,Luis Cruz,Arie van Deursen

2024-01-15

Abstract:Although several fairness definitions and bias mitigation techniques exist in the literature, all existing solutions evaluate fairness of Machine Learning (ML) systems after the training stage. In this paper, we take the first steps towards evaluating a more holistic approach by testing for fairness both before and after model training. We evaluate the effectiveness of the proposed approach and position it within the ML development lifecycle, using an empirical analysis of the relationship between model dependent and independent fairness metrics. The study uses 2 fairness metrics, 4 ML algorithms, 5 real-world datasets and 1600 fairness evaluation cycles. We find a linear relationship between data and model fairness metrics when the distribution and the size of the training data changes. Our results indicate that testing for fairness prior to training can be a ``cheap'' and effective means of catching a biased data collection process early; detecting data drifts in production systems and minimising execution of full training cycles thus reducing development time and costs.

Machine Learning,Computers and Society,Software Engineering
A Review on Fairness in Machine Learning

Dana Pessach,Erez Shmueli

DOI: https://doi.org/10.1145/3494672

IF: 16.6

2023-04-30

ACM Computing Surveys

Abstract:An increasing number of decisions regarding the daily lives of human beings are being controlled by artificial intelligence and machine learning (ML) algorithms in spheres ranging from healthcare, transportation, and education to college admissions, recruitment, provision of loans, and many more realms. Since they now touch on many aspects of our lives, it is crucial to develop ML algorithms that are not only accurate but also objective and fair. Recent studies have shown that algorithmic decision making may be inherently prone to unfairness, even when there is no intention for it. This article presents an overview of the main concepts of identifying, measuring, and improving algorithmic fairness when using ML algorithms, focusing primarily on classification tasks. The article begins by discussing the causes of algorithmic bias and unfairness and the common definitions and measures for fairness. Fairness-enhancing mechanisms are then reviewed and divided into pre-process, in-process, and post-process mechanisms. A comprehensive comparison of the mechanisms is then conducted, toward a better understanding of which mechanisms should be used in different scenarios. The article ends by reviewing several emerging research sub-fields of algorithmic fairness, beyond classification.

computer science, theory & methods
Towards Fair Machine Learning Software: Understanding and Addressing Model Bias Through Counterfactual Thinking

Zichong Wang,Yang Zhou,Meikang Qiu,Israat Haque,Laura Brown,Yi He,Jianwu Wang,David Lo,Wenbin Zhang

DOI: https://doi.org/10.48550/arXiv.2302.08018

2023-02-16

Abstract:The increasing use of Machine Learning (ML) software can lead to unfair and unethical decisions, thus fairness bugs in software are becoming a growing concern. Addressing these fairness bugs often involves sacrificing ML performance, such as accuracy. To address this issue, we present a novel counterfactual approach that uses counterfactual thinking to tackle the root causes of bias in ML software. In addition, our approach combines models optimized for both performance and fairness, resulting in an optimal solution in both aspects. We conducted a thorough evaluation of our approach on 10 benchmark tasks using a combination of 5 performance metrics, 3 fairness metrics, and 15 measurement scenarios, all applied to 8 real-world datasets. The conducted extensive evaluations show that the proposed method significantly improves the fairness of ML software while maintaining competitive performance, outperforming state-of-the-art solutions in 84.6% of overall cases based on a recent benchmarking tool.

Software Engineering,Artificial Intelligence
Metrics and methods for a systematic comparison of fairness-aware machine learning algorithms

Gareth P. Jones,James M. Hickey,Pietro G. Di Stefano,Charanpal Dhanjal,Laura C. Stoddart,Vlasios Vasileiou

DOI: https://doi.org/10.48550/arXiv.2010.03986

2020-10-08

Abstract:Understanding and removing bias from the decisions made by machine learning models is essential to avoid discrimination against unprivileged groups. Despite recent progress in algorithmic fairness, there is still no clear answer as to which bias-mitigation approaches are most effective. Evaluation strategies are typically use-case specific, rely on data with unclear bias, and employ a fixed policy to convert model outputs to decision outcomes. To address these problems, we performed a systematic comparison of a number of popular fairness algorithms applicable to supervised classification. Our study is the most comprehensive of its kind. It utilizes three real and four synthetic datasets, and two different ways of converting model outputs to decisions. It considers fairness, predictive-performance, calibration quality, and speed of 28 different modelling pipelines, corresponding to both fairness-unaware and fairness-aware algorithms. We found that fairness-unaware algorithms typically fail to produce adequately fair models and that the simplest algorithms are not necessarily the fairest ones. We also found that fairness-aware algorithms can induce fairness without material drops in predictive power. Finally, we found that dataset idiosyncracies (e.g., degree of intrinsic unfairness, nature of correlations) do affect the performance of fairness-aware approaches. Our results allow the practitioner to narrow down the approach(es) they would like to adopt without having to know in advance their fairness requirements.

Machine Learning,Artificial Intelligence,Computers and Society
Fairness in Machine Learning: Definition, Testing, Debugging, and Application

Xuanqi Gao,Chao Shen,Weipeng Jiang,Chenhao Lin,Qian Li,Qian Wang,Qi Li,Xiaohong Guan

DOI: https://doi.org/10.1007/s11432-023-4060-x

2024-01-01

Abstract:In recent years, artificial intelligence technology has been widely used in many fields, such as computer vision, natural language processing and autonomous driving. Machine learning algorithms, as the core technique of AI, have significantly facilitated people’s lives. However, underlying fairness issues in machine learning systems can pose risks to individual fairness and social security. Studying fairness definitions, sources of problems, and testing and debugging methods of fairness can help ensure the fairness of machine learning systems and promote the wide application of artificial intelligence technology in various fields. This paper introduces relevant definitions of machine learning fairness and analyzes the sources of fairness problems. Besides, it provides guidance on fairness testing and debugging methods and summarizes popular datasets. This paper also discusses the technical advancements in machine learning fairness and highlights future challenges in this area.
Fairness Testing: A Comprehensive Survey and Analysis of Trends

Zhenpeng Chen,Jie M. Zhang,Max Hort,Mark Harman,Federica Sarro

DOI: https://doi.org/10.1145/3652155

IF: 3.685

2024-03-11

ACM Transactions on Software Engineering and Methodology

Abstract:Unfair behaviors of Machine Learning (ML) software have garnered increasing attention and concern among software engineers. To tackle this issue, extensive research has been dedicated to conducting fairness testing of ML software, and this paper offers a comprehensive survey of existing studies in this field. We collect 100 papers and organize them based on the testing workflow (i.e., how to test) and testing components (i.e., what to test). Furthermore, we analyze the research focus, trends, and promising directions in the realm of fairness testing. We also identify widely-adopted datasets and open-source tools for fairness testing.

computer science, software engineering
What is Fair? Defining Fairness in Machine Learning for Health

Jianhui Gao,Benson Chou,Zachary R. McCaw,Hilary Thurston,Paul Varghese,Chuan Hong,Jessica Gronsbell

2024-11-13

Abstract:Ensuring that machine learning (ML) models are safe, effective, and equitable across all patient groups is essential for clinical decision-making and for preventing the reinforcement of existing health disparities. This review examines notions of fairness used in ML for health, including a review of why ML models can be unfair and how fairness has been quantified in a wide range of real-world examples. We provide an overview of commonly used fairness metrics and supplement our discussion with a case-study of an openly available electronic health record (EHR) dataset. We also discuss the outlook for future research, highlighting current challenges and opportunities in defining fairness in health.

Machine Learning,Computers and Society
Is it Still Fair? A Comparative Evaluation of Fairness Algorithms through the Lens of Covariate Drift

Oscar Blessed Deho,Michael Bewong,Selasi Kwashie,Jiuyong Li,Jixue Liu,Lin Liu,Srecko Joksimovic

2024-09-19

Abstract:Over the last few decades, machine learning (ML) applications have grown exponentially, yielding several benefits to society. However, these benefits are tempered with concerns of discriminatory behaviours exhibited by ML models. In this regard, fairness in machine learning has emerged as a priority research area. Consequently, several fairness metrics and algorithms have been developed to mitigate against discriminatory behaviours that ML models may possess. Yet still, very little attention has been paid to the problem of naturally occurring changes in data patterns (\textit{aka} data distributional drift), and its impact on fairness algorithms and metrics. In this work, we study this problem comprehensively by analyzing 4 fairness-unaware baseline algorithms and 7 fairness-aware algorithms, carefully curated to cover the breadth of its typology, across 5 datasets including public and proprietary data, and evaluated them using 3 predictive performance and 10 fairness metrics. In doing so, we show that (1) data distributional drift is not a trivial occurrence, and in several cases can lead to serious deterioration of fairness in so-called fair models; (2) contrary to some existing literature, the size and direction of data distributional drift is not correlated to the resulting size and direction of unfairness; and (3) choice of, and training of fairness algorithms is impacted by the effect of data distributional drift which is largely ignored in the literature. Emanating from our findings, we synthesize several policy implications of data distributional drift on fairness algorithms that can be very relevant to stakeholders and practitioners.

Machine Learning,Artificial Intelligence,Computers and Society,Human-Computer Interaction
Fairness-aware Configuration of Machine Learning Libraries

Saeid Tizpaz-Niari,Ashish Kumar,Gang Tan,Ashutosh Trivedi

DOI: https://doi.org/10.1145/3510003.3510202

2022-02-13

Abstract:This paper investigates the parameter space of machine learning (ML) algorithms in aggravating or mitigating fairness bugs. Data-driven software is increasingly applied in social-critical applications where ensuring fairness is of paramount importance. The existing approaches focus on addressing fairness bugs by either modifying the input dataset or modifying the learning algorithms. On the other hand, the selection of hyperparameters, which provide finer controls of ML algorithms, may enable a less intrusive approach to influence the fairness. Can hyperparameters amplify or suppress discrimination present in the input dataset? How can we help programmers in detecting, understanding, and exploiting the role of hyperparameters to improve the fairness? We design three search-based software testing algorithms to uncover the precision-fairness frontier of the hyperparameter space. We complement these algorithms with statistical debugging to explain the role of these parameters in improving fairness. We implement the proposed approaches in the tool Parfait-ML (PARameter FAIrness Testing for ML Libraries) and show its effectiveness and utility over five mature ML algorithms as used in six social-critical applications. In these applications, our approach successfully identified hyperparameters that significantly improve (vis-a-vis the state-of-the-art techniques) the fairness without sacrificing precision. Surprisingly, for some algorithms (e.g., random forest), our approach showed that certain configuration of hyperparameters (e.g., restricting the search space of attributes) can amplify biases across applications. Upon further investigation, we found intuitive explanations of these phenomena, and the results corroborate similar observations from the literature.

Software Engineering,Computers and Society,Machine Learning
Fix Fairness, Don't Ruin Accuracy: Performance Aware Fairness Repair using AutoML

Giang Nguyen,Sumon Biswas,Hridesh Rajan

DOI: https://doi.org/10.1145/3611643.3616257

2023-08-29

Abstract:Machine learning (ML) is increasingly being used in critical decision-making software, but incidents have raised questions about the fairness of ML predictions. To address this issue, new tools and methods are needed to mitigate bias in ML-based software. Previous studies have proposed bias mitigation algorithms that only work in specific situations and often result in a loss of accuracy. Our proposed solution is a novel approach that utilizes automated machine learning (AutoML) techniques to mitigate bias. Our approach includes two key innovations: a novel optimization function and a fairness-aware search space. By improving the default optimization function of AutoML and incorporating fairness objectives, we are able to mitigate bias with little to no loss of accuracy. Additionally, we propose a fairness-aware search space pruning method for AutoML to reduce computational cost and repair time. Our approach, built on the state-of-the-art Auto-Sklearn tool, is designed to reduce bias in real-world scenarios. In order to demonstrate the effectiveness of our approach, we evaluated our approach on four fairness problems and 16 different ML models, and our results show a significant improvement over the baseline and existing bias mitigation techniques. Our approach, Fair-AutoML, successfully repaired 60 out of 64 buggy cases, while existing bias mitigation techniques only repaired up to 44 out of 64 cases.

Software Engineering,Machine Learning
Should Fairness be a Metric or a Model? A Model-based Framework for Assessing Bias in Machine Learning Pipelines

John P. Lalor,Ahmed Abbasi,Kezia Oketch,Yi Yang,Nicole Forsgren

DOI: https://doi.org/10.1145/3641276

IF: 4.657

2024-01-23

ACM Transactions on Information Systems

Abstract:Fairness measurement is crucial for assessing algorithmic bias in various types of machine learning (ML) models, including ones used for search relevance, recommendation, personalization, talent analytics, and natural language processing. However, the fairness measurement paradigm is currently dominated by fairness metrics that examine disparities in allocation and/or prediction error as univariate key performance indicators (KPIs) for a protected attribute or group. Although important and effective in assessing ML bias in certain contexts such as recidivism, existing metrics don’t work well in many real-world applications of ML characterized by imperfect models applied to an array of instances encompassing a multivariate mixture of protected attributes, that are part of a broader process pipeline. Consequently, the upstream representational harm quantified by existing metrics based on how the model represents protected groups doesn’t necessarily relate to allocational harm in the application of such models in downstream policy/decision contexts. We propose FAIR-Frame, a model-based framework for parsimoniously modeling fairness across multiple protected attributes in regard to the representational and allocational harm associated with the upstream design/development and downstream usage of ML models. We evaluate the efficacy of our proposed framework on two testbeds pertaining to text classification using pretrained language models. The upstream testbeds encompass over fifty thousand documents associated with twenty-eight thousand users, seven protected attributes and five different classification tasks. The downstream testbeds span three policy outcomes and over 5.41 million total observations. Results in comparison with several existing metrics show that the upstream representational harm measures produced by FAIR-Frame and other metrics are significantly different from one another, and that FAIR-Frame’s representational fairness measures have the highest percentage alignment and lowest error with allocational harm observed in downstream applications. Our findings have important implications for various ML contexts, including information retrieval, user modeling, digital platforms, and text classification, where responsible and trustworthy AI are becoming an imperative.

computer science, information systems
Evaluating Fairness Using Permutation Tests

Cyrus DiCiccio,Sriram Vasudevan,Kinjal Basu,Krishnaram Kenthapadi,Deepak Agarwal

DOI: https://doi.org/10.48550/arXiv.2007.05124

2020-07-10

Abstract:Machine learning models are central to people's lives and impact society in ways as fundamental as determining how people access information. The gravity of these models imparts a responsibility to model developers to ensure that they are treating users in a fair and equitable manner. Before deploying a model into production, it is crucial to examine the extent to which its predictions demonstrate biases. This paper deals with the detection of bias exhibited by a machine learning model through statistical hypothesis testing. We propose a permutation testing methodology that performs a hypothesis test that a model is fair across two groups with respect to any given metric. There are increasingly many notions of fairness that can speak to different aspects of model fairness. Our aim is to provide a flexible framework that empowers practitioners to identify significant biases in any metric they wish to study. We provide a formal testing mechanism as well as extensive experiments to show how this method works in practice.

Applications
Fairness Deconstructed: A Sociotechnical View of 'Fair' Algorithms in Criminal Justice

Rajiv Movva

DOI: https://doi.org/10.48550/arXiv.2106.13455

2021-06-25

Computers and Society

Abstract:Early studies of risk assessment algorithms used in criminal justice revealed widespread racial biases. In response, machine learning researchers have developed methods for fairness, many of which rely on equalizing empirical metrics across protected attributes. Here, I recall sociotechnical perspectives to delineate the significant gap between fairness in theory and practice, focusing on criminal justice. I (1) illustrate how social context can undermine analyses that are restricted to an AI system's outputs, and (2) argue that much of the fair ML literature fails to account for epistemological issues with underlying crime data. Instead of building AI that reifies power imbalances, like risk assessment algorithms, I ask whether data science can be used to understand the root causes of structural marginalization.
Fairness in Machine Learning Meets with Equity in Healthcare

Shaina Raza,Parisa Osivand Pour,Syed Raza Bashir

DOI: https://doi.org/10.1609/aaaiss.v1i1.27493

2023-10-03

Abstract:With the growing utilization of machine learning in healthcare, there is increasing potential to enhance healthcare outcomes. However, this also brings the risk of perpetuating biases in data and model design that can harm certain demographic groups based on factors such as age, gender, and race. This study proposes an artificial intelligence framework, grounded in software engineering principles, for identifying and mitigating biases in data and models while ensuring fairness in healthcare settings. A case study is presented to demonstrate how systematic biases in data can lead to amplified biases in model predictions, and machine learning methods are suggested to pre-vent such biases. Future research aims to test and validate the proposed ML framework in real-world clinical settings to evaluate its impact on promoting health equity.
The Flawed Foundations of Fair Machine Learning

Robert Lee Poe,Soumia Zohra El Mestari

2023-06-02

Abstract:The definition and implementation of fairness in automated decisions has been extensively studied by the research community. Yet, there hides fallacious reasoning, misleading assertions, and questionable practices at the foundations of the current fair machine learning paradigm. Those flaws are the result of a failure to understand that the trade-off between statistically accurate outcomes and group similar outcomes exists as independent, external constraint rather than as a subjective manifestation as has been commonly argued. First, we explain that there is only one conception of fairness present in the fair machine learning literature: group similarity of outcomes based on a sensitive attribute where the similarity benefits an underprivileged group. Second, we show that there is, in fact, a trade-off between statistically accurate outcomes and group similar outcomes in any data setting where group disparities exist, and that the trade-off presents an existential threat to the equitable, fair machine learning approach. Third, we introduce a proof-of-concept evaluation to aid researchers and designers in understanding the relationship between statistically accurate outcomes and group similar outcomes. Finally, suggestions for future work aimed at data scientists, legal scholars, and data ethicists that utilize the conceptual and experimental framework described throughout this article are provided.

Computers and Society,Machine Learning
Measuring Non-Expert Comprehension of Machine Learning Fairness Metrics

Debjani Saha,Candice Schumann,Duncan C. McElfresh,John P. Dickerson,Michelle L. Mazurek,Michael Carl Tschantz

DOI: https://doi.org/10.48550/arXiv.2001.00089

2019-12-17

Computers and Society

Abstract:Bias in machine learning has manifested injustice in several areas, such as medicine, hiring, and criminal justice. In response, computer scientists have developed myriad definitions of fairness to correct this bias in fielded algorithms. While some definitions are based on established legal and ethical norms, others are largely mathematical. It is unclear whether the general public agrees with these fairness definitions, and perhaps more importantly, whether they understand these definitions. We take initial steps toward bridging this gap between ML researchers and the public, by addressing the question: does a lay audience understand a basic definition of ML fairness? We develop a metric to measure comprehension of three such definitions--demographic parity, equal opportunity, and equalized odds. We evaluate this metric using an online survey, and investigate the relationship between comprehension and sentiment, demographics, and the definition itself.
Do the Machine Learning Models on a Crowd Sourced Platform Exhibit Bias? An Empirical Study on Model Fairness

Sumon Biswas,Hridesh Rajan

DOI: https://doi.org/10.1145/3368089.3409704

2020-09-23

Abstract:Machine learning models are increasingly being used in important decision-making software such as approving bank loans, recommending criminal sentencing, hiring employees, and so on. It is important to ensure the fairness of these models so that no discrimination is made based on protected attribute (e.g., race, sex, age) while decision making. Algorithms have been developed to measure unfairness and mitigate them to a certain extent. In this paper, we have focused on the empirical evaluation of fairness and mitigations on real-world machine learning models. We have created a benchmark of 40 top-rated models from Kaggle used for 5 different tasks, and then using a comprehensive set of fairness metrics, evaluated their fairness. Then, we have applied 7 mitigation techniques on these models and analyzed the fairness, mitigation results, and impacts on performance. We have found that some model optimization techniques result in inducing unfairness in the models. On the other hand, although there are some fairness control mechanisms in machine learning libraries, they are not documented. The mitigation algorithm also exhibit common patterns such as mitigation in the post-processing is often costly (in terms of performance) and mitigation in the pre-processing stage is preferred in most cases. We have also presented different trade-off choices of fairness mitigation decisions. Our study suggests future research directions to reduce the gap between theoretical fairness aware algorithms and the software engineering methods to leverage them in practice.

Machine Learning,Software Engineering
An Empirical Comparison of Bias Reduction Methods on Real-World Problems in High-Stakes Policy Settings

Hemank Lamba,Kit T. Rodolfa,Rayid Ghani

DOI: https://doi.org/10.48550/arXiv.2105.06442

2021-05-14

Abstract:Applications of machine learning (ML) to high-stakes policy settings -- such as education, criminal justice, healthcare, and social service delivery -- have grown rapidly in recent years, sparking important conversations about how to ensure fair outcomes from these systems. The machine learning research community has responded to this challenge with a wide array of proposed fairness-enhancing strategies for ML models, but despite the large number of methods that have been developed, little empirical work exists evaluating these methods in real-world settings. Here, we seek to fill this research gap by investigating the performance of several methods that operate at different points in the ML pipeline across four real-world public policy and social good problems. Across these problems, we find a wide degree of variability and inconsistency in the ability of many of these methods to improve model fairness, but post-processing by choosing group-specific score thresholds consistently removes disparities, with important implications for both the ML research community and practitioners deploying machine learning to inform consequential policy decisions.

Machine Learning,Computers and Society
Assessing Perceived Fairness from Machine Learning Developer's Perspective

Anoop Mishra,Deepak Khazanchi

2023-04-08

Abstract:Fairness in machine learning (ML) applications is an important practice for developers in research and industry. In ML applications, unfairness is triggered due to bias in the data, curation process, erroneous assumptions, and implicit bias rendered within the algorithmic development process. As ML applications come into broader use developing fair ML applications is critical. Literature suggests multiple views on how fairness in ML is described from the users perspective and students as future developers. In particular, ML developers have not been the focus of research relating to perceived fairness. This paper reports on a pilot investigation of ML developers perception of fairness. In describing the perception of fairness, the paper performs an exploratory pilot study to assess the attributes of this construct using a systematic focus group of developers. In the focus group, we asked participants to discuss three questions- 1) What are the characteristics of fairness in ML? 2) What factors influence developers belief about the fairness of ML? and 3) What practices and tools are utilized for fairness in ML development? The findings of this exploratory work from the focus group show that to assess fairness developers generally focus on the overall ML application design and development, i.e., business-specific requirements, data collection, pre-processing, in-processing, and post-processing. Thus, we conclude that the procedural aspects of organizational justice theory can explain developers perception of fairness. The findings of this study can be utilized further to assist development teams in integrating fairness in the ML application development lifecycle. It will also motivate ML developers and organizations to develop best practices for assessing the fairness of ML-based applications.

Machine Learning,Computers and Society,Human-Computer Interaction
Fairness Measures of Machine Learning Models in Judicial Penalty Prediction

Yanjun Li,Huan Huang,Qiang Geng,Xinwei Guo,Yuyu Yuan

DOI: https://doi.org/10.53106/160792642022092305019

2022-01-01

Abstract:Machine learning (ML) has been widely adopted in many software applications across domains. However, accompanying the outstanding performance, the behaviors of the ML models, which are essentially a kind of black-box software, could be unfair and hard to understand in many cases. In our human-centered society, an unfair decision could potentially damage human value, even causing severe social consequences, especially in decision-critical scenarios such as legal judgment. Although some existing works investigated the ML models in terms of robustness, accuracy, security, privacy, quality, etc., the study on the fairness of ML is still in the early stage. In this paper, we first proposed a set of fairness metrics for ML models from different perspectives. Based on this, we performed a comparative study on the fairness of existing widely used classic ML and deep learning models in the domain of real-world judicial judgments. The experiment results reveal that the current state-of-the-art ML models could still raise concerns for unfair decision-making. The ML models with high accuracy and fairness are urgently demanding.

Fluorosteroids. V. Preparation and biological activity of 6α-fluoro-17β-carboxylic acids of the androstane series

Data vs. Model Machine Learning Fairness Testing: An Empirical Study

A Review on Fairness in Machine Learning

Towards Fair Machine Learning Software: Understanding and Addressing Model Bias Through Counterfactual Thinking

Metrics and methods for a systematic comparison of fairness-aware machine learning algorithms

Fairness in Machine Learning: Definition, Testing, Debugging, and Application

Fairness Testing: A Comprehensive Survey and Analysis of Trends

What is Fair? Defining Fairness in Machine Learning for Health

Is it Still Fair? A Comparative Evaluation of Fairness Algorithms through the Lens of Covariate Drift

Fairness-aware Configuration of Machine Learning Libraries

Fix Fairness, Don't Ruin Accuracy: Performance Aware Fairness Repair using AutoML

Should Fairness be a Metric or a Model? A Model-based Framework for Assessing Bias in Machine Learning Pipelines

Evaluating Fairness Using Permutation Tests

Fairness Deconstructed: A Sociotechnical View of 'Fair' Algorithms in Criminal Justice

Fairness in Machine Learning Meets with Equity in Healthcare

The Flawed Foundations of Fair Machine Learning

Measuring Non-Expert Comprehension of Machine Learning Fairness Metrics

Do the Machine Learning Models on a Crowd Sourced Platform Exhibit Bias? An Empirical Study on Model Fairness

An Empirical Comparison of Bias Reduction Methods on Real-World Problems in High-Stakes Policy Settings

Assessing Perceived Fairness from Machine Learning Developer's Perspective

Fairness Measures of Machine Learning Models in Judicial Penalty Prediction