Abstract:This paper investigates the parameter space of machine learning (ML) algorithms in aggravating or mitigating fairness bugs. Data-driven software is increasingly applied in social-critical applications where ensuring fairness is of paramount importance. The existing approaches focus on addressing fairness bugs by either modifying the input dataset or modifying the learning algorithms. On the other hand, the selection of hyperparameters, which provide finer controls of ML algorithms, may enable a less intrusive approach to influence the fairness. Can hyperparameters amplify or suppress discrimination present in the input dataset? How can we help programmers in detecting, understanding, and exploiting the role of hyperparameters to improve the fairness? We design three search-based software testing algorithms to uncover the precision-fairness frontier of the hyperparameter space. We complement these algorithms with statistical debugging to explain the role of these parameters in improving fairness. We implement the proposed approaches in the tool Parfait-ML (PARameter FAIrness Testing for ML Libraries) and show its effectiveness and utility over five mature ML algorithms as used in six social-critical applications. In these applications, our approach successfully identified hyperparameters that significantly improve (vis-a-vis the state-of-the-art techniques) the fairness without sacrificing precision. Surprisingly, for some algorithms (e.g., random forest), our approach showed that certain configuration of hyperparameters (e.g., restricting the search space of attributes) can amplify biases across applications. Upon further investigation, we found intuitive explanations of these phenomena, and the results corroborate similar observations from the literature.

Fairness Testing: A Comprehensive Survey and Analysis of Trends

Fairness in Machine Learning: Definition, Testing, Debugging, and Application

50 Years of Test (Un)fairness: Lessons for Machine Learning

Long-Term Fairness Inquiries and Pursuits in Machine Learning: A Survey of Notions, Methods, and Challenges

A Review on Fairness in Machine Learning

Towards Fair Machine Learning Software: Understanding and Addressing Model Bias Through Counterfactual Thinking

Data vs. Model Machine Learning Fairness Testing: An Empirical Study

What-is and How-to for Fairness in Machine Learning: A Survey, Reflection, and Perspective

Analyzing Fairness of Computer Vision and Natural Language Processing Models

A Catalog of Fairness-Aware Practices in Machine Learning Engineering

Lazy Data Practices Harm Fairness Research

Survey on Fairness Notions and Related Tensions

Fairness testing: testing software for discrimination

Fairness issues, current approaches, and challenges in machine learning models

Evaluating Fairness Using Permutation Tests

Putting Fairness Principles into Practice: Challenges, Metrics, and Improvements

A novel approach for assessing fairness in deployed machine learning algorithms

Fairness-aware Configuration of Machine Learning Libraries

What is Fair? Defining Fairness in Machine Learning for Health

Fairness for machine learning software in education: A systematic mapping study

A Survey on Bias and Fairness in Machine Learning