Abstract:The development of face recognition algorithms by academic and commercial organizations is growing rapidly due to the onset of deep learning and the widespread availability of training data. Though tests of face recognition algorithm performance indicate yearly performance gains, error rates for many of these systems differ based on the demographic composition of the test set. These "demographic differentials" in algorithm performance can contribute to unequal or unfair outcomes for certain groups of people, raising concerns with increased worldwide adoption of face recognition systems. Consequently, regulatory bodies in both the United States and Europe have proposed new rules requiring audits of biometric systems for "discriminatory impacts" (European Union Artificial Intelligence Act) and "fairness" (U.S. Federal Trade Commission). However, no standard for measuring fairness in biometric systems yet exists. This paper characterizes two proposed measures of face recognition algorithm fairness (fairness measures) from scientists in the U.S. and Europe. We find that both proposed methods are challenging to interpret when applied to disaggregated face recognition error rates as they are commonly experienced in practice. To address this, we propose a set of interpretability criteria, termed the Functional Fairness Measure Criteria (FFMC), that outlines a set of properties desirable in a face recognition algorithm fairness measure. We further develop a new fairness measure, the Gini Aggregation Rate for Biometric Equitability (GARBE), and show how, in conjunction with the Pareto optimization, this measure can be used to select among alternative algorithms based on the accuracy/fairness trade-space. Finally, we have open-sourced our dataset of machine-readable, demographically disaggregated error rates. We believe this is currently the largest open-source dataset of its kind.

A Comprehensive Study on Face Recognition Biases Beyond Demographics

A Deep Dive into Dataset Imbalance and Bias in Face Identification

Explaining Bias in Deep Face Recognition via Image Characteristics

Fairness Under Cover: Evaluating the Impact of Occlusions on Demographic Bias in Facial Recognition

Rethinking Bias Mitigation: Fairer Architectures Make for Fairer Face Recognition

Beyond Identity: What Information Is Stored in Biometric Face Templates?

On the Potential of Algorithm Fusion for Demographic Bias Mitigation in Face Recognition

Investigating Bias and Fairness in Facial Expression Recognition

Exploring Causes of Demographic Variations In Face Recognition Accuracy

Racial Bias within Face Recognition: A Survey

FineFACE: Fair Facial Attribute Classification Leveraging Fine-grained Features

Comparing Human and Machine Bias in Face Recognition

Bias and Diversity in Synthetic-based Face Recognition

Surveying Racial Bias in Facial Recognition: Balancing Datasets and Algorithmic Enhancements

Metrics for Dataset Demographic Bias: A Case Study on Facial Expression Recognition

Less can be more: representational vs. stereotypical gender bias in facial expression recognition

Benchmarking Algorithmic Bias in Face Recognition: An Experimental Approach Using Synthetic Faces and Human Evaluation

Synthetic Data for the Mitigation of Demographic Biases in Face Recognition

Evaluating Proposed Fairness Models for Face Recognition Algorithms

Investigating Bias in Deep Face Analysis: The KANFace Dataset and Empirical Study