Lucas Rosenblatt,R. Teal Witter
Abstract:Making fair decisions is crucial to ethically implementing machine learning algorithms in social settings. In this work, we consider the celebrated definition of counterfactual fairness [Kusner et al., NeurIPS, 2017]. We begin by showing that an algorithm which satisfies counterfactual fairness also satisfies demographic parity, a far simpler fairness constraint. Similarly, we show that all algorithms satisfying demographic parity can be trivially modified to satisfy counterfactual fairness. Together, our results indicate that counterfactual fairness is basically equivalent to demographic parity, which has important implications for the growing body of work on counterfactual fairness. We then validate our theoretical findings empirically, analyzing three existing algorithms for counterfactual fairness against three simple benchmarks. We find that two simple benchmark algorithms outperform all three existing algorithms -- in terms of fairness, accuracy, and efficiency -- on several data sets. Our analysis leads us to formalize a concrete fairness goal: to preserve the order of individuals within protected groups. We believe transparency around the ordering of individuals within protected groups makes fair algorithms more trustworthy. By design, the two simple benchmark algorithms satisfy this goal while the existing algorithms for counterfactual fairness do not.
What problem does this paper attempt to address?
The main problem that this paper attempts to solve is to explore the relationship between counterfactual fairness and demographic parity. Specifically, the author attempts to prove that an algorithm that satisfies counterfactual fairness must also satisfy demographic parity, and an algorithm that satisfies demographic parity can be simply modified to satisfy counterfactual fairness. This finding is of great significance to the existing research on counterfactual fairness, because counterfactual fairness is usually considered as a more complex and advanced definition of fairness, while demographic parity is relatively simple but has some flaws.
### Main contributions of the paper
1. **Theoretical contributions**:
- It is proved that an algorithm that satisfies counterfactual fairness also satisfies demographic parity.
- It is proved that an algorithm that satisfies demographic parity can be simply modified to satisfy counterfactual fairness.
- These results indicate that counterfactual fairness is largely equivalent to demographic parity, which has an important impact on the fairness community, because counterfactual fairness is usually regarded as a novel tool, while demographic parity is regarded as a simple and flawed method.
2. **Empirical analysis**:
- Three existing counterfactual fairness algorithms are analyzed and compared with three simple benchmark algorithms.
- Multiple datasets and causal models are used for verification, and it is found that two simple benchmark algorithms are superior to the existing counterfactual fairness algorithms in terms of fairness, accuracy, and efficiency.
- A new fairness objective is introduced: maintaining the order of individuals within the protected group. The author believes that transparently handling the order of individuals within the protected group can make the fair algorithm more credible.
### Formulas and definitions
#### Definition 1: Demographic parity
Given a predictor \(\hat{Y}: X\times A\rightarrow Y\), if for all protected attributes \(a\) and \(a'\), we have:
\[ \Pr(\hat{Y}(x, a)=y\mid A = a)=\Pr(\hat{Y}(x, a') = y\mid A = a') \]
where the probability is taken over the conditional distribution of \(X\) and the possible randomness of \(\hat{Y}\).
#### Definition 2: Counterfactual fairness
Let \(A, U, X, Y\) be a set of random variables in a causal model. Given a protected group \(a'\in A\) and an outcome \(y\in Y\), a predictor \(\hat{Y}: U\times A\rightarrow Y\) is counterfactually fair if for all observed values \(x\in X\) and \(a\in A\), we have:
\[ \Pr(\hat{Y}_{A\leftarrow a}(u)=y\mid X = x, A = a)=\Pr(\hat{Y}_{A\leftarrow a'}(u)=y\mid X = x, A = a) \]
where the probability is with respect to the posterior distribution of \(u\) induced by the fixed observed values \(x\) and \(a\).
### Empirical analysis methods
- **Algorithms**:
- **Level 1**: Only use the remaining variables independent of the protected attributes to learn the outcome.
- **Level 2**: Use the complete causal model and learn the outcome by estimating the latent variables.
- **Level 3**: Between Level 1 and Level 2, use the relationships in the causal model to express the remaining variables of each individual.
- **Simple benchmark algorithm**: A simple heuristic algorithm for achieving demographic parity.
- **Full - linear model**: A linear model using all protected attributes and remaining variables to learn the outcome.
- **Fairness measures**:
- **\((\epsilon,\delta)\)-approximate counterfactual fairness**: Defined as:
\[ \Pr(|\hat{Y}_{A