Abstract:Although machine learning algorithms demonstrate impressive performance, their trustworthiness remains a critical issue, particularly concerning fairness when implemented in real-world applications. Many notions of group fairness aim to minimize disparities in performance across protected groups. However, it can inadvertently reduce performance in certain groups, leading to sub-optimal outcomes. In contrast, Min-max group fairness notion prioritizes the improvement for the worst-performing group, thereby advocating a utility-promoting approach to fairness. However, it has been proven that existing efforts to achieve Min-max fairness exhibit limited effectiveness. In response to this challenge, we leverage the recently proposed "Neural Collapse'' framework to re-examine Empirical Risk Minimization (ERM) training, specifically investigating the root causes of poor performance in minority groups. The layer-peeled model is employed to decompose a network into two parts: an encoder to learn latent representation, and a subsequent classifier, with a systematic characterization of their training behaviors being conducted. Our analysis reveals that while classifiers achieve maximum separation, the separability of representations is insufficient, particularly for minority groups. This indicates the sub-optimal performance in minority groups stems from less separable representations, rather than classifiers. To tackle this issue, we introduce a novel strategy that incorporates a frozen classifier to directly enhance representation. Furthermore, we introduce two easily implemented loss functions to guide the learning process. The experimental assessments carried out on real-world benchmark datasets spanning the domains of Computer Vision, Natural Language Processing, and Tabular data demonstrate that our approach outperforms existing state-of-the-art methods in promoting the Min-max fairness notion.

Learning with Shared Representations: Statistical Rates and Efficient Algorithms

Privacy-Preserving Collaborative Deep Learning with Unreliable Participants.

Efficient Collaborative Learning over Unreliable D2D Network: Adaptive Cluster Head Selection and Resource Allocation

Exploiting Shared Representations for Personalized Federated Learning

Asynchronous Byzantine-Robust Stochastic Aggregation with Variance Reduction for Distributed Learning

Guarantees for Nonlinear Representation Learning: Non-identical Covariates, Dependent Data, Fewer Samples

DePRL: Achieving Linear Convergence Speedup in Personalized Decentralized Learning with Shared Representations

Adversarial Representation Sharing: A Quantitative and Secure Collaborative Learning Framework

On Sample Complexity of Learning Shared Representations: the Asymptotic Regime

Learning from Similar Linear Representations: Adaptivity, Minimaxity, and Robustness

An Information Theoretic Approach for Collaborative Distributed Parameter Estimation.

Sharp Rates in Dependent Learning Theory: Avoiding Sample Size Deflation for the Square Loss

Neural Collapse Inspired Debiased Representation Learning for Min-max Fairness

Data Sharing for Mean Estimation Among Heterogeneous Strategic Agents

Few-Shot Learning via Learning the Representation, Provably

Fast rates in statistical and online learning

The Copycat Perceptron: Smashing Barriers Through Collective Learning

On the Fundamental Limit of Distributed Learning with Interchangable Constrained Statistics

Sample-Efficient Linear Representation Learning from Non-IID Non-Isotropic Data

A Non-parametric View of FedAvg and FedProx: Beyond Stationary Points

Fairness-Driven Private Collaborative Machine Learning