Abstract:Statistical heterogeneity severely limits the performance of federated learning (FL), motivating several explorations e.g., FedProx, MOON and FedDyn, to alleviate this problem. Despite effectiveness, their considered scenario generally requires samples from almost all classes during the local training of each client, although some covariate shifts may exist among clients. In fact, the natural case of partially class-disjoint data (PCDD), where each client contributes a few classes (instead of all classes) of samples, is practical yet underexplored. Specifically, the unique collapse and invasion characteristics of PCDD can induce the biased optimization direction in local training, which prevents the efficiency of federated learning. To address this dilemma, we propose a manifold reshaping approach called FedMR to calibrate the feature space of local training. Our FedMR adds two interplaying losses to the vanilla federated learning: one is intra-class loss to decorrelate feature dimensions for anti-collapse; and the other one is inter-class loss to guarantee the proper margin among categories in the feature expansion. We conduct extensive experiments on a range of datasets to demonstrate that our FedMR achieves much higher accuracy and better communication efficiency. Source code is available at: <a class="link-external link-https" href="https://github.com/MediaBrain-SJTU/FedMR.git" rel="external noopener nofollow">this https URL</a>.

What problem does this paper attempt to address?

The problem that this paper attempts to solve is the severely limited performance of Federated Learning (FL) in the case of Partially Class - Disjoint Data (PCDD). Specifically: 1. **Statistical heterogeneity problem**: In typical federated learning, each client usually contains samples of all classes, but due to covariate shifts, the data distribution may vary among different clients. However, in practical applications, a more common situation is that each client only contributes samples of a part of the classes, that is, PCDD. 2. **Feature space degradation problem**: PCDD will lead to dimensional collapse and space invasion in the feature space. Due to the lack of sample support from all classes, the feature representation in local training will be randomly distributed on the low - dimensional manifold, resulting in deviation in the optimization direction and affecting the efficiency and accuracy of federated learning. To address these problems, the authors propose a method called FedMR (Federated Learning via Manifold Reshaping), which corrects the feature space by introducing two interacting loss functions: - **Intra - class Loss**: Used to decorrelate feature dimensions and prevent dimensional collapse. - **Inter - class Loss**: Ensures an appropriate separation between classes through global class prototypes and prevents space invasion. Experimental results show that FedMR achieves higher accuracy and better communication efficiency on multiple benchmark datasets and a real - world medical dataset ISIC2019. ### Formula summary - **Intra - class Loss**: \[ \ell_{\text{intra}}^k=\frac{1}{|C_k|} \sum_{c \in C_k}\|M_c^k\|_F^2 \] where \(M_c^k\) is the intra - class covariance matrix of class \(c\) in the \(k\)-th client. - **Inter - class Loss**: \[ \ell_{\text{inter}}^k = \frac{1}{|C_k|(|C_k|-1)} \sum_{c_i \in C_k} \sum_{c_j \in C_k\setminus c_i} D_{c_i, c_j} \] where \(D_{c_i, c_j}\) is defined as: \[ D_{c_i, c_j}=\frac{1}{N_c^k} \sum_{n = 1}^{N_c^k} \max \left\{\|z_{c_i}^{k,n}-g_t^{c_i}\|-\|z_{c_i}^{k,n}-g_t^{c_j}\|, 0\right\} \] These formulas and methods work together to enable FedMR to significantly improve the performance of federated learning in the PCDD scenario.

Federated Learning under Partially Class-Disjoint Data via Manifold Reshaping

Federated Learning under Partially Disjoint Data via Manifold Reshaping

FedDGP: Disentangling Global and Personal Models for Federated Learning

Federated Learning with Manifold Regularization and Normalized Update Reaggregation

FedFed: Feature Distillation Against Data Heterogeneity in Federated Learning

Federated Learning with Bilateral Curation for Partially Class-Disjoint Data

Tackling Data Heterogeneity in Federated Learning via Loss Decomposition

FedPartial: Enabling Model-Heterogeneous Federated Learning Via Partial Model Transmission and Aggregation

Local Learning Matters: Rethinking Data Heterogeneity in Federated Learning

Completely Heterogeneous Federated Learning

Precision Collaboration for Federated Learning

FedUP: Bridging Fairness and Efficiency in Cross-Silo Federated Learning

HyperFed: Hyperbolic Prototypes Exploration with Consistent Aggregation for Non-IID Data in Federated Learning

FedReMa: Improving Personalized Federated Learning via Leveraging the Most Relevant Clients

FedH2L: A Federated Learning Approach with Model and Statistical Heterogeneity

Hybrid Federated Learning: Algorithms and Implementation.

Hybrid Federated Learning: Algorithms and Implementation

FedCross: Towards Accurate Federated Learning via Multi-Model Cross-Aggregation

Federated mutual learning

A Unified Personalized Federated Learning Framework Ensuring Domain Generalization

Joint Local Relational Augmentation and Global Nash Equilibrium for Federated Learning with Non-IID Data