Abstract:Despite Federated Learning (FL)'s trend for learning machine learning models in a distributed manner, it is susceptible to performance drops when training on heterogeneous data. In addition, FL inevitability faces the challenges of synchronization, efficiency, and privacy. Recently, dataset distillation has been explored in order to improve the efficiency and scalability of FL by creating a smaller, synthetic dataset that retains the performance of a model trained on the local private datasets. We discover that using distilled local datasets can amplify the heterogeneity issue in FL. To address this, we propose a new method, called Federated Virtual Learning on Heterogeneous Data with Local-Global Distillation (FedLGD), which trains FL using a smaller synthetic dataset (referred as virtual data) created through a combination of local and global dataset distillation. Specifically, to handle synchronization and class imbalance, we propose iterative distribution matching to allow clients to have the same amount of balanced local virtual data; to harmonize the domain shifts, we use federated gradient matching to distill global virtual data that are shared with clients without hindering data privacy to rectify heterogeneous local training via enforcing local-global feature similarity. We experiment on both benchmark and real-world datasets that contain heterogeneous data from different sources, and further scale up to an FL scenario that contains large number of clients with heterogeneous and class imbalance data. Our method outperforms state-of-the-art heterogeneous FL algorithms under various settings with a very limited amount of distilled virtual data.

Toward Secure and Robust Federated Distillation in Distributed Cloud: Challenges and Design Issues

Towards Secure and Robust Federated Distillation in Distributed Cloud: Challenges and Design Issues

Federated Distillation: A Survey

Secure and Efficient Decentralized Federated Learning with Data Representation Protection

Communication-Efficient Federated Distillation with Active Data Sampling

Research on Privacy Protection in Federated Learning Combining Distillation Defense and Blockchain

A Generalized Look at Federated Learning: Survey and Perspectives

To Distill or Not to Distill: Toward Fast, Accurate, and Communication-Efficient Federated Distillation Learning.

BlockDFL: A Blockchain-based Fully Decentralized Peer-to-Peer Federated Learning Framework

GFL: A Decentralized Federated Learning Framework Based On Blockchain.

To Distill or Not To Distill: Towards Fast, Accurate and Communication Efficient Federated Distillation Learning

DFRD: Data-Free Robustness Distillation for Heterogeneous Federated Learning

Towards Data-Independent Knowledge Transfer in Model-Heterogeneous Federated Learning

Decentralized Federated Learning Via Mutual Knowledge Distillation.

Federated Virtual Learning on Heterogeneous Data with Local-global Distillation

When federated learning meets blockchain: A new distributed learning paradigm

Federated Learning Via Input-Output Collaborative Distillation

Privacy-preserving Decentralized Federated Deep Learning.

Fair and Robust Federated Learning via Decentralized and Adaptive Aggregation based on Blockchain

Privacy-Preserving and Reliable Decentralized Federated Learning

An Efficient Decentralized Federated Learning Framework Based on Directed Acyclic Graph