Abstract:Federated learning (FL) enables distributed clients to collaboratively learn a shared model while keeping their raw data private. To mitigate the system heterogeneity issues of FL and overcome the resource constraints of clients, we investigate a novel paradigm in which heterogeneous clients learn uniquely designed models with different architectures, and transfer knowledge to the server to train a larger server model that in turn helps to enhance client models. For efficient knowledge transfer between client models and server model, we propose FedHKT, a Hierarchical Knowledge Transfer framework for FL. The main idea of FedHKT is to allow clients with similar data distributions to collaboratively learn to specialize in certain classes, then the specialized knowledge of clients is aggregated to a super knowledge covering all specialties to train the server model, and finally the server model knowledge is distilled to client models. Specifically, we tailor a hybrid knowledge transfer mechanism for FedHKT, where the model parameters based and knowledge distillation (KD) based methods are respectively used for client-edge and edge-cloud knowledge transfer, which can harness the pros and evade the cons of these two approaches in learning performance and resource efficiency. Besides, to efficiently aggregate knowledge for conducive server model training, we propose a weighted ensemble distillation scheme with server-assisted knowledge selection, which aggregates knowledge by its prediction confidence, selects qualified knowledge during server model training, and uses selected knowledge to help improve client models. Extensive experiments demonstrate the superior performance of FedHKT compared to state-of-the-art baselines.

FedTAIL: A Federated Learning Approach with Trans-Architecture Intermediate Links

FedDGP: Disentangling Global and Personal Models for Federated Learning

Understanding the Training Dynamics in Federated Deep Learning via Aggregation Weight Optimization

Tailored Federated Learning: Leveraging Direction Regulation & Knowledge Distillation

FedEmb: A Vertical and Hybrid Federated Learning Algorithm using Network And Feature Embedding Aggregation

Federated Learning with Flexible Architectures

Federated Learning FedLTailor: A Dynamic Weight Adjustment and Personalized Fusion Approach

Fine-tuning Global Model Via Data-Free Knowledge Distillation for Non-IID Federated Learning

An Aggregation-Free Federated Learning for Tackling Data Heterogeneity

FedAGA: A federated learning framework for enhanced inter-client relationship learning

FedIN: Federated Intermediate Layers Learning for Model Heterogeneity

Handling Data Heterogeneity in Federated Learning via Knowledge Distillation and Fusion

FedDistill: Global Model Distillation for Local Model De-Biasing in Non-IID Federated Learning

FRAug: Tackling Federated Learning with Non-IID Features via Representation Augmentation

Joint Local Relational Augmentation and Global Nash Equilibrium for Federated Learning with Non-IID Data

A Hierarchical Knowledge Transfer Framework for Heterogeneous Federated Learning.

FedTweet: Two-fold Knowledge Distillation for non-IID Federated Learning

FedALA: Adaptive Local Aggregation for Personalized Federated Learning

TailorFL: Dual-Personalized Federated Learning under System and Data Heterogeneity.

Local Learning Matters: Rethinking Data Heterogeneity in Federated Learning

FedRAD: Heterogeneous Federated Learning via Relational Adaptive Distillation