Abstract:Self-supervised learning (SSL) is capable of learning remarkable representations from centrally available data. Recent works further implement federated learning with SSL to learn from rapidly growing decentralized unlabeled images (e.g., from cameras and phones), often resulted from privacy constraints. Extensive attention has been paid to SSL approaches based on Siamese networks. However, such an effort has not yet revealed deep insights into various fundamental building blocks for the federated self-supervised learning (FedSSL) architecture. We aim to fill in this gap via in-depth empirical study and propose a new method to tackle the non-independently and identically distributed (non-IID) data problem of decentralized data. Firstly, we introduce a generalized FedSSL framework that embraces existing SSL methods based on Siamese networks and presents flexibility catering to future methods. In this framework, a server coordinates multiple clients to conduct SSL training and periodically updates local models of clients with the aggregated global model. Using the framework, our study uncovers unique insights of FedSSL: 1) stop-gradient operation, previously reported to be essential, is not always necessary in FedSSL; 2) retaining local knowledge of clients in FedSSL is particularly beneficial for non-IID data. Inspired by the insights, we then propose a new approach for model update, Federated Divergence-aware Exponential Moving Average update (FedEMA). FedEMA updates local models of clients adaptively using EMA of the global model, where the decay rate is dynamically measured by model divergence. Extensive experiments demonstrate that FedEMA outperforms existing methods by 3-4% on linear evaluation. We hope that this work will provide useful insights for future research.

FedIDE: Federated Semi-Supervised Learning with Instance Discrimination & LocalEMA

SemiFed: Semi-supervised Federated Learning with Consistency and Pseudo-Labeling

Semi-Supervised Federated Learning with Non-Iid Data: Algorithm and System Design

Non-IID always Bad? Semi-Supervised Heterogeneous Federated Learning with Local Knowledge Enhancement

Federated Semi-supervised Learning with Dual Regulator

Enhancing Federated Learning with Server-Side Unlabeled Data by Adaptive Client and Data Selection

FedEmb: A Vertical and Hybrid Federated Learning Algorithm using Network And Feature Embedding Aggregation

Enhancing Federated Learning with In-Cloud Unlabeled Data

Class Balanced Adaptive Pseudo Labeling for Federated Semi-Supervised Learning

Semi-supervised Federated Learning on Evolving Data Streams

FedCD: A Classifier Debiased Federated Learning Framework for Non-IID Data

Federated semi-supervised learning with tolerant guidance and powerful classifier in edge scenarios

FedEL: Federated ensemble learning for non-iid data

FedINC: an Exemplar-Free Continual Federated Learning Framework with Small Labeled Data

A Survey towards Federated Semi-supervised Learning

FedEntropy: Information-entropy-aided Training Optimization of Semi-Supervised Federated Learning

Divergence-aware Federated Self-Supervised Learning

Towards Instance-adaptive Inference for Federated Learning

FedCo: Self-Supervised Learning in Federated Learning with Momentum Contrast.

FedCD: Federated Semi-Supervised Learning with Class Awareness Balance via Dual Teachers

Navigating Data Heterogeneity in Federated Learning A Semi-Supervised Federated Object Detection