Abstract:Self-supervised learning (SSL) is capable of learning remarkable representations from centrally available data. Recent works further implement federated learning with SSL to learn from rapidly growing decentralized unlabeled images (e.g., from cameras and phones), often resulted from privacy constraints. Extensive attention has been paid to SSL approaches based on Siamese networks. However, such an effort has not yet revealed deep insights into various fundamental building blocks for the federated self-supervised learning (FedSSL) architecture. We aim to fill in this gap via in-depth empirical study and propose a new method to tackle the non-independently and identically distributed (non-IID) data problem of decentralized data. Firstly, we introduce a generalized FedSSL framework that embraces existing SSL methods based on Siamese networks and presents flexibility catering to future methods. In this framework, a server coordinates multiple clients to conduct SSL training and periodically updates local models of clients with the aggregated global model. Using the framework, our study uncovers unique insights of FedSSL: 1) stop-gradient operation, previously reported to be essential, is not always necessary in FedSSL; 2) retaining local knowledge of clients in FedSSL is particularly beneficial for non-IID data. Inspired by the insights, we then propose a new approach for model update, Federated Divergence-aware Exponential Moving Average update (FedEMA). FedEMA updates local models of clients adaptively using EMA of the global model, where the decay rate is dynamically measured by model divergence. Extensive experiments demonstrate that FedEMA outperforms existing methods by 3-4% on linear evaluation. We hope that this work will provide useful insights for future research.

FGSS: Federated Global Self-Supervised Framework for Large-Scale Unlabeled Data

A Framework for Self-Supervised Federated Domain Adaptation

Federated Semi-supervised Learning with Dual Regulator

GDST: Global Distillation Self-Training for Semi-Supervised Federated Learning

Dual Class-Aware Contrastive Federated Semi-Supervised Learning

TabFedSL: A Self-Supervised Approach to Labeling Tabular Data in Federated Learning Environments

FedGL: Federated graph learning framework with global self-supervision

Clients Help Clients: Alternating Collaboration for Semi-Supervised Federated Learning

Class Balanced Adaptive Pseudo Labeling for Federated Semi-Supervised Learning

RSCFed: Random Sampling Consensus Federated Semi-supervised Learning

Federated Semi-Supervised Learning with Annotation Heterogeneity

Federated Self-Supervised Contrastive Learning via Ensemble Similarity Distillation

Rethinking Semi-Supervised Federated Learning: How to co-train fully-labeled and fully-unlabeled client imaging data

Divergence-aware Federated Self-Supervised Learning

Rethinking the Representation in Federated Unsupervised Learning with Non-IID Data

Non-IID always Bad? Semi-Supervised Heterogeneous Federated Learning with Local Knowledge Enhancement

Navigating Data Heterogeneity in Federated Learning A Semi-Supervised Federated Object Detection

Distributed Unsupervised Visual Representation Learning with Fused Features

Federated Semi-Supervised Learning with Local and Global Updating Frequency Optimization

SemiFed: Semi-supervised Federated Learning with Consistency and Pseudo-Labeling

Federated Learning from Only Unlabeled Data with Class-Conditional-Sharing Clients