Abstract:The growth and diversity of machine learning applications motivate a rethinking of learning with mobile and edge devices. How can we address diverse client goals and learn with scarce heterogeneous data? While federated learning aims to address these issues, it has challenges hindering a unified solution. Large transformer models have been shown to work across a variety of tasks achieving remarkable few-shot adaptation. This raises the question: Can clients use a single general-purpose model, rather than custom models for each task, while obeying device and network constraints? In this work, we investigate pretrained transformers (PTF) to achieve these on-device learning goals and thoroughly explore the roles of model size and modularity, where the latter refers to adaptation through modules such as prompts or adapters. Focusing on federated learning, we demonstrate that: (1) Larger scale shrinks the accuracy gaps between alternative approaches and improves heterogeneity robustness. Scale allows clients to run more local SGD epochs which can significantly reduce the number of communication rounds. At the extreme, clients can achieve respectable accuracy locally highlighting the potential of fully-local learning. (2) Modularity, by design, enables $>$100$\times$ less communication in bits. Surprisingly, it also boosts the generalization capability of local adaptation methods and the robustness of smaller PTFs. Finally, it enables clients to solve multiple unrelated tasks simultaneously using a single PTF, whereas full updates are prone to catastrophic forgetting. These insights on scale and modularity motivate a new federated learning approach we call "You Only Load Once" (FedYolo): The clients load a full PTF model once and all future updates are accomplished through communication-efficient modules with limited catastrophic-forgetting, where each task is assigned to its own module.

FedTrans: Efficient Federated Learning via Multi-Model Transformation

FedDGP: Disentangling Global and Personal Models for Federated Learning

FedCross: Towards Accurate Federated Learning via Multi-Model Cross-Aggregation

FedConv: A Learning-on-Model Paradigm for Heterogeneous Federated Clients

Cooperative Multi-Model Training for Personalized Federated Learning over Heterogeneous Devices

Federated Learning with Additional Mechanisms on Clients to Reduce Communication Costs

FlexiFed: Personalized Federated Learning for Edge Clients with Heterogeneous Model Architectures

Improving Generalization and Personalization in Model-Heterogeneous Federated Learning

FedPartial: Enabling Model-Heterogeneous Federated Learning Via Partial Model Transmission and Aggregation

Towards Efficient Model-Heterogeneity Federated Learning for Large Models

FedHM: Efficient Federated Learning for Heterogeneous Models via Low-rank Factorization

Robust Multi-model Personalized Federated Learning Via Model Distillation

Towards Multi-modal Transformers in Federated Learning

FedYolo: Augmenting Federated Learning with Pretrained Transformers

FedMPT: Federated Learning for Multiple Personalized Tasks Over Mobile Computing

Federated mutual learning

Enhancing Federated Learning by One-Shot Transferring of Intermediate Features from Clients

MGFL: Multi-granularity Federated Learning in Edge Computing Systems

No One Left Behind: Inclusive Federated Learning over Heterogeneous Devices

Towards Fairer and More Efficient Federated Learning via Multidimensional Personalized Edge Models

Model-heterogeneous Federated Learning with Partial Model Training