Abstract:The growth and diversity of machine learning applications motivate a rethinking of learning with mobile and edge devices. How can we address diverse client goals and learn with scarce heterogeneous data? While federated learning aims to address these issues, it has challenges hindering a unified solution. Large transformer models have been shown to work across a variety of tasks achieving remarkable few-shot adaptation. This raises the question: Can clients use a single general-purpose model, rather than custom models for each task, while obeying device and network constraints? In this work, we investigate pretrained transformers (PTF) to achieve these on-device learning goals and thoroughly explore the roles of model size and modularity, where the latter refers to adaptation through modules such as prompts or adapters. Focusing on federated learning, we demonstrate that: (1) Larger scale shrinks the accuracy gaps between alternative approaches and improves heterogeneity robustness. Scale allows clients to run more local SGD epochs which can significantly reduce the number of communication rounds. At the extreme, clients can achieve respectable accuracy locally highlighting the potential of fully-local learning. (2) Modularity, by design, enables $>$100$\times$ less communication in bits. Surprisingly, it also boosts the generalization capability of local adaptation methods and the robustness of smaller PTFs. Finally, it enables clients to solve multiple unrelated tasks simultaneously using a single PTF, whereas full updates are prone to catastrophic forgetting. These insights on scale and modularity motivate a new federated learning approach we call "You Only Load Once" (FedYolo): The clients load a full PTF model once and all future updates are accomplished through communication-efficient modules with limited catastrophic-forgetting, where each task is assigned to its own module.

Scaling Language Model Size in Cross-Device Federated Learning

Scaling Federated Learning for Fine-tuning of Large Language Models

Efficient Language Model Architectures for Differentially Private Federated Learning

Efficient Federated Finetuning of Tiny Transformers with Resource-Constrained Devices

Scaling Laws for Neural Language Models

FedYolo: Augmenting Federated Learning with Pretrained Transformers

Language models scale reliably with over-training and on downstream tasks

Federated Learning of Large ASR Models in the Real World

Scaling Laws for Multilingual Language Models

Federated Full-Parameter Tuning of Billion-Sized Language Models with Communication Cost under 18 Kilobytes

Scaling Laws with Vocabulary: Larger Models Deserve Larger Vocabularies

Communication-Efficient and Tensorized Federated Fine-Tuning of Large Language Models

Optimizing Distributed Training on Frontier for Large Language Models

Conquering the Communication Constraints to Enable Large Pre-Trained Models in Federated Learning

Cloud-Device Collaborative Learning for Multimodal Large Language Models

Scaling laws for language encoding models in fMRI

Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity

Train Large, Then Compress: Rethinking Model Size for Efficient Training and Inference of Transformers

FedPT: Federated Proxy-Tuning of Large Language Models on Resource-Constrained Edge Devices

Computational Bottlenecks of Training Small-scale Large Language Models

Scaling Laws for Downstream Task Performance of Large Language Models