Abstract:The growth and diversity of machine learning applications motivate a rethinking of learning with mobile and edge devices. How can we address diverse client goals and learn with scarce heterogeneous data? While federated learning aims to address these issues, it has challenges hindering a unified solution. Large transformer models have been shown to work across a variety of tasks achieving remarkable few-shot adaptation. This raises the question: Can clients use a single general-purpose model, rather than custom models for each task, while obeying device and network constraints? In this work, we investigate pretrained transformers (PTF) to achieve these on-device learning goals and thoroughly explore the roles of model size and modularity, where the latter refers to adaptation through modules such as prompts or adapters. Focusing on federated learning, we demonstrate that: (1) Larger scale shrinks the accuracy gaps between alternative approaches and improves heterogeneity robustness. Scale allows clients to run more local SGD epochs which can significantly reduce the number of communication rounds. At the extreme, clients can achieve respectable accuracy locally highlighting the potential of fully-local learning. (2) Modularity, by design, enables $>$100$\times$ less communication in bits. Surprisingly, it also boosts the generalization capability of local adaptation methods and the robustness of smaller PTFs. Finally, it enables clients to solve multiple unrelated tasks simultaneously using a single PTF, whereas full updates are prone to catastrophic forgetting. These insights on scale and modularity motivate a new federated learning approach we call "You Only Load Once" (FedYolo): The clients load a full PTF model once and all future updates are accomplished through communication-efficient modules with limited catastrophic-forgetting, where each task is assigned to its own module.

Federated Transformer Hawkes Processes for Distributed Event Sequence Prediction

TFEformer: Temporal Feature Enhanced Transformer for Multivariate Time Series Forecasting

Hidformer: Hierarchical Dual-Tower Transformer Using Multi-Scale Mergence for Long-Term Time Series Forecasting

FedMTPP: Federated Multivariate Temporal Point Processes for Distributed Event Sequence Forecasting

GTHP: a novel graph transformer Hawkes process for spatiotemporal event prediction

Tri-Transformer Hawkes Process: Three Heads are better than one

Enhancing Time Series Forecasting: A Hierarchical Transformer with Probabilistic Decomposition Representation

A Spatio-Temporal Transformer Network for Human Motion Prediction in Human-Robot Collaboration

Transformer-enhanced Hawkes process with decoupling training for information cascade prediction

Transformers with Attentive Federated Aggregation for Time Series Stock Forecasting

Interpretable Transformer Hawkes Processes: Unveiling Complex Interactions in Social Networks

Temporal Attention Augmented Transformer Hawkes Process

RoTHP: Rotary Position Embedding-based Transformer Hawkes Process

Tackling Data Heterogeneity in Federated Time Series Forecasting

The Prospect of Enhancing Large-Scale Heterogeneous Federated Learning with Transformers

Transformer Embeddings of Irregularly Spaced Events and Their Participants

Deep Transfer Hashing for Adaptive Learning on Federated Streaming Data

Mamba Hawkes Process

Federated Transformer: Multi-Party Vertical Federated Learning on Practical Fuzzily Linked Data

FedYolo: Augmenting Federated Learning with Pretrained Transformers

UniTST: Effectively Modeling Inter-Series and Intra-Series Dependencies for Multivariate Time Series Forecasting