Abstract:The growth and diversity of machine learning applications motivate a rethinking of learning with mobile and edge devices. How can we address diverse client goals and learn with scarce heterogeneous data? While federated learning aims to address these issues, it has challenges hindering a unified solution. Large transformer models have been shown to work across a variety of tasks achieving remarkable few-shot adaptation. This raises the question: Can clients use a single general-purpose model, rather than custom models for each task, while obeying device and network constraints? In this work, we investigate pretrained transformers (PTF) to achieve these on-device learning goals and thoroughly explore the roles of model size and modularity, where the latter refers to adaptation through modules such as prompts or adapters. Focusing on federated learning, we demonstrate that: (1) Larger scale shrinks the accuracy gaps between alternative approaches and improves heterogeneity robustness. Scale allows clients to run more local SGD epochs which can significantly reduce the number of communication rounds. At the extreme, clients can achieve respectable accuracy locally highlighting the potential of fully-local learning. (2) Modularity, by design, enables $>$100$\times$ less communication in bits. Surprisingly, it also boosts the generalization capability of local adaptation methods and the robustness of smaller PTFs. Finally, it enables clients to solve multiple unrelated tasks simultaneously using a single PTF, whereas full updates are prone to catastrophic forgetting. These insights on scale and modularity motivate a new federated learning approach we call "You Only Load Once" (FedYolo): The clients load a full PTF model once and all future updates are accomplished through communication-efficient modules with limited catastrophic-forgetting, where each task is assigned to its own module.

Fed-urlBERT: Client-side Lightweight Federated Transformers for URL Threat Analysis

FEDBFPT: an Efficient Federated Learning Framework for BERT Further Pre-Training

Malicious URL Detection via Pretrained Language Model Guided Multi-Level Feature Attention Network

FedBERT: when Federated Learning Meets Pre-training

TransURL

TransURL: Improving Malicious URL Detection with Multi-Layer Transformer Encoding and Multi-Scale Pyramid Features

URLBERT:A Contrastive and Adversarial Pre-trained Model for URL Classification

FedYolo: Augmenting Federated Learning with Pretrained Transformers

PhishTransformer: A Novel Approach to Detect Phishing Attacks Using URL Collection and Transformer

A Large-Scale Pretrained Deep Model for Phishing URL Detection

Federated Split BERT for Heterogeneous Text Classification

PyraTrans: Attention-Enriched Pyramid Transformer for Malicious URL Detection

DomURLs_BERT: Pre-trained BERT-based Model for Malicious Domains and URLs Detection and Classification

Enabling Foundation Models: A Distributed Collaboration Framework Based on Graph Federated Learning

Efficient Federated Intrusion Detection in 5G ecosystem using optimized BERT-based model

TCURL: Exploring Hybrid Transformer and Convolutional Neural Network on Phishing URL Detection

FedTrojan: Corrupting Federated Learning Via Zero-Knowledge Federated Trojan Attacks

BR-FEEL: A Backdoor Resilient Approach for Federated Edge Learning with Fragment-Sharing

Federated transfer learning for disaster classification in social computing networks

FedTP: Federated Learning by Transformer Personalization

PyraTrans: Learning Attention-Enriched Multi-Scale Pyramid Network from Pre-Trained Transformers for Effective Malicious URL Detection