Abstract:Training a general-purpose time series foundation models with robust generalization capabilities across diverse applications from scratch is still an open challenge. Efforts are primarily focused on fusing cross-domain time series datasets to extract shared subsequences as tokens for training models on Transformer architecture. However, due to significant statistical heterogeneity across domains, this cross-domain fusing approach doesn't work effectively as the same as fusing texts and images. To tackle this challenge, this paper proposes a novel federated learning approach to address the heterogeneity in time series foundation models training, namely FFTS. Specifically, each data-holding organization is treated as an independent client in a collaborative learning framework with federated settings, and then many client-specific local models will be trained to preserve the unique characteristics per dataset. Moreover, a new regularization mechanism will be applied to both client-side and server-side, thus to align the shared knowledge across heterogeneous datasets from different domains. Extensive experiments on benchmark datasets demonstrate the effectiveness of the proposed federated learning approach. The newly learned time series foundation models achieve superior generalization capabilities on cross-domain time series analysis tasks, including forecasting, imputation, and anomaly detection.

What problem does this paper attempt to address?

The problem that this paper attempts to solve is the challenge of training a time - series foundation model (TSFM) with strong generalization ability on heterogeneous time - series data. Specifically, due to the statistical heterogeneity in different domains, traditional cross - domain fusion methods are not effective in processing time - series data. These heterogeneities lead to inconsistent convergence speeds of the model in different domains and the inability to effectively utilize heterogeneous data with similar patterns. In addition, the contextual meanings in different domains often have unique interpretations at different time scales, which further increases the difficulty of data fusion and training. To solve these problems, the paper proposes a new federated learning method - FFTS (Federated Foundation Models on Heterogeneous Time Series), aiming to address the heterogeneity problem in time - series foundation model training through the federated learning framework. The main features of FFTS include: 1. **Federated learning framework**: Each organization holding data is regarded as an independent client and conducts collaborative learning in a federated setting. Each client trains a local model to retain the unique characteristics of its respective dataset, and then the server - side aggregates these local models to form a global model. 2. **Regularization mechanism**: A new regularization mechanism is introduced and applied to the client - side and server - side to align the shared knowledge among heterogeneous datasets from different domains. 3. **Adaptive Trend - aware Module (ATM)**: An adaptive trend - aware module is designed to identify similar patterns in heterogeneous sequences, thereby reducing the impact of heterogeneity on the global model. 4. **Unified downstream adaptation architecture**: A unified downstream task adaptation architecture is provided, which supports multiple downstream tasks such as prediction, imputation, and anomaly detection. Through these methods, FFTS can train a foundation model with stronger generalization ability on heterogeneous time - series data and achieve superior performance in cross - domain time - series analysis tasks.

Federated Foundation Models on Heterogeneous Time Series

Time-FFM: Towards LM-Empowered Federated Foundation Model for Time Series Forecasting

Tackling Data Heterogeneity in Federated Time Series Forecasting

Federated Time Series Generation on Feature and Temporally Misaligned Data

FedAWS: A Federated Tuning Approach with Adaptive Weight Shrinking for Pre-trained Foundation Models

Spatio-temporal Heterogeneous Federated Learning for Time Series Classification with Multi-view Orthogonal Training

Federated Generative Learning with Foundation Models

Learning to Generalize in Heterogeneous Federated Networks

Toward a Foundation Model for Time Series Data

Leveraging Foundation Models to Improve Lightweight Clients in Federated Learning

FedDAT: An Approach for Foundation Model Finetuning in Multi-Modal Heterogeneous Federated Learning

A Survey of Time Series Foundation Models: Generalizing Time Series Representation with Large Language Model

ChronusFed: Reinforcement-Based Adaptive Partial Training for Heterogeneous Federated Learning

FoundTS: Comprehensive and Unified Benchmarking of Foundation Models for Time Series Forecasting

A federated large language model for long-term time series forecasting

Advances and Open Challenges in Federated Foundation Models

Cross-Training with Multi-View Knowledge Fusion for Heterogenous Federated Learning

FedMS: Federated Learning with Mixture of Sparsely Activated Foundations Models

Federated Foundation Models: Privacy-Preserving and Collaborative Learning for Large Models

Towards Efficient Model-Heterogeneity Federated Learning for Large Models

Federated Knowledge Transfer for Heterogeneous Visual Models.