Abstract:A foundation model is a machine learning model trained on a large and diverse set of data, typically using self-supervised learning-based pre-training techniques, that can be adapted to various downstream tasks. However, current research on time series pre-training has mostly focused on models pre-trained solely on data from a single domain, resulting in a lack of knowledge about other types of time series. However, current research on time series pre-training has predominantly focused on models trained exclusively on data from a single domain. As a result, these models possess domain-specific knowledge that may not be easily transferable to time series from other domains. In this paper, we aim to develop an effective time series foundation model by leveraging unlabeled samples from multiple domains. To achieve this, we repurposed the publicly available UCR Archive and evaluated four existing self-supervised learning-based pre-training methods, along with a novel method, on the datasets. We tested these methods using four popular neural network architectures for time series to understand how the pre-training methods interact with different network designs. Our experimental results show that pre-training improves downstream classification tasks by enhancing the convergence of the fine-tuning process. Furthermore, we found that the proposed pre-training method, when combined with the Transformer model, outperforms the alternatives.

What problem does this paper attempt to address?

### What problems does this paper attempt to solve? This paper aims to solve the problems of construction and application of foundation models in the field of time - series data. Specifically, current time - series pre - training research mainly focuses on data in a single domain, resulting in these models having domain - specific knowledge and being difficult to transfer to time - series tasks in other domains. Therefore, the author proposes and explores the following key issues: 1. **Effectiveness of pre - training on multi - domain data**: - Research whether pre - training on time - series data in multiple domains can improve the performance of downstream single - domain classification tasks. 2. **The most effective neural network architecture**: - Explore which neural network architecture (such as LSTM, GRU, ResNet, Transformer) is most suitable for constructing time - series foundation models. 3. **The most effective self - supervised learning method**: - Compare several existing self - supervised learning methods and propose a new self - supervised learning method (TimeCLR) to determine which method is most suitable for constructing time - series foundation models. ### Main contributions of the paper - **Dataset reuse and expansion**: The author reuses the publicly available UCR Archive dataset and expands it to include time - series data from multiple domains to support multi - domain pre - training. - **Experimental verification**: Through extensive experimental verification, the superior performance of the foundation model pre - trained with multi - domain data in downstream tasks is verified. In particular, the pre - trained model shows a smoother convergence curve in the fine - tuning stage, which helps to improve the final performance. - **Proposal of a new method**: A new self - supervised learning method TimeCLR is proposed, which combines multiple time - series data augmentation techniques and significantly improves the pre - training effect. Especially when combined with the Transformer architecture, it performs particularly well. ### Experimental results The experimental results show that the foundation model pre - trained with multi - domain data outperforms the non - pre - trained model in most downstream tasks. In particular, the TimeCLR method performs best in all test configurations, especially when combined with the Transformer architecture, achieving the best or second - best performance in almost 93% of downstream tasks. In conclusion, this paper proves the effectiveness of pre - training on multi - domain time - series data through systematic research and experiments, and provides important references and directions for future research on time - series foundation models.

Toward a Foundation Model for Time Series Data

A Survey of Time Series Foundation Models: Generalizing Time Series Representation with Large Language Model

Foundation Models for Time Series Analysis: A Tutorial and Survey

Federated Foundation Models on Heterogeneous Time Series

TimeDiT: General-purpose Diffusion Transformers for Time Series Foundation Model

Specialized Foundation Models Struggle to Beat Supervised Baselines

Measuring Pre-training Data Quality without Labels for Time Series Foundation Models

MOMENT: A Family of Open Time-series Foundation Models

On Foundation Models for Dynamical Systems from Purely Synthetic Data

Self-Supervised Pre-training for Time Series Classification

In-Context Fine-Tuning for Time-Series Foundation Models

FedAWS: A Federated Tuning Approach with Adaptive Weight Shrinking for Pre-trained Foundation Models

Time Series Representation Models

FoundTS: Comprehensive and Unified Benchmarking of Foundation Models for Time Series Forecasting

One Fits All:Power General Time Series Analysis by Pretrained LM

NuTime: Numerically Multi-Scaled Embedding for Large-Scale Time-Series Pretraining

Revisiting Data Analysis with Pre-trained Foundation Models

Foundation Model for Endoscopy Video Analysis via Large-scale Self-supervised Pre-train

Toward Digital Twin: Leveraging Pre-training Approaches for Multivariate Time Series Forecasting

Time-FFM: Towards LM-Empowered Federated Foundation Model for Time Series Forecasting