Abstract:Since labeled samples are typically scarce in real-world scenarios, self-supervised representation learning in time series is critical. Existing approaches mainly employ the contrastive learning framework, which automatically learns to understand similar and dissimilar data pairs. However, they are constrained by the request for cumbersome sampling policies and prior knowledge of constructing pairs. Also, few works have focused on effectively modeling temporal-spectral correlations to improve the capacity of representations. In this article, we propose the cross reconstruction transformer (CRT) to solve the aforementioned issues. CRT achieves time series representation learning through a cross-domain dropping-reconstruction task. Specifically, we obtain the frequency domain of the time series via the fast Fourier transform (FFT) and randomly drop certain patches in both time and frequency domains. Dropping is employed to maximally preserve the global context while masking leads to the distribution shift. Then a Transformer architecture is utilized to adequately discover the cross-domain correlations between temporal and spectral information through reconstructing data in both domains, which is called Dropped Temporal-Spectral Modeling. To discriminate the representations in global latent space, we propose instance discrimination constraint (IDC) to reduce the mutual information between different time series samples and sharpen the decision boundaries. Additionally, a specified curriculum learning (CL) strategy is employed to improve the robustness during the pretraining phase, which progressively increases the dropping ratio in the training process. We conduct extensive experiments to evaluate the effectiveness of the proposed method on multiple real-world datasets. Results show that CRT consistently achieves the best performance over existing methods by 2%-9%. The code is publicly available at https://github.com/BobZwr/Cross-Reconstruction-Transformer.

Learning to Embed Time Series Patches Independently

TFEformer: Temporal Feature Enhanced Transformer for Multivariate Time Series Forecasting

Masked Autoencoders for Point Cloud Self-supervised Learning.

A Time Series is Worth 64 Words: Long-term Forecasting with Transformers

TimeMAE: Self-Supervised Representations of Time Series with Decoupled Masked Autoencoders

Towards Unsupervised Time Series Representation Learning: A Decomposition Perspective

Extracting Spatio-Temporal Coupling Feature of Patches for Long-Term Multivariate Time Series Forecasting

Masked Autoencoders As Spatiotemporal Learners

Ti-MAE: Self-Supervised Masked Time Series Autoencoders

Point-MPP: Point Cloud Self-Supervised Learning from Masked Position Prediction

Bootstrap Masked Visual Modeling via Hard Patches Mining

LaT-PFN: A Joint Embedding Predictive Architecture for In-context Time-series Forecasting

Enhancing Time Series Forecasting: A Hierarchical Transformer with Probabilistic Decomposition Representation

HDMixer: Hierarchical Dependency with Extendable Patch for Multivariate Time Series Forecasting

Partial Channel Dependence with Channel Masks for Time Series Foundation Models

TS-MAE: A masked autoencoder for time series representation learning

Multi-Patch Prediction: Adapting Language Models for Time Series Representation Learning

Self-Supervised Time Series Representation Learning via Cross Reconstruction Transformer

Multi-Patch Prediction: Adapting LLMs for Time Series Representation Learning

Continuous-Time Linear Positional Embedding for Irregular Time Series Forecasting

TSLANet: Rethinking Transformers for Time Series Representation Learning