Abstract:Since labeled samples are typically scarce in real-world scenarios, self-supervised representation learning in time series is critical. Existing approaches mainly employ the contrastive learning framework, which automatically learns to understand similar and dissimilar data pairs. However, they are constrained by the request for cumbersome sampling policies and prior knowledge of constructing pairs. Also, few works have focused on effectively modeling temporal-spectral correlations to improve the capacity of representations. In this article, we propose the cross reconstruction transformer (CRT) to solve the aforementioned issues. CRT achieves time series representation learning through a cross-domain dropping-reconstruction task. Specifically, we obtain the frequency domain of the time series via the fast Fourier transform (FFT) and randomly drop certain patches in both time and frequency domains. Dropping is employed to maximally preserve the global context while masking leads to the distribution shift. Then a Transformer architecture is utilized to adequately discover the cross-domain correlations between temporal and spectral information through reconstructing data in both domains, which is called Dropped Temporal-Spectral Modeling. To discriminate the representations in global latent space, we propose instance discrimination constraint (IDC) to reduce the mutual information between different time series samples and sharpen the decision boundaries. Additionally, a specified curriculum learning (CL) strategy is employed to improve the robustness during the pretraining phase, which progressively increases the dropping ratio in the training process. We conduct extensive experiments to evaluate the effectiveness of the proposed method on multiple real-world datasets. Results show that CRT consistently achieves the best performance over existing methods by 2%-9%. The code is publicly available at https://github.com/BobZwr/Cross-Reconstruction-Transformer.

Time Series Prediction with Interpretable Data Reconstruction

Interpretable and Generalizable Spatiotemporal Predictive Learning with Disentangled Consistency

Interpretable System Identification and Long-term Prediction on Time-Series Data

Enhancing Time Series Forecasting: A Hierarchical Transformer with Probabilistic Decomposition Representation

Discovering Predictable Latent Factors for Time Series Forecasting

Deep Learning with Long Short-Term Memory for Time Series Prediction

Time Series Prediction Based on LSTM-Attention-LSTM Model

Time Series Prediction Method Based on Variant LSTM Recurrent Neural Network

In-context Time Series Predictor

A novel general-purpose hybrid model for time series forecasting

Revisiting the Efficacy of Signal Decomposition in AI-based Time Series Prediction

ExoTST: Exogenous-Aware Temporal Sequence Transformer for Time Series Prediction

Interpretation of Time-Series Deep Models: A Survey

DeepLINK-T: deep learning inference for time series data using knockoffs and LSTM

Learning task-agnostic and interpretable subsequence-based representation of time series and its applications in fMRI analysis

TSINR: Capturing Temporal Continuity via Implicit Neural Representations for Time Series Anomaly Detection

Fuzzy inference-based LSTM for long-term time series prediction

Reinforced Decoder: Towards Training Recurrent Neural Networks for Time Series Forecasting

Self-Interpretable Time Series Prediction with Counterfactual Explanations

DeLELSTM: Decomposition-based Linear Explainable LSTM to Capture Instantaneous and Long-term Effects in Time Series

Self-Supervised Time Series Representation Learning via Cross Reconstruction Transformer