Abstract:Self-supervised learning has garnered significant attention for its ability to learn meaningful representations. Recent advancements have introduced self-supervised methods for time series forecasting. However, these efforts have faced limitations due to two primary drawbacks. Firstly, these approaches often borrow techniques from vision and language domains without adequately addressing the unique temporal dependencies inherent in time series data. Secondly, time series often show that the distribution shifts over time, which makes accurate forecasting challenging. In response to these issues, we propose TempSSL, a self-supervised learning framework designed for time series forecasting. TempSSL divides the time series data into context (history data) and target (future data), employing two pre-training strategies: (1) Temporal Masked Modeling (TMM) designed to capture temporal dependencies by reconstructing future time series based on historical context; (2) Temporal Contrastive Learning (TCL) employs context and target as positive samples to enhance discriminative representations and mitigate distribution shifts within the time series. TempSSL's innovation lies in two key aspects. Firstly, it underscores the importance of temporal dependencies for time series forecasting by designing specific pre-training tasks. Secondly, it effectively integrates contrastive learning and masked modeling, leveraging their respective strengths to develop time series representation with strong instance discriminability and local perceptibility. Extensive experiments across seven widely used benchmark datasets demonstrate that TempSSL consistently outperforms existing self-supervised and end-to-end forecasting methods, achieving improvements ranging from 1.92% ∼ 78.12%. Additionally, TempSSL's practical effectiveness is further demonstrated through successful application in natural gas demand forecasting.

Revisiting the Temporal Modeling in Spatio-Temporal Predictive Learning under A Unified View

Interpretable and Generalizable Spatiotemporal Predictive Learning with Disentangled Consistency

Temporal Attention Unit: Towards Efficient Spatiotemporal Predictive Learning

Wavelet-Driven Spatiotemporal Predictive Learning: Bridging Frequency and Time Variations

SimVP: Towards Simple yet Powerful Spatiotemporal Predictive Learning

Triplet Attention Transformer for Spatiotemporal Predictive Learning

OpenSTL: A Comprehensive Benchmark of Spatio-Temporal Predictive Learning

Enhancing spatiotemporal predictive learning: an approach with nested attention module

A Unified Replay-based Continuous Learning Framework for Spatio-Temporal Prediction on Streaming Data

Towards Unifying Diffusion Models for Probabilistic Spatio-Temporal Graph Learning

Is Single Enough? A Joint Spatiotemporal Feature Learning Framework for Multivariate Time Series Prediction

PredRNN: Recurrent Neural Networks for Predictive Learning using Spatiotemporal LSTMs

Rethinking self-supervised learning for time series forecasting: A temporal perspective

Self-Attention ConvLSTM for Spatiotemporal Prediction

Cross Space and Time: A Spatio-Temporal Unitized Model for Traffic Flow Forecasting

Spatiotemporal Attention for Multivariate Time Series Prediction and Interpretation

Marked Temporal Dynamics Modeling based on Recurrent Neural Network

PredRNN: A Recurrent Neural Network for Spatiotemporal Predictive Learning

Exploring the Generalizability of Spatio-Temporal Traffic Prediction: Meta-Modeling and an Analytic Framework

UniST: A Prompt-Empowered Universal Model for Urban Spatio-Temporal Prediction

PredRNN++: Towards A Resolution of the Deep-in-Time Dilemma in Spatiotemporal Predictive Learning