Abstract:Self-supervised representation learning of Multivariate Time Series (MTS) is a challenging task and attracts increasing research interests in recent years. Many previous works focus on the pretext task of self-supervised learning and usually neglect the complex problem of MTS encoding, leading to unpromising results. In this paper, we tackle this challenge from two aspects: encoder and pretext task, and propose a unified channel-aware self-supervised learning framework CaSS. Specifically, we first design a new Transformer-based encoder Channel-aware Transformer (CaT) to capture the complex relationships between different time channels of MTS. Second, we combine two novel pretext tasks Next Trend Prediction (NTP) and Contextual Similarity (CS) for the self-supervised representation learning with our proposed encoder. Extensive experiments are conducted on several commonly used benchmark datasets. The experimental results show that our framework achieves new state-of-the-art comparing with previous self-supervised MTS representation learning methods (up to +7.70\% improvement on LSST dataset) and can be well applied to the downstream MTS classification.

What problem does this paper attempt to address?

The problem that this paper attempts to solve is self - supervised representation learning of multivariate time series (MTS). Specifically, the paper addresses two main problems existing in existing methods when dealing with MTS: 1. **Insufficient encoder design**: Most previous works only focus on feature fusion in the time dimension and ignore the modeling of complex relationships between different channels. This results in the encoder being unable to comprehensively capture the overall features of MTS. 2. **Single pre - training task design**: Traditional self - supervised learning methods mostly rely on pre - training tasks based on the time dimension and lack the effective use of features in the channel dimension, thus affecting the learning effect. To solve these problems, the author proposes a unified, channel - aware self - supervised learning framework CaSS (Channel - aware Self - supervised), which is improved in terms of the encoder and pre - training tasks. The specific contributions are as follows: - **Propose a new Transformer - based encoder CaT (Channel - aware Transformer)**, which can effectively integrate features in the time dimension and the channel dimension simultaneously. - **Design two novel pre - training tasks**: - **Next Trend Prediction (NTP)**: For the first time, predict future trends from the perspective of the channel dimension rather than specific values. Experiments show that predicting trends is more efficient than regressing specific values. - **Contextual Similarity (CS)**: By maximizing the similarity between similar samples and combining data augmentation strategies, the learning effect is further improved. Through extensive experiments on multiple commonly - used benchmark datasets, it is proved that the CaSS framework has achieved significant performance improvement in multivariate time series classification tasks compared with existing methods (for example, the highest improvement of 7.70% on the LSST dataset). In addition, this framework also performs well in few - shot learning, demonstrating its strong generalization ability and robustness. In summary, this paper aims to improve the self - supervised representation learning effect of multivariate time series by improving the encoder design and pre - training tasks, so as to be better applied to downstream classification tasks.

CaSS: A Channel-aware Self-supervised Representation Learning Framework for Multivariate Time Series Classification

Multi-view Self-Supervised Contrastive Learning for Multivariate Time Series

DCS: Debiased Contrastive Learning with Weak Supervision for Time Series Classification

A Deep Multi-Task Representation Learning Method for Time Series Classification and Retrieval.

MICOS: Mixed supervised contrastive learning for multivariate time series classification

Self-supervised Contrastive Representation Learning for Semi-supervised Time-Series Classification

Multi-view self-supervised learning for multivariate variable-channel time series

Unsupervised Time-Series Representation Learning with Iterative Bilinear Temporal-Spectral Fusion.

Attention Based CNN-LSTM Network for Anomaly Pattern Classification of Multivariate Time Series

Multi-scale Self-Supervised Representation Learning with Temporal Alignment for Multi-Rate Time Series Modeling

Multi-Task Self-Supervised Time-Series Representation Learning

FormerTime: Hierarchical Multi-Scale Representations for Multivariate Time Series Classification

Multiscale spatial‐temporal transformer with consistency representation learning for multivariate time series classification

MT-SLVR: Multi-Task Self-Supervised Learning for Transformation In(Variant) Representations

Time-Series Representation Learning via Temporal and Contextual Contrasting

TCLN: A Transformer-based Conv-LSTM Network for Multivariate Time Series Forecasting

Multitask Learning for Time Series Data with 2D Convolution

DSDCLNet: Dual-stream encoder and dual-level contrastive learning network for supervised multivariate time series classification

A Shapelet-based Framework for Unsupervised Multivariate Time Series Representation Learning