Abstract:Multivariate time-series data in numerous real-world applications (e.g., healthcare and industry) are informative but challenging due to the lack of labels and high dimensionality. Recent studies in self-supervised learning have shown their potential in learning rich representations without relying on labels, yet they fall short in learning disentangled embeddings and addressing issues of inductive bias (e.g., transformation-invariance). To tackle these challenges, we propose TimeDRL, a generic multivariate time-series representation learning framework with disentangled dual-level embeddings. TimeDRL is characterized by three novel features: (i) disentangled derivation of timestamp-level and instance-level embeddings from patched time-series data using a [CLS] token strategy; (ii) utilization of timestamp-predictive and instance-contrastive tasks for disentangled representation learning, with the former optimizing timestamp-level embeddings with predictive loss, and the latter optimizing instance-level embeddings with contrastive loss; and (iii) avoidance of augmentation methods to eliminate inductive biases, such as transformation-invariance from cropping and masking. Comprehensive experiments on 6 time-series forecasting datasets and 5 time-series classification datasets have shown that TimeDRL consistently surpasses existing representation learning approaches, achieving an average improvement of forecasting by 58.02% in MSE and classification by 1.48% in accuracy. Furthermore, extensive ablation studies confirmed the relative contribution of each component in TimeDRL's architecture, and semi-supervised learning evaluations demonstrated its effectiveness in real-world scenarios, even with limited labeled data. The code is available at <a class="link-external link-https" href="https://github.com/blacksnail789521/TimeDRL" rel="external noopener nofollow">this https URL</a>.

DRN: Bringing Greedy Layer-Wise Training into Time Dimension

UDRN: Unified Dimensional Reduction Neural Network for Feature Selection and Feature Projection

DMT-EV: an Explainable Deep Network for Dimension Reduction.

UDRN: Unified Dimensional Reduction Neural Network for Feature Selection and Feature Projection.

Residual Recurrent Neural Networks for Learning Sequential Representations.

DRRNets: Dynamic Recurrent Routing Via Low-Rank Regularization in Recurrent Neural Networks.

Probabilistic Sequential Network for Deep Learning of Complex Process Data and Soft Sensor Application

CDRGN-SDE: Cross-Dimensional Recurrent Graph Network with neural Stochastic Differential Equation for temporal knowledge graph embedding

Deep Differential Recurrent Neural Networks

Learning various length dependence by dual recurrent neural networks

Deep Tabular Data Modeling with Dual-Route Structure-Adaptive Graph Networks

A Deep Learning Approach to Link Prediction in Dynamic Networks

Deep Recurrent Stochastic Configuration Networks for Modelling Nonlinear Dynamic Systems

Predrnn++: Towards a resolution of the deep-in-time dilemma in spatiotemporal predictive learning

Predrnn: A recurrent neural network for spatiotemporal predictive learning

Deep Learning-Based Dynamic Community Discovery

TimeDRL: Disentangled Representation Learning for Multivariate Time-Series

Sparse Deep Learning for Time Series Data: Theory and Applications

A Novel Domain Adaptive Deep Recurrent Network for Multivariate Time Series Prediction

Identifying Brain Networks at Multiple Time Scales via Deep Recurrent Neural Network

DeepLINK-T: deep learning inference for time series data using knockoffs and LSTM