Abstract:Self-supervised learning has garnered increasing attention in time series analysis for benefiting various downstream tasks and reducing reliance on labeled data. Despite its effectiveness, existing methods often struggle to comprehensively capture both long-term dynamic evolution and subtle local patterns in a unified manner. In this work, we propose TimeDART, a novel self-supervised time series pre-training framework that unifies two powerful generative paradigms to learn more transferable representations. Specifically, we first employ a causal Transformer encoder, accompanied by a patch-based embedding strategy, to model the evolving trends from left to right. Building on this global modeling, we further introduce a denoising diffusion process to capture fine-grained local patterns through forward diffusion and reverse denoising. Finally, we optimize the model in an autoregressive manner. As a result, TimeDART effectively accounts for both global and local sequence features in a coherent way. We conduct extensive experiments on public datasets for time series forecasting and classification. The experimental results demonstrate that TimeDART consistently outperforms previous compared methods, validating the effectiveness of our approach. Our code is available at <a class="link-external link-https" href="https://github.com/Melmaphother/TimeDART" rel="external noopener nofollow">this https URL</a>.

What problem does this paper attempt to address?

### What problem does this paper attempt to solve? This paper aims to solve an important problem in time - series analysis: **How to simultaneously capture the long - term dynamic evolution and subtle local patterns of time - series data within the self - supervised learning framework**. Specifically, existing self - supervised methods often struggle to comprehensively handle these two characteristics, resulting in less - than - ideal performance of the model in downstream tasks (such as prediction and classification). #### Main challenges 1. **Long - term dynamic evolution**: Time - series data usually has long - term trends and periodic changes, and existing methods have limitations in capturing these global features. 2. **Subtle local patterns**: Time - series also contains many subtle local changes, which are very important for certain applications (such as anomaly detection), but existing methods also face challenges in capturing these details. 3. **Dependence on labeled data**: Traditional supervised learning methods rely highly on labeled data, while a large amount of unlabeled time - series data in practical applications has not been fully utilized. #### Proposed solutions To solve the above problems, the authors propose **TimeDART**, a new self - supervised time - series pre - training framework that combines two powerful generation paradigms: - **Causal Transformer encoder**: Used to model the left - to - right time - evolution trend. - **Denoising diffusion process**: Captures fine - grained local patterns through forward diffusion and reverse denoising. In addition, TimeDART adopts an autoregressive optimization method to ensure that the model can effectively integrate global and local features. In this way, TimeDART can significantly outperform existing methods on a variety of public datasets, verifying its effectiveness in time - series prediction and classification tasks. #### Presentation of formulas The formulas involved in the paper include: - Noise addition in the forward diffusion process: \[ q(x_s^j|x_{s - 1}^j)=\mathcal{N}(x_s^j; \sqrt{\alpha(s)}x_{s - 1}^j,(1 - \alpha(s))I) \] - Reverse denoising process: \[ z_j^{\text{out}} = g(\hat{z}_j^{\text{in}}, f(z_{1:j - 1}^{\text{in}})),\quad1\leq j\leq N \] - Self - supervised optimization objective: \[ L_{\text{diff}}=\sum_{j = 1}^{N}\mathbb{E}_{\epsilon,q(x_0^j)}\left[\|x_0^j - g(\hat{z}_j^{\text{in}}, f(z_{1:j - 1}^{\text{in}}))\|^2\right] \] These formulas show how TimeDART learns more transferable time - series representations through the diffusion and denoising processes and autoregressive optimization.

TimeDART: A Diffusion Autoregressive Transformer for Self-Supervised Time Series Representation

TFEformer: Temporal Feature Enhanced Transformer for Multivariate Time Series Forecasting

TimeDRL: Disentangled Representation Learning for Multivariate Time-Series

Multi-view Self-Supervised Contrastive Learning for Multivariate Time Series

[Experimental study on TCRbeta idiotypic antigenic determinants DNA vaccine to induce anti-lymphoma antibodies].

TimeDiT: General-purpose Diffusion Transformers for Time Series Foundation Model

Self-Supervised Time Series Representation Learning via Cross Reconstruction Transformer

DRFormer: Multi-Scale Transformer Utilizing Diverse Receptive Fields for Long Time-Series Forecasting

A Deep Multi-Task Representation Learning Method for Time Series Classification and Retrieval.

Time Series Representation Models

Self-Supervised Learning of Disentangled Representations for Multivariate Time-Series

Physically-guided Temporal Diffusion Transformer for Long-Term Time Series Forecasting

TimeMAE: Self-Supervised Representations of Time Series with Decoupled Masked Autoencoders

Self-Distilled Representation Learning for Time Series

A Time Series is Worth 64 Words: Long-term Forecasting with Transformers

Non-autoregressive Conditional Diffusion Models for Time Series Prediction

Latent Diffusion Transformer for Probabilistic Time Series Forecasting

DART: Denoising Autoregressive Transformer for Scalable Text-to-Image Generation

TimeLDM: Latent Diffusion Model for Unconditional Time Series Generation

Rethinking self-supervised learning for time series forecasting: A temporal perspective

DifFormer: Multi-Resolutional Differencing Transformer With Dynamic Ranging for Time Series Analysis