Abstract:Masked time series modeling has recently gained much attention as a self-supervised representation learning strategy for time series. Inspired by masked image modeling in computer vision, recent works first patchify and partially mask out time series, and then train Transformers to capture the dependencies between patches by predicting masked patches from unmasked patches. However, we argue that capturing such patch dependencies might not be an optimal strategy for time series representation learning; rather, learning to embed patches independently results in better time series representations. Specifically, we propose to use 1) the simple patch reconstruction task, which autoencode each patch without looking at other patches, and 2) the simple patch-wise MLP that embeds each patch independently. In addition, we introduce complementary contrastive learning to hierarchically capture adjacent time series information efficiently. Our proposed method improves time series forecasting and classification performance compared to state-of-the-art Transformer-based models, while it is more efficient in terms of the number of parameters and training/inference time. Code is available at this repository: https://github.com/seunghan96/pits.

What problem does this paper attempt to address?

This paper attempts to solve a key problem in time - series representation learning: that is, under the self - supervised learning framework, how to more effectively capture the feature representation of time - series data. Specifically, the author believes that existing methods for representation learning by capturing the dependencies between time - series patches may not be the optimal strategy. Instead, the author proposes a new method, namely **independently embedding time - series patches**, which can not only improve the performance of representation learning, but also significantly reduce the number of model parameters and training / inference time. ### Main Problems and Solutions 1. **Problems with Existing Methods** - Existing Masked Time Series Modeling (MTM) methods learn representations by partially masking time series and predicting the masked parts. These methods usually use architectures such as Transformer to capture the dependencies between patches. - The author believes that this learning of dependencies may not be the optimal strategy, especially in time - series representation learning. 2. **Proposed Solutions** - **Patch Independence for TimeSeries (PITS)**: The author proposes a new framework, PITS, whose core idea is to independently embed time - series patches rather than relying on the interactions between patches. - **Patch - Independent Task**: A simple patch reconstruction task is introduced, that is, each patch is reconstructed only based on its own information without referring to other patches. - **Patch - Independent Architecture**: A simple multi - layer perceptron (MLP) is used as an encoder to avoid complex attention mechanisms and thus improve efficiency. - **Complementary Contrastive Learning**: To further improve the effect of representation learning, the author introduces complementary contrastive learning, which generates two views through random masking to efficiently capture adjacent time - series information. ### Main Contributions of the Paper 1. **Advantages of Proposed Independent Patch Embedding**: The advantages of independent patch embedding in performance and efficiency are verified through experiments. 2. **Introduction of Supplementary Analytic Learning**: A new contrastive learning strategy is proposed, which can improve the effect of representation learning without increasing additional computational costs. 3. **Extensive Experimental Verification**: Experiments are carried out on multiple time - series prediction and classification tasks, proving the superiority of PITS in standard settings and transfer - learning scenarios. ### Summary The core problem of this paper is to explore how to more effectively perform time - series representation learning under the self - supervised learning framework. By introducing the idea of independent patch embedding, the author proposes a new method, PITS, and verifies its advantages in performance and efficiency through a large number of experiments. This research provides a new perspective and an effective tool for time - series analysis.

Learning to Embed Time Series Patches Independently

TFEformer: Temporal Feature Enhanced Transformer for Multivariate Time Series Forecasting

Masked Autoencoders for Point Cloud Self-supervised Learning.

A Time Series is Worth 64 Words: Long-term Forecasting with Transformers

TimeMAE: Self-Supervised Representations of Time Series with Decoupled Masked Autoencoders

Towards Unsupervised Time Series Representation Learning: A Decomposition Perspective

Masked Autoencoders As Spatiotemporal Learners

Ti-MAE: Self-Supervised Masked Time Series Autoencoders

Point-MPP: Point Cloud Self-Supervised Learning from Masked Position Prediction

Bootstrap Masked Visual Modeling via Hard Patches Mining

LaT-PFN: A Joint Embedding Predictive Architecture for In-context Time-series Forecasting

Enhancing Time Series Forecasting: A Hierarchical Transformer with Probabilistic Decomposition Representation

HDMixer: Hierarchical Dependency with Extendable Patch for Multivariate Time Series Forecasting

Partial Channel Dependence with Channel Masks for Time Series Foundation Models

TS-MAE: A masked autoencoder for time series representation learning

[Experimental study on TCRbeta idiotypic antigenic determinants DNA vaccine to induce anti-lymphoma antibodies].

Multi-Patch Prediction: Adapting LLMs for Time Series Representation Learning

Continuous-Time Linear Positional Embedding for Irregular Time Series Forecasting

TSLANet: Rethinking Transformers for Time Series Representation Learning

PatchMixer: A Patch-Mixing Architecture for Long-Term Time Series Forecasting

Time-Transformer: Integrating Local and Global Features for Better Time Series Generation