UniTST: Effectively Modeling Inter-Series and Intra-Series Dependencies for Multivariate Time Series Forecasting

Juncheng Liu,Chenghao Liu,Gerald Woo,Yiwei Wang,Bryan Hooi,Caiming Xiong,Doyen Sahoo

2024-06-07

Abstract:Transformer-based models have emerged as powerful tools for multivariate time series forecasting (MTSF). However, existing Transformer models often fall short of capturing both intricate dependencies across variate and temporal dimensions in MTS data. Some recent models are proposed to separately capture variate and temporal dependencies through either two sequential or parallel attention mechanisms. However, these methods cannot directly and explicitly learn the intricate inter-series and intra-series dependencies. In this work, we first demonstrate that these dependencies are very important as they usually exist in real-world data. To directly model these dependencies, we propose a transformer-based model UniTST containing a unified attention mechanism on the flattened patch tokens. Additionally, we add a dispatcher module which reduces the complexity and makes the model feasible for a potentially large number of variates. Although our proposed model employs a simple architecture, it offers compelling performance as shown in our extensive experiments on several datasets for time series forecasting.

Machine Learning,Artificial Intelligence

What problem does this paper attempt to address?

The paper introduces a new model called UniTST (Unified Time Series Transformer) aimed at addressing the challenges in multivariate time series forecasting (MTSF). Existing Transformer models struggle to capture both intricate inter-series (between different variate time series) and intra-series (within the same variate time series) dependencies effectively. Some recent models attempt to address these dependencies through separate mechanisms, but they cannot directly and explicitly model these complex relationships.

### Problem Statement

The paper identifies the following key problems:

1. **Inability to Capture Complex Dependencies**: Existing Transformer models lack the ability to directly and explicitly model the intricate inter-series and intra-series dependencies that are crucial for accurate forecasting.

2. **Sequential or Parallel Mechanisms**: Some models use sequential or parallel attention mechanisms to handle variate and temporal dependencies, but these approaches do not model cross-time and cross-variate dependencies simultaneously.

### Proposed Solution: UniTST

UniTST is designed to address these limitations by proposing a unified attention mechanism that can directly model inter-series and intra-series dependencies simultaneously. Here are the main components and innovations of UniTST:

- **Unified Attention Mechanism**: UniTST flattens all patches from different variates into a unified sequence, allowing the model to capture both inter-series and intra-