Abstract:In the burgeoning ecosystem of Internet of Things, multivariate time series (MTS) data has become ubiquitous, highlighting the fundamental role of time series forecasting across numerous applications. The crucial challenge of long-term MTS forecasting requires adept models capable of capturing both intra- and inter-series dependencies. Recent advancements in deep learning, notably Transformers, have shown promise. However, many prevailing methods either marginalize inter-series dependencies or overlook them entirely. To bridge this gap, this paper introduces a novel series-aware framework, explicitly designed to emphasize the significance of such dependencies. At the heart of this framework lies our specific implementation: the SageFormer. As a Series-aware Graph-enhanced Transformer model, SageFormer proficiently discerns and models the intricate relationships between series using graph structures. Beyond capturing diverse temporal patterns, it also curtails redundant information across series. Notably, the series-aware framework seamlessly integrates with existing Transformer-based models, enriching their ability to comprehend inter-series relationships. Extensive experiments on real-world and synthetic datasets validate the superior performance of SageFormer against contemporary state-of-the-art approaches.

What problem does this paper attempt to address?

The paper primarily addresses the long-term forecasting problem of Multivariate Time Series (MTS) data by proposing a new framework—the Series-aware Framework, and a specific model—SageFormer. ### Research Background and Motivation With the development of the Internet of Things (IoT), MTS data has become increasingly prevalent and plays a core role in numerous applications. The main challenge of long-term MTS forecasting is how to effectively capture intra-series and inter-series dependencies. Although Transformer-based methods have shown excellent performance in handling such problems in recent years, many existing methods either ignore inter-series dependencies or fail to fully utilize this information. ### Solution To address the above issues, the paper presents the following contributions: 1. **Series-aware Framework**: This is a general framework designed to emphasize the importance of inter-series dependencies. The framework addresses the modeling of inter-series dependencies by introducing a specific implementation—SageFormer. SageFormer is a Transformer model enhanced with Graph Neural Networks (GNN), capable of efficiently identifying and modeling complex relationships between series. 2. **SageFormer Model**: As an instance of the Series-aware Framework, SageFormer utilizes a graph structure to enhance the model's understanding of inter-series dependencies while using Transformer to capture intra-series dependencies. This ensures that the model can comprehensively understand the intrinsic dynamics in a series-aware setting. 3. **Experimental Validation**: Extensive experiments on multiple real-world and synthetic datasets demonstrate that SageFormer outperforms current state-of-the-art methods. ### Key Innovations - **Series-aware Global Token**: A key innovation is the integration of a global token to enhance series awareness. This method captures the global information of each series and facilitates information exchange between series. - **Graph Structure Learning**: The adjacency matrix of the graph is learned in an end-to-end manner, allowing the model to capture relationships between series without prior knowledge. This approach improves the model's applicability and flexibility. - **Iterative Message Passing Process**: Tokens in SageFormer are processed through a series of encoding layers, which include an iterative process of temporal and graph aggregation. This helps propagate information within and between series, enhancing the model's expressiveness. ### Experimental Results The paper conducts experiments on six mainstream real-world datasets, including Traffic, Electricity, Weather, etc., as well as two synthetic datasets. The experimental results show that SageFormer outperforms various existing baseline models on most datasets, especially when dealing with datasets with a large number of series (such as Traffic and Electricity). Additionally, extra experiments were conducted to verify the model's stability and effectiveness under different numbers of series.

SageFormer: Series-Aware Framework for Long-term Multivariate Time Series Forecasting

Foreformer: an Enhanced Transformer-Based Framework for Multivariate Time Series Forecasting

Hidformer: Hierarchical Dual-Tower Transformer Using Multi-Scale Mergence for Long-Term Time Series Forecasting

Generalizable Memory-driven Transformer for Multivariate Long Sequence Time-series Forecasting

GTformer: Graph-Based Temporal-Order-Aware Transformer for Long-Term Series Forecasting

Adapt to Small-Scale and Long-Term Time Series Forecasting with Enhanced Multidimensional Correlation

A hybrid framework for multivariate long-sequence time series forecasting

Integrating Mamba and Transformer for Long-Short Range Time Series Forecasting

Multi-scale convolution enhanced transformer for multivariate long-term time series forecasting

Enhancing Time Series Forecasting: A Hierarchical Transformer with Probabilistic Decomposition Representation

Integration of Mamba and Transformer -- MAT for Long-Short Range Time Series Forecasting with Application to Weather Dynamics

Sparse transformer with local and seasonal adaptation for multivariate time series forecasting

Discovering Predictable Latent Factors for Time Series Forecasting

RSMformer: an efficient multiscale transformer-based framework for long sequence time-series forecasting

TCLN: A Transformer-based Conv-LSTM Network for Multivariate Time Series Forecasting

DRFormer: Multi-Scale Transformer Utilizing Diverse Receptive Fields for Long Time-Series Forecasting

Enhancing Transformer-based models for Long Sequence Time Series Forecasting via Structured Matrix

Scalable Transformer for High Dimensional Multivariate Time Series Forecasting

Bi-Mamba4TS: Bidirectional Mamba for Time Series Forecasting

SMARTformer: Semi-Autoregressive Transformer with Efficient Integrated Window Attention for Long Time Series Forecasting.