Abstract:Multivariate Time Series (MTS) analysis is crucial to understanding and managing complex systems, such as traffic and energy systems, and a variety of approaches to MTS forecasting have been proposed recently. However, we often observe inconsistent or seemingly contradictory performance findings across different studies. This hinders our understanding of the merits of different approaches and slows down progress. We address the need for means of assessing MTS forecasting proposals reliably and fairly, in turn enabling better exploitation of MTS as seen in different applications. Specifically, we first propose BasicTS+, a benchmark designed to enable fair, comprehensive, and reproducible comparison of MTS forecasting solutions. BasicTS+ establishes a unified training pipeline and reasonable settings, enabling an unbiased evaluation. Second, we identify the heterogeneity across different MTS as an important consideration and enable classification of MTS based on their temporal and spatial characteristics. Disregarding this heterogeneity is a prime reason for difficulties in selecting the most promising technical directions. Third, we apply BasicTS+ along with rich datasets to assess the capabilities of more than 45 MTS forecasting solutions. This provides readers with an overall picture of the cutting-edge research on MTS forecasting. The code can be accessed at <a class="link-external link-https" href="https://github.com/GestaltCogTeam/BasicTS" rel="external noopener nofollow">this https URL</a>.

What problem does this paper attempt to address?

The main problems that this paper attempts to solve are a series of challenges in multivariate time series (MTS) prediction, especially the inconsistency and seemingly contradictory phenomena in the performance results among different studies. Specifically, the paper mainly focuses on the following aspects: 1. **Evaluating the reliability and fairness of MTS prediction methods**: - The paper points out that in different studies, the performance of the same prediction method under the same data set and experimental settings may vary significantly. This inconsistency hinders the understanding of the advantages and disadvantages of different methods and slows down the progress in the field. - To solve this problem, the paper proposes BasicTS +, which is a benchmark platform for MTS prediction solutions, aiming to provide a fair, comprehensive, and reproducible comparison. 2. **Identifying and classifying the heterogeneity of MTS data sets**: - Different MTS data sets may have completely different patterns. Ignoring these heterogeneities will lead to difficulties in choosing the most effective technical direction. - The paper classifies MTS data sets according to their temporal and spatial characteristics to better understand different types of MTS data and their impact on prediction performance. 3. **Providing detailed analysis and comparison**: - Using BasicTS + and a rich heterogeneous data set, the paper conducts a detailed analysis and comparison of more than 45 MTS prediction solutions, providing readers with an overview of the current cutting - edge research. - These analyses help researchers evaluate the performance of different models more accurately and provide valuable insights for future research. ### Main contributions - **Proposing the BasicTS + benchmark platform**: This is the first benchmark platform specifically designed for MTS prediction, covering long - term time series prediction (LTSF) and spatio - temporal prediction (STF), and supporting the evaluation of more than 45 popular models on 20 commonly used data sets. - **Identifying the heterogeneity of MTS data sets**: Classifying data sets by temporal and spatial characteristics reveals that ignoring heterogeneity is the reason for the difficulty in choosing technical directions. - **Detailed model analysis and comparison**: Using BasicTS + and heterogeneous data sets, a detailed performance and efficiency analysis of popular MTS prediction models is carried out, providing important insights into the progress of existing research. ### Formula examples In the paper, evaluation metrics such as MAE, RMSE, MAPE, and WAPE are used to measure prediction performance. The following are the definitions of these metrics: \[ \text{MAE}(y, \hat{y})=\frac{1}{|\Omega|} \sum_{i \in \Omega}\left|y_{i}-\hat{y}_{i}\right| \] \[ \text{RMSE}(y, \hat{y})=\sqrt{\frac{1}{|\Omega|} \sum_{i \in \Omega}\left(y_{i}-\hat{y}_{i}\right)^{2}} \] \[ \text{MAPE}(y, \hat{y})=\frac{1}{|\Omega|} \sum_{i \in \Omega}\left|\frac{y_{i}-\hat{y}_{i}}{y_{i}}\right| \] \[ \text{WAPE}(y, \hat{y})=\frac{\sum_{i \in \Omega}\left|y_{i}-\hat{y}_{i}\right|}{\sum_{i \in \Omega}\left|y_{i}\right|} \] where \(\Omega\) represents the index set of all observed samples, \(y_{i}\) represents the \(i\) - th actual sample, and \(\hat{y}_{i}\) represents the corresponding predicted value. Through these formulas, the paper ensures the accuracy and comparability of the evaluation results.

Exploring Progress in Multivariate Time Series Forecasting: Comprehensive Benchmarking and Heterogeneity Analysis

Foreformer: an Enhanced Transformer-Based Framework for Multivariate Time Series Forecasting

MvTS-library: an Open Library for Deep Multivariate Time Series Forecasting

TFB: Towards Comprehensive and Fair Benchmarking of Time Series Forecasting Methods

FoundTS: Comprehensive and Unified Benchmarking of Foundation Models for Time Series Forecasting

Effective LSTMs with Seasonal-Trend Decomposition and Adaptive Learning and Niching-Based Backtracking Search Algorithm for Time Series Forecasting

Parallel Extraction of Long-term Trends and Short-term Fluctuation Framework for Multivariate Time Series Forecasting

ProbTS: Benchmarking Point and Distributional Forecasting across Diverse Prediction Horizons

Adapt to Small-Scale and Long-Term Time Series Forecasting with Enhanced Multidimensional Correlation

A Multi-View Multi-Task Learning Framework for Multi-Variate Time Series Forecasting

Learning Structured Components: Towards Modular and Interpretable Multivariate Time Series Forecasting

Scalable Transformer for High Dimensional Multivariate Time Series Forecasting

A lightweight multi-layer perceptron for efficient multivariate time series forecasting

SST: Multi-Scale Hybrid Mamba-Transformer Experts for Long-Short Range Time Series Forecasting

Disentangling Structured Components: Towards Adaptive, Interpretable and Scalable Time Series Forecasting

Experimental Study of Multivariate Time Series Forecasting Models

BayesTSF: Measuring Uncertainty Estimation in Industrial Time Series Forecasting from a Bayesian Perspective

MTS-UNMixers: Multivariate Time Series Forecasting via Channel-Time Dual Unmixing

A Hybrid Method with Adaptive Sub-Series Clustering and Attention-Based Stacked Residual LSTMs for Multivariate Time Series Forecasting

SimpleTS: an Efficient and Universal Model Selection Framework for Time Series Forecasting