Transformer Multivariate Forecasting: Less is More?

Jingjing Xu,Caesar Wu,Yuan-Fang Li,Pascal Bouvry

2024-03-07

Abstract:In the domain of multivariate forecasting, transformer models stand out as powerful apparatus, displaying exceptional capabilities in handling messy datasets from real-world contexts. However, the inherent complexity of these datasets, characterized by numerous variables and lengthy temporal sequences, poses challenges, including increased noise and extended model runtime. This paper focuses on reducing redundant information to elevate forecasting accuracy while optimizing runtime efficiency. We propose a novel transformer forecasting framework enhanced by Principal Component Analysis (PCA) to tackle this challenge. The framework is evaluated by five state-of-the-art (SOTA) models and four diverse real-world datasets. Our experimental results demonstrate the framework's ability to minimize prediction errors across all models and datasets while significantly reducing runtime. From the model perspective, one of the PCA-enhanced models: PCA+Crossformer, reduces mean square errors (MSE) by 33.3% and decreases runtime by 49.2% on average. From the dataset perspective, the framework delivers 14.3% MSE and 76.6% runtime reduction on Electricity datasets, as well as 4.8% MSE and 86.9% runtime reduction on Traffic datasets. This study aims to advance various SOTA models and enhance transformer-based time series forecasting for intricate data. Code is available at:

Machine Learning,Artificial Intelligence

What problem does this paper attempt to address?

The problem that this paper attempts to solve is how to improve prediction accuracy and optimize operational efficiency by reducing redundant information in multivariate time - series prediction. Specifically, the paper proposes a Transformer prediction framework enhanced by Principal Component Analysis (PCA) to address the challenges brought by high - complexity datasets, numerous variables, and long time - series in practical applications, such as increased noise and extended model running time. Through evaluation on five state - of - the - art (SOTA) models and four different real - world datasets, the experimental results show that this framework can significantly reduce prediction errors and greatly reduce running time. For example, the PCA + Crossformer model reduces the mean - squared error (MSE) by an average of 33.3% and the running time by 49.2%. From the perspective of datasets, this framework achieves a 14.3% MSE reduction and a 76.6% running - time reduction on the power dataset, and on the traffic dataset, it reduces the MSE by 4.8% and the running time by 86.9% respectively. This research aims to promote various SOTA models and enhance the time - series prediction ability based on Transformer, especially in dealing with complex data.