Abstract:Time series forecasting has played a significant role in many practical fields. But time series data generated from real-world applications always exhibits high variance and lots of noise, which makes it difficult to capture the inherent periodic patterns of the data, hurting the prediction accuracy significantly. To address this issue, we propose the Esiformer, which apply interpolation on the original data, decreasing the overall variance of the data and alleviating the influence of noise. What's more, we enhanced the vanilla transformer with a robust Sparse FFN. It can enhance the representation ability of the model effectively, and maintain the excellent robustness, avoiding the risk of overfitting compared with the vanilla implementation. Through evaluations on challenging real-world datasets, our method outperforms leading model PatchTST, reducing MSE by 6.5% and MAE by 5.8% in multivariate time series forecasting. Code is available at: <a class="link-external link-https" href="https://github.com/yyg1282142265/Esiformer/tree/main" rel="external noopener nofollow">this https URL</a>.

What problem does this paper attempt to address?

The problems that this paper attempts to solve are as follows: In practical applications, time - series data usually has high variance and a large amount of noise, which makes it difficult to capture the inherent periodic patterns of the data and significantly reduces the accuracy of prediction. Specifically, the author points out: 1. **Problems of high variance and noise**: - The high variance and noise in time - series data can mask the potential trends and periodic patterns of the data, increasing the difficulty for prediction models to capture these patterns. - Such high volatility increases the degree of dispersion between data points, leading to greater uncertainty in prediction results. 2. **Limitations of existing methods**: - Although increasing the hidden - layer dimension of the Transformer model can improve the representational ability of the model, it also increases the risk of over - fitting, and thus cannot effectively improve the prediction performance. To solve these problems, the author proposes the Esiformer model, which mainly improves time - series prediction through the following two methods: 1. **Interpolation techniques**: - By performing interpolation processing on the original time - series data, the overall variance of the data is reduced, the influence of noise is alleviated, and thus the accuracy of prediction is improved. - The new data points generated by interpolation are generated from the original data points, retaining the information content but reducing the overall variance. 2. **Sparse FFN (Feed - Forward Neural Network)**: - A sparse FFN is introduced to enhance the standard Transformer model. Larger hidden - layer dimensions are used to improve the representational ability of the model, and at the same time, over - fitting is prevented through a sparsification strategy. - The sparse FFN not only reduces the number of parameters and the complexity of the model, but also enhances the locality of features through a sparse connection pattern, helping the model to better learn important features and reduce noise interference. Through these improvements, Esiformer performs excellently on multiple real - world datasets. Compared with the existing PatchTST model, Esiformer reduces the MSE by 6.5% and the MAE by 5.8% in multivariate time - series prediction.

Less is more: Embracing sparsity and interpolation with Esiformer for time series forecasting

Foreformer: an Enhanced Transformer-Based Framework for Multivariate Time Series Forecasting

TFEformer: Temporal Feature Enhanced Transformer for Multivariate Time Series Forecasting

A Fusion Transformer for Multivariable Time Series Forecasting: the Mooney Viscosity Prediction Case

Sparse-VQ Transformer: An FFN-Free Framework with Vector Quantization for Enhanced Time Series Forecasting

SparseTSF: Modeling Long-term Time Series Forecasting with 1k Parameters

Sparse transformer with local and seasonal adaptation for multivariate time series forecasting

Minusformer: Improving Time Series Forecasting by Progressively Learning Residuals

TimeXer: Empowering Transformers for Time Series Forecasting with Exogenous Variables

Discovering Predictable Latent Factors for Time Series Forecasting

Frequency-Enhanced Transformer with Symmetry-Based Lightweight Multi-Representation for Multivariate Time Series Forecasting

DRFormer: Multi-Scale Transformer Utilizing Diverse Receptive Fields for Long Time-Series Forecasting

Adaptive Sparsity Level during Training for Efficient Time Series Forecasting with Transformers

FDF: Flexible Decoupled Framework for Time Series Forecasting with Conditional Denoising and Polynomial Modeling

A Time Series is Worth 64 Words: Long-term Forecasting with Transformers

Take an Irregular Route: Enhance the Decoder of Time-Series Forecasting Transformer

The Bigger the Better? Rethinking the Effective Model Scale in Long-term Time Series Forecasting

Less Is More: Fast Multivariate Time Series Forecasting with Light Sampling-oriented MLP Structures