Abstract:Time series data are prevalent in the real world, particularly playing a crucial role in key domains such as meteorology, electricity, and finance. Comprising observations at historical time points, these data, when subjected to in-depth analysis and modeling, enable researchers to predict future trends and patterns, providing support for decision making. In current research, especially in the analysis of long time series, effectively extracting and integrating long-term dependencies with short-term features remains a significant challenge. Long-term dependencies refer to the correlation between data points spaced far apart in a time series, while short-term features focus on more recent changes. Understanding and combining these two features correctly are crucial for constructing accurate and reliable predictive models. To efficiently extract and integrate long-term dependencies and short-term features in long time series, this paper proposes a pyramid attention structure model based on multi-scale feature extraction, referred to as the MSFformer model. Initially, a coarser-scale construction module is designed to obtain coarse-grained information. A pyramid data structure is constructed through feature convolution, with the bottom layer representing the original data and each subsequent layer containing feature information extracted across different time step lengths. As a result, nodes higher up in the pyramid integrate information from more time points, such as every Monday or the beginning of each month, while nodes lower down retain their individual information. Additionally, a Skip-PAM is introduced, where a node only calculates attention with its neighboring nodes, parent node, and child nodes, effectively reducing the model's time complexity to some extent. Notably, the child nodes refer to nodes selected from the next layer by skipping specific time steps. In this study, we not only propose an innovative time series prediction model but also validate the effectiveness of these methods through a series of comprehensive experiments. To comprehensively evaluate the performance of the designed model, we conducted comparative experiments with baseline models, ablation experiments, and hyperparameter studies. The experimental results demonstrate that the MSFformer model improves by 35.87% and 42.6% on the MAE and MSE indicators, respectively, compared to traditional Transformer models. These results highlight the outstanding performance of our proposed deep learning model in handling complex time series data, particularly in capturing long-term dependencies and integrating short-term features.

What problem does this paper attempt to address?

The paper aims to address the issues of long-term dependency and short-term feature extraction and integration in time series forecasting. Specifically, in current research, especially when analyzing long time series, effectively extracting and integrating long-term dependencies and short-term features remains a significant challenge. To tackle this problem, the paper proposes a time series forecasting model based on multi-scale feature extraction, called the MSFformer model. The main contributions of this model include: 1. **Introduction of Skip-PAM**: Aimed at enabling deep learning models to effectively capture long-term and short-term features in long time series data. In the encoder, the pyramid attention mechanism processes previous feature vectors to obtain long-term dependencies and local features. 2. **Improved CSCM**: Aimed at establishing a pyramid data structure suitable for Skip-PAM. After encoding the input data, coarse-grained feature information at different time steps is obtained through the proposed feature convolution. 3. **Experimental Validation**: Based on three time series datasets, compared with three baseline methods, demonstrating the good performance of this method. Through these improvements, the MSFformer model shows significant performance enhancement in handling complex time series data, with improvements of 35.87% and 42.6% in MAE and MSE metrics, respectively. This indicates that the model performs excellently in capturing long-term dependencies and integrating short-term features.

Time Series Prediction Based on Multi-Scale Feature Extraction

Foreformer: an Enhanced Transformer-Based Framework for Multivariate Time Series Forecasting

TFEformer: Temporal Feature Enhanced Transformer for Multivariate Time Series Forecasting

Hidformer: Hierarchical Dual-Tower Transformer Using Multi-Scale Mergence for Long-Term Time Series Forecasting

Multi-scale convolution enhanced transformer for multivariate long-term time series forecasting

Long Time Series Deep Forecasting with Multiscale Feature Extraction and Seq2seq Attention Mechanism

Adapt to Small-Scale and Long-Term Time Series Forecasting with Enhanced Multidimensional Correlation

Multi-scale Transformer Pyramid Networks for Multivariate Time Series Forecasting

FormerTime: Hierarchical Multi-Scale Representations for Multivariate Time Series Classification

RSMformer: an efficient multiscale transformer-based framework for long sequence time-series forecasting

Segmented Frequency‐Domain Correlation Prediction Model for Long‐Term Time Series Forecasting Using Transformer

Muformer: A Long Sequence Time-Series Forecasting Model Based on Modified Multi-Head Attention

Time Series Forecasting with Multi-scale Decomposition and Fourier Neural Operators

MEAformer: an All-Mlp Transformer with Temporal External Attention for Long-Term Time Series Forecasting

Multi-resolution Time-Series Transformer for Long-term Forecasting

DRFormer: Multi-Scale Transformer Utilizing Diverse Receptive Fields for Long Time-Series Forecasting

Preformer: Predictive Transformer with Multi-Scale Segment-wise Correlations for Long-Term Time Series Forecasting

A Hybrid Method with Adaptive Sub-Series Clustering and Attention-Based Stacked Residual LSTMs for Multivariate Time Series Forecasting

Adaptive Multi-Scale Decomposition Framework for Time Series Forecasting

Integrating Mamba and Transformer for Long-Short Range Time Series Forecasting

Stecformer: Spatio-temporal Encoding Cascaded Transformer for Multivariate Long-term Time Series Forecasting