Abstract:Transformer-based models have significantly improved performance in Long-term Time Series Forecasting (LTSF). These models employ various self-attention mechanisms to discover long-term dependencies. However, the computational efficiency is hampered by the inherent permutation invariance of self-attention, and they primarily focus on relationships within the sequence while neglecting potential relationships between different sample sequences. This limits the ability and flexibility of self-attention in LTSF. In addition, the Transformer's decoder outputs sequences in an autoregressive manner, leading to slow inference speed and error accumulation effects, especially for LTSF. Regarding the issues with Transformer-based models for LTSF, we propose a model better suited for LTSF, named MEAformer. MEAformer adopts a fully connected Multi-Layer Perceptron (MLP) architecture consisting of two types of layers: encoder layers and MLP layers. Unlike most encoder layers in Transformer-based models, the MEAformer replaces self-attention with temporal external attention. Temporal external attention explores potential relationships between different sample sequences in the training dataset. Compared to the quadratic complexity of self-attention mechanisms, temporal external attention has efficient linear complexity. Encoder layers can be stacked multiple times to capture time-dependent relationships at different scales. Furthermore, the MEAformer replaces the intricate decoder layers of the original model with more straightforward MLP layers. This modification aims to enhance inference speed and facilitate single-pass sequence generation, effectively mitigating the problem of error accumulation effects. Regarding long-term forecasting, MEAformer achieves state-of-the-art performance on six benchmark datasets, covering five real-world domains: energy, transportation, economy, weather, and disease. Code is available at: https://github.com/huangsiyuan924/MEAformer.

SALSTM: Segmented Self-Attention Long Short-Term Memory for Long-Term Forecasting

xLSTMTime : Long-term Time Series Forecasting With xLSTM

Effective LSTMs with Seasonal-Trend Decomposition and Adaptive Learning and Niching-Based Backtracking Search Algorithm for Time Series Forecasting

Generalizable Memory-driven Transformer for Multivariate Long Sequence Time-series Forecasting

Bidformer: A Transformer-Based Model Via Bidirectional Sparse Self-Attention Mechanism for Long Sequence Time-Series Forecasting

A deep LSTM‐CNN based on self‐attention mechanism with input data reduction for short‐term load forecasting

LSTM with Spatiotemporal Attention for IoT-based Wireless Sensor Collected Hydrological Time-Series Forecasting

Long Time Series Deep Forecasting with Multiscale Feature Extraction and Seq2seq Attention Mechanism

RSMformer: an efficient multiscale transformer-based framework for long sequence time-series forecasting

Temporal Self-Attention-based Conv-LSTM Network for Multivariate Time Series Prediction

Unlocking the Power of LSTM for Long Term Time Series Forecasting

Temporal Chain Network with Intuitive Attention Mechanism for Long-Term Series Forecasting.

Time Series Prediction Based on LSTM-Attention-LSTM Model

SegRNN: Segment Recurrent Neural Network for Long-Term Time Series Forecasting

Fuzzy inference-based LSTM for long-term time series prediction

Expanding the Prediction Capacity in Long Sequence Time-Series Forecasting

Are Transformers Effective for Time Series Forecasting?

MEAformer: an All-Mlp Transformer with Temporal External Attention for Long-Term Time Series Forecasting

FL-Net: A multi-scale cross-decomposition network with frequency external attention for long-term time series forecasting

Does Long-Term Series Forecasting Need Complex Attention and Extra Long Inputs?

Time Series Prediction Based on LSTM and High-Order Fuzzy Cognitive Map with Attention Mechanism