Abstract:Recurrent Neural Networks (RNNs), particularly when equipped with output windows—a standard practice in contemporary time series forecasting—have shown proficiency in handling short-term dependencies. Nonetheless, RNNs can encounter challenges in maintaining hidden states over extended forecasting periods, particularly in longer-term predictions where increased hidden state sizes and extended look-back windows can lead to gradient instability. In contrast, Transformer-based models, with their distinctive architecture designed to encode complex contextual relationships and enable computations to be done in parallel, are emerging as a popular alternative in this field. However, current research has mainly focused on modifying attention mechanisms, overlooking opportunities to improve the feedforward layer, which could lead to efficiency limitations. Moreover, prevailing methods often assume absolute independence between channels, disregarding distinct features among variables and failing to fully leverage channel-specific information. To address these gaps, we propose an efficient transformer design for multivariate time series prediction. Our approach integrates two key components: (i) a gated residual attention unit that enhances predictive accuracy and computational efficiency, and (ii) a channel embedding technique that differentiates between series and boosts performance. Theoretically, we prove that our model has recurrent dynamics introduced by the RNN layer. Through extensive experiments on real-world data, we demonstrate that our proposed method achieves competitive predictive accuracy compared to prior approaches while exhibiting accelerated processing relative to state-of-the-art transformers. Our code, data, and trained models are available at https://github.com/MythosAd/GRAformer.

CGN: A Simple Yet Effective Multi-Channel Gated Network for Long-Term Time Series Forecasting.

AGCNT: Adaptive Graph Convolutional Network for Transformer-based Long Sequence Time-Series Forecasting

Hidformer: Hierarchical Dual-Tower Transformer Using Multi-Scale Mergence for Long-Term Time Series Forecasting

GRAformer: A Gated Residual Attention Transformer for Multivariate Time Series Forecasting

Generalizable Memory-driven Transformer for Multivariate Long Sequence Time-series Forecasting

MP3Net:Multi-scale Patch Parallel Prediction Networks for Multivariate Time Series Forecasting

GCformer: an Efficient Solution for Accurate and Scalable Long-Term Multivariate Time Series Forecasting

GCformer: An Efficient Framework for Accurate and Scalable Long-Term Multivariate Time Series Forecasting

Temporal Chain Network with Intuitive Attention Mechanism for Long-Term Series Forecasting.

Gated Transformer Networks for Multivariate Time Series Classification

CGTS: A Transformer framework for time series prediction based on feature extraction

STGHTN: Spatial-temporal gated hybrid transformer network for traffic flow forecasting

TCLN: A Transformer-based Conv-LSTM Network for Multivariate Time Series Forecasting

Multi-resolution Time-Series Transformer for Long-term Forecasting

GLFNet: Global and Local Frequency-domain Network for Long-term Time Series Forecasting

GTformer: Graph-Based Temporal-Order-Aware Transformer for Long-Term Series Forecasting

TCCT: Tightly-coupled convolutional transformer on time series forecasting

Towards Long-Term Time-Series Forecasting: Feature, Pattern, and Distribution

sTransformer: A Modular Approach for Extracting Inter-Sequential and Temporal Information for Time-Series Forecasting

CTFNet: Long-Sequence Time-Series Forecasting Based on Convolution and Time–Frequency Analysis

Non-stationary Transformers: Exploring the Stationarity in Time Series Forecasting