Abstract:Transformer-based models have traditionally been the primary focus of research for addressing time series forecasting challenges. However, the emergence of recently introduced high-performance linear models has cast doubt upon the effectiveness of transformer architecture in time series forecasting tasks. Throughout, most Transformer variants have represented time series using time point-wise tokenization, which does not provide sufficient semantic information for the attention mechanism. PatchTST expands the receptive field through patch-wise tokenization, mitigating the problem of inadequate information. However, when confronted with multivariate time series forecasting tasks, it does not consider the potential impact of delays and correlation between variates on prediction performance. The recently proposed iTransformer addresses the issue of misalignment between variates by employing series-wise tokenization, yet its embedding method is limited to shallow temporal feature representation. In this work, we propose the Temporal Feature Enhanced Transformer (TFEformer), which deeply integrates patch-wise and series-wise tokenization to enhance the temporal representation of multivariate tokens. Furthermore, we introduce a multi-scale patch fusion mechanism capable of capturing and adaptively integrating temporal features across multiple resolutions. We also enhanced the FFN module to serve as a temporal feature extractor and introduced variate-wise attention to capture the correlations between variables. Extensive experiments on eight real-world datasets have demonstrated that TFEformer outperforms all existing models, achieving state-of-the-art performance. Through experiments, we have also shown that TFEformer improves transformer-based models with superior generalization ability, better utilization of extended lookback windows, and effective suppression of distribution shifts.

Transformer Based on Deconstruction and Dot Product for Time Series Forecasting of Air Quality

TFEformer: Temporal Feature Enhanced Transformer for Multivariate Time Series Forecasting

Foreformer: an Enhanced Transformer-Based Framework for Multivariate Time Series Forecasting

Hidformer: Hierarchical Dual-Tower Transformer Using Multi-Scale Mergence for Long-Term Time Series Forecasting

Short-Term Speed Forecasting of Large-Scale Urban Road Network Based on Transformer

Enhancing Time Series Forecasting: A Hierarchical Transformer with Probabilistic Decomposition Representation

Itransformer: Inverted Transformers Are Effective for Time Series Forecasting

NTDformer: A Multi-Scale Forecasting Model for Non-Stationary Multilevel Time Series

AD-autoformer: decomposition transformers with attention distilling for long sequence time-series forecasting

DSformer: A Double Sampling Transformer for Multivariate Time Series Long-term Prediction

Transformer network with decoupled spatial–temporal embedding for traffic flow forecasting

Autoformer: Decomposition Transformers with Auto-Correlation for Long-Term Series Forecasting

A Differential Attention Fusion Model Based on Transformer for Time Series Forecasting

Dateformer: Time-modeling Transformer for Longer-term Series Forecasting

InParformer: Evolutionary Decomposition Transformers with Interactive Parallel Attention for Long-Term Time Series Forecasting

DBAFormer: A Double-Branch Attention Transformer for Long-Term Time Series Forecasting

Probabilistic Decomposition Transformer for Time Series Forecasting

Towards Long-Term Time-Series Forecasting: Feature, Pattern, and Distribution

Time Series Forecasting (TSF) Using Various Deep Learning Models

Take an Irregular Route: Enhance the Decoder of Time-Series Forecasting Transformer

ETSformer: Exponential Smoothing Transformers for Time-series Forecasting