Abstract:Transformer-based models have traditionally been the primary focus of research for addressing time series forecasting challenges. However, the emergence of recently introduced high-performance linear models has cast doubt upon the effectiveness of transformer architecture in time series forecasting tasks. Throughout, most Transformer variants have represented time series using time point-wise tokenization, which does not provide sufficient semantic information for the attention mechanism. PatchTST expands the receptive field through patch-wise tokenization, mitigating the problem of inadequate information. However, when confronted with multivariate time series forecasting tasks, it does not consider the potential impact of delays and correlation between variates on prediction performance. The recently proposed iTransformer addresses the issue of misalignment between variates by employing series-wise tokenization, yet its embedding method is limited to shallow temporal feature representation. In this work, we propose the Temporal Feature Enhanced Transformer (TFEformer), which deeply integrates patch-wise and series-wise tokenization to enhance the temporal representation of multivariate tokens. Furthermore, we introduce a multi-scale patch fusion mechanism capable of capturing and adaptively integrating temporal features across multiple resolutions. We also enhanced the FFN module to serve as a temporal feature extractor and introduced variate-wise attention to capture the correlations between variables. Extensive experiments on eight real-world datasets have demonstrated that TFEformer outperforms all existing models, achieving state-of-the-art performance. Through experiments, we have also shown that TFEformer improves transformer-based models with superior generalization ability, better utilization of extended lookback windows, and effective suppression of distribution shifts.

STC-ViT: Spatio Temporal Continuous Vision Transformer for Weather Forecasting

TFEformer: Temporal Feature Enhanced Transformer for Multivariate Time Series Forecasting

Hidformer: Hierarchical Dual-Tower Transformer Using Multi-Scale Mergence for Long-Term Time Series Forecasting

Spatial-Temporal Convolutional Transformer Network for Multivariate Time Series Forecasting

Spatio-Temporal Transformer Network for Weather Forecasting

Multi-Task Spatial-Temporal Transformer for Multi-Variable Meteorological Forecasting

Stecformer: Spatio-temporal Encoding Cascaded Transformer for Multivariate Long-term Time Series Forecasting

STVformer: A Spatial-Temporal-variable Transformer with Auxiliary Knowledge for Sea Surface Temperature Prediction

TENT: Tensorized Encoder Transformer for Temperature Forecasting

CViT: Continuous Vision Transformer for Operator Learning

Spatio-Temporal Transformer Network with Physical Knowledge Distillation for Weather Forecasting

HEAL-ViT: Vision Transformers on a spherical mesh for medium-range weather forecasting

WeatherFormer: Empowering Global Numerical Weather Forecasting with Space-Time Transformer

Multi-resolution Time-Series Transformer for Long-term Forecasting

Continuous Spatiotemporal Transformers

sTransformer: A Modular Approach for Extracting Inter-Sequential and Temporal Information for Time-Series Forecasting

NAST: Non-Autoregressive Spatial-Temporal Transformer for Time Series Forecasting

ST2T: A Spatio-Temporal Transformer for Cellular Traffic Prediction in Digital Twin Systems

Leveraging 2D Information for Long-term Time Series Forecasting with Vanilla Transformers

Ister: Inverted Seasonal-Trend Decomposition Transformer for Explainable Multivariate Time Series Forecasting

TSformer: A Non-autoregressive Spatial-temporal Transformers for 30-day Ocean Eddy-Resolving Forecasting