Abstract:Long-term forecasting is of paramount importance in numerous scenarios, including predicting future energy, water, and food consumption. For instance, extreme weather events and natural disasters can profoundly impact infrastructure operations and pose severe safety concerns. Traditional CNN-based models often struggle to capture long-distance dependencies effectively. In contrast, Transformers-based models have shown significant promise in long-term forecasting. This paper investigates the long-term forecasting problem and identifies a common limitation in existing Transformer-based models: they tend to reduce computational complexity at the expense of time information aggregation capability. Moreover, the order of time series plays a crucial role in accurate predictions, but current Transformer-based models lack sensitivity to time series order, rendering them unreasonable. To address these issues, we propose a novel Deformable-Local (DL) aggregation mechanism. This mechanism enhances the model's ability to aggregate time information and allows the model to adaptively adjust the size of the time aggregation window. Consequently, the model can discern more complex time patterns, leading to more accurate predictions. Additionally, our model incorporates a Frequency Selection module to reinforce effective features and reduce noise. Furthermore, we introduce Position Weights to mitigate the order-insensitivity problem present in existing methods. In extensive evaluations of long-term forecasting tasks, we conducted benchmark tests on six datasets covering various practical applications, including energy, traffic, economics, weather, and disease. Our method achieved state-of-the-art (SOTA) results, demonstrating significant improvements. For instance, on the ETT dataset, our model achieved an average MSE improvement of approximately 19% and an average MAE improvement of around 27%. Remarkably, for predicted lengths of 96 and 192, we achieved outstanding MSE and MAE improvements of 32.1% and 30.9%, respectively.

WeatherFormer: A Pretrained Encoder Model for Learning Robust Weather Representations from Small Datasets

TFEformer: Temporal Feature Enhanced Transformer for Multivariate Time Series Forecasting

WeatherFormer: Empowering Global Numerical Weather Forecasting with Space-Time Transformer

TENT: Tensorized Encoder Transformer for Temperature Forecasting

Analyzing and Exploring Training Recipes for Large-Scale Transformer-Based Weather Prediction

LightWeather: Harnessing Absolute Positional Encoding to Efficient and Scalable Global Weather Forecasting

Spatio-Temporal Transformer Network for Weather Forecasting

Multi-Task Spatial-Temporal Transformer for Multi-Variable Meteorological Forecasting

Scaling transformer neural networks for skillful and reliable medium-range weather forecasting

W-MAE: Pre-trained weather model with masked autoencoder for multi-variable weather forecasting

Efficient Subseasonal Weather Forecast using Teleconnection-informed Transformers

Fusion of Satellite Images and Weather Data With Transformer Networks for Downy Mildew Disease Detection

Numerical Weather Forecasting using Convolutional-LSTM with Attention and Context Matcher Mechanisms

Robust Weather Recognition in Images via Interactive CNN-Transformer

MTTF: a Multimodal Transformer for Temperature Forecasting

Learning Robust Precipitation Forecaster by Temporal Frame Interpolation

Enhancing Time Series Forecasting: A Hierarchical Transformer with Probabilistic Decomposition Representation

PWDformer: Deformable Transformer for Long-Term Series Forecasting

STC-ViT: Spatio Temporal Continuous Vision Transformer for Weather Forecasting

Advancing Meteorological Forecasting: AI-based Approach to Synoptic Weather Map Analysis

CRA5: Extreme Compression of ERA5 for Portable Global Climate and Weather Research via an Efficient Variational Transformer