Abstract:Transformer-based models have traditionally been the primary focus of research for addressing time series forecasting challenges. However, the emergence of recently introduced high-performance linear models has cast doubt upon the effectiveness of transformer architecture in time series forecasting tasks. Throughout, most Transformer variants have represented time series using time point-wise tokenization, which does not provide sufficient semantic information for the attention mechanism. PatchTST expands the receptive field through patch-wise tokenization, mitigating the problem of inadequate information. However, when confronted with multivariate time series forecasting tasks, it does not consider the potential impact of delays and correlation between variates on prediction performance. The recently proposed iTransformer addresses the issue of misalignment between variates by employing series-wise tokenization, yet its embedding method is limited to shallow temporal feature representation. In this work, we propose the Temporal Feature Enhanced Transformer (TFEformer), which deeply integrates patch-wise and series-wise tokenization to enhance the temporal representation of multivariate tokens. Furthermore, we introduce a multi-scale patch fusion mechanism capable of capturing and adaptively integrating temporal features across multiple resolutions. We also enhanced the FFN module to serve as a temporal feature extractor and introduced variate-wise attention to capture the correlations between variables. Extensive experiments on eight real-world datasets have demonstrated that TFEformer outperforms all existing models, achieving state-of-the-art performance. Through experiments, we have also shown that TFEformer improves transformer-based models with superior generalization ability, better utilization of extended lookback windows, and effective suppression of distribution shifts.

TFWT: Tabular Feature Weighting with Transformer

TFEformer: Temporal Feature Enhanced Transformer for Multivariate Time Series Forecasting

Tokenize features, enhancing tables: the FT-TABPFN model for tabular classification

A Dynamic Weighted Tabular Method for Convolutional Neural Networks

DIWIFT: Discovering Instance-wise Influential Features for Tabular Data

T2G-Former: Organizing Tabular Features into Relation Graphs Promotes Heterogeneous Feature Interaction

Leveraging 2D Information for Long-term Time Series Forecasting with Vanilla Transformers

Table Transformers for Imputing Textual Attributes

Deep Feature Embedding for Tabular Data

Enhancing Tabular Data Optimization with a Flexible Graph-based Reinforced Exploration Strategy

Transformer-Dw: A Transformer Network With Dynamic And Weighted Head

Beyond Deep Learning: An Evolutionary Feature Engineering Approach to Tabular Data Classification

Cross-Table Pretraining towards a Universal Function Space for Heterogeneous Tabular Data

A Joint Time-Frequency Domain Transformer for multivariate time series forecasting

TransFeat-TPP: An Interpretable Deep Covariate Temporal Point Processes

DenseFormer: Enhancing Information Flow in Transformers via Depth Weighted Averaging

FeatNavigator: Automatic Feature Augmentation on Tabular Data

Unlocking the Transferability of Tokens in Deep Models for Tabular Data

XTab: Cross-table Pretraining for Tabular Transformers

Itransformer: Inverted Transformers Are Effective for Time Series Forecasting

TabDPT: Scaling Tabular Foundation Models