Abstract:Tabular data, widely used in industries like healthcare, finance, and transportation, presents unique challenges for deep learning due to its heterogeneous nature and lack of spatial structure. This survey reviews the evolution of deep learning models for tabular data, from early fully connected networks (FCNs) to advanced architectures like TabNet, SAINT, TabTranSELU, and MambaNet. These models incorporate attention mechanisms, feature embeddings, and hybrid architectures to address tabular data complexities. TabNet uses sequential attention for instance-wise feature selection, improving interpretability, while SAINT combines self-attention and intersample attention to capture complex interactions across features and data points, both advancing scalability and reducing computational overhead. Hybrid architectures such as TabTransformer and FT-Transformer integrate attention mechanisms with multi-layer perceptrons (MLPs) to handle categorical and numerical data, with FT-Transformer adapting transformers for tabular datasets. Research continues to balance performance and efficiency for large datasets. Graph-based models like GNN4TDL and GANDALF combine neural networks with decision trees or graph structures, enhancing feature representation and mitigating overfitting in small datasets through advanced regularization techniques. Diffusion-based models like the Tabular Denoising Diffusion Probabilistic Model (TabDDPM) generate synthetic data to address data scarcity, improving model robustness. Similarly, models like TabPFN and Ptab leverage pre-trained language models, incorporating transfer learning and self-supervised techniques into tabular tasks. This survey highlights key advancements and outlines future research directions on scalability, generalization, and interpretability in diverse tabular data applications.

Categorical Embeddings for Tabular Data using PyTorch

Deep Feature Embedding for Tabular Data

Enriching Tabular Data with Contextual LLM Embeddings: A Comprehensive Ablation Study for Ensemble Classifiers

Embeddings for Tabular Data: A Survey

Multi-view Heterogeneous Fusion and Embedding for Categorical Attributes on Mixed Data.

PyTorch Tabular: A Framework for Deep Learning with Tabular Data

A Survey on Deep Tabular Learning

Tabular data generation with tensor contraction layers and transformers

Tabular Transformers for Modeling Multivariate Time Series

Tabular Learning: Encoding for Entity and Context Embeddings

Employing Two-Dimensional Word Embedding for Difficult Tabular Data Stream Classification

Transformers with Stochastic Competition for Tabular Data Modelling

TabMDA: Tabular Manifold Data Augmentation for Any Classifier using Transformers with In-context Subsetting

Deep Learning with Tabular Data: A Self-supervised Approach

SwitchTab: Switched Autoencoders Are Effective Tabular Learners

MambaTab: A Plug-and-Play Model for Learning Tabular Data

Using Person Embedding to Enrich Features and Data Augmentation for Classification

Cat2Vec: Learning Distributed Representation of Multi-field Categorical Data

From Categorical to Numerical: Multiple Transitive Distance Learning and Embedding.

Dense Representation Learning and Retrieval for Tabular Data Prediction

Table2Vec: Neural Word and Entity Embeddings for Table Population and Retrieval