Abstract:Tabular data is the most widely used data format in machine learning (ML). While tree-based methods outperform DL-based methods in supervised learning, recent literature reports that self-supervised learning with Transformer-based models outperforms tree-based methods. In the existing literature on self-supervised learning for tabular data, contrastive learning is the predominant method. In contrastive learning, data augmentation is important to generate different views. However, data augmentation for tabular data has been difficult due to the unique structure and high complexity of tabular data. In addition, three main components are proposed together in existing methods: model structure, self-supervised learning methods, and data augmentation. Therefore, previous works have compared the performance without comprehensively considering these components, and it is not clear how each component affects the actual performance. In this study, we focus on data augmentation to address these issues. We propose a novel data augmentation method, $\textbf{M}$ask $\textbf{T}$oken $\textbf{R}$eplacement ($\texttt{MTR}$), which replaces the mask token with a portion of each tokenized column; $\texttt{MTR}$ takes advantage of the properties of Transformer, which is becoming the predominant DL-based architecture for tabular data, to perform data augmentation for each column embedding. Through experiments with 13 diverse public datasets in both supervised and self-supervised learning scenarios, we show that $\texttt{MTR}$ achieves competitive performance against existing data augmentation methods and improves model performance. In addition, we discuss specific scenarios in which $\texttt{MTR}$ is most effective and identify the scope of its application. The code is available at <a class="link-external link-https" href="https://github.com/somaonishi/MTR/" rel="external noopener nofollow">this https URL</a>.

Unlocking the Transferability of Tokens in Deep Models for Tabular Data

Tokenize features, enhancing tables: the FT-TABPFN model for tabular classification

Transfer Learning with Deep Tabular Models

Making Pre-trained Language Models Great on Tabular Prediction

Cross-Table Pretraining towards a Universal Function Space for Heterogeneous Tabular Data

Deep Model Transferability from Attribution Maps

TTNet: Tabular Transfer Network for Few-samples Prediction

XTab: Cross-table Pretraining for Tabular Transformers

Rethinking Data Augmentation for Tabular Data in Deep Learning

PTab: Using the Pre-trained Language Model for Modeling Tabular Data

CARTE: Pretraining and Transfer for Tabular Learning

Token-Aware Virtual Adversarial Training in Natural Language Understanding.

Training-Free Generalization on Heterogeneous Tabular Data via Meta-Representation

Robust Tickets Can Transfer Better: Drawing More Transferable Subnetworks in Transfer Learning

Deep Feature Embedding for Tabular Data

Knowledge Distillation via Token-level Relationship Graph

Zoo-Tuning: Adaptive Transfer from A Zoo of Models.

UniTabE: A Universal Pretraining Protocol for Tabular Foundation Model in Data Science

Bridge the Gap between Language models and Tabular Understanding

Tree-Regularized Tabular Embeddings

TabDPT: Scaling Tabular Foundation Models