Abstract:Tabular data is the most widely used data format in machine learning (ML). While tree-based methods outperform DL-based methods in supervised learning, recent literature reports that self-supervised learning with Transformer-based models outperforms tree-based methods. In the existing literature on self-supervised learning for tabular data, contrastive learning is the predominant method. In contrastive learning, data augmentation is important to generate different views. However, data augmentation for tabular data has been difficult due to the unique structure and high complexity of tabular data. In addition, three main components are proposed together in existing methods: model structure, self-supervised learning methods, and data augmentation. Therefore, previous works have compared the performance without comprehensively considering these components, and it is not clear how each component affects the actual performance. In this study, we focus on data augmentation to address these issues. We propose a novel data augmentation method, $\textbf{M}$ask $\textbf{T}$oken $\textbf{R}$eplacement ($\texttt{MTR}$), which replaces the mask token with a portion of each tokenized column; $\texttt{MTR}$ takes advantage of the properties of Transformer, which is becoming the predominant DL-based architecture for tabular data, to perform data augmentation for each column embedding. Through experiments with 13 diverse public datasets in both supervised and self-supervised learning scenarios, we show that $\texttt{MTR}$ achieves competitive performance against existing data augmentation methods and improves model performance. In addition, we discuss specific scenarios in which $\texttt{MTR}$ is most effective and identify the scope of its application. The code is available at <a class="link-external link-https" href="https://github.com/somaonishi/MTR/" rel="external noopener nofollow">this https URL</a>.

Semi-Supervised Learning with Data Augmentation for Tabular Data

Boosting Unsupervised Contrastive Learning Using Diffusion-Based Data Augmentation from Scratch

Progressive Feature Upgrade in Semi-supervised Learning on Tabular Domain

Regularizing Deep Networks with Semantic Data Augmentation

SemanticMask: A Contrastive View Design for Anomaly Detection in Tabular Data

T-JEPA: Augmentation-Free Self-Supervised Learning for Tabular Data

TabFedSL: A Self-Supervised Approach to Labeling Tabular Data in Federated Learning Environments

A Survey on Self-Supervised Learning for Non-Sequential Tabular Data

MET: Masked Encoding for Tabular Data

Rethinking Data Augmentation for Tabular Data in Deep Learning

A Simple Baseline for Semi-supervised Semantic Segmentation with Strong Data Augmentation

Tabular Data Contrastive Learning via Class-Conditioned and Feature-Correlation Based Augmentation

Exploring the combination of self and mutual teaching for tabular-data-related semi-supervised regression

TabAug: Data Driven Augmentation for Enhanced Table Structure Recognition

Incorporating Supervised Domain Generalization into Data Augmentation

SDA: Simple Discrete Augmentation for Contrastive Sentence Representation Learning

Context Decoupling Augmentation for Weakly Supervised Semantic Segmentation

Image Data Augmentation for Deep Learning: A Survey

Inter-Class and Inter-Domain Semantic Augmentation for Domain Generalization

Tabular Data Augmentation for Machine Learning: Progress and Prospects of Embracing Generative AI

TabMDA: Tabular Manifold Data Augmentation for Any Classifier using Transformers with In-context Subsetting