Abstract:Tabular data is the most widely used data format in machine learning (ML). While tree-based methods outperform DL-based methods in supervised learning, recent literature reports that self-supervised learning with Transformer-based models outperforms tree-based methods. In the existing literature on self-supervised learning for tabular data, contrastive learning is the predominant method. In contrastive learning, data augmentation is important to generate different views. However, data augmentation for tabular data has been difficult due to the unique structure and high complexity of tabular data. In addition, three main components are proposed together in existing methods: model structure, self-supervised learning methods, and data augmentation. Therefore, previous works have compared the performance without comprehensively considering these components, and it is not clear how each component affects the actual performance. In this study, we focus on data augmentation to address these issues. We propose a novel data augmentation method, $\textbf{M}$ask $\textbf{T}$oken $\textbf{R}$eplacement ($\texttt{MTR}$), which replaces the mask token with a portion of each tokenized column; $\texttt{MTR}$ takes advantage of the properties of Transformer, which is becoming the predominant DL-based architecture for tabular data, to perform data augmentation for each column embedding. Through experiments with 13 diverse public datasets in both supervised and self-supervised learning scenarios, we show that $\texttt{MTR}$ achieves competitive performance against existing data augmentation methods and improves model performance. In addition, we discuss specific scenarios in which $\texttt{MTR}$ is most effective and identify the scope of its application. The code is available at <a class="link-external link-https" href="https://github.com/somaonishi/MTR/" rel="external noopener nofollow">this https URL</a>.

MSLR: A Self-supervised Representation Learning Method for Tabular Data Based on Multi-scale Ladder Reconstruction

MsVRL: Self-Supervised Multiscale Visual Representation Learning Via Cross-Level Consistency for Medical Image Segmentation

Distributionally robust self-supervised learning for tabular data

A Deep Multi-Task Representation Learning Method for Time Series Classification and Retrieval.

MambaTab: A Plug-and-Play Model for Learning Tabular Data

Multi-scale Self-Supervised Representation Learning with Temporal Alignment for Multi-Rate Time Series Modeling

MDLR: A Multi-Task Disentangled Learning Representations for unsupervised time series domain adaptation

PTab: Using the Pre-trained Language Model for Modeling Tabular Data

Unleashing the Potential of Large Language Models for Predictive Tabular Tasks in Data Science

An Unsupervised Short- and Long-Term Mask Representation for Multivariate Time Series Anomaly Detection

Self-Paced Unified Representation Learning for Hierarchical Multi-Label Classification

Predictive Multi-level Patient Representations from Electronic Health Records

Tackling Ordinal Regression Problem for Heterogeneous Data: Sparse and Deep Multi-Task Learning Approaches

Understanding the limitations of self-supervised learning for tabular anomaly detection

Multi-Level Cascade Sparse Representation Learning for Small Data Classification

MTS-LOF: Medical Time-Series Representation Learning via Occlusion-Invariant Features

Self-Reinforcement Attention Mechanism For Tabular Learning

Rethinking Data Augmentation for Tabular Data in Deep Learning

XTab: Cross-table Pretraining for Tabular Transformers

Making Pre-trained Language Models Great on Tabular Prediction