Abstract:We have described a novel approach for training tabular data using the TabTransformer model with self-supervised learning. Traditional machine learning models for tabular data, such as GBDT are being widely used though our paper examines the effectiveness of the TabTransformer which is a Transformer based model optimised specifically for tabular data. The TabTransformer captures intricate relationships and dependencies among features in tabular data by leveraging the self-attention mechanism of Transformers. We have used a self-supervised learning approach in this study, where the TabTransformer learns from unlabelled data by creating surrogate supervised tasks, eliminating the need for the labelled data. The aim is to find the most effective TabTransformer model representation of categorical and numerical features. To address the challenges faced during the construction of various input settings into the Transformers. Furthermore, a comparative analysis is also been conducted to examine performance of the TabTransformer model against baseline models such as MLP and supervised TabTransformer.

What problem does this paper attempt to address?

### What problems does this paper attempt to solve? This master's thesis aims to address the challenges in the application of deep learning to tabular data. Specifically, the goal of the paper is to train tabular data through self - supervised learning methods to improve its accuracy and performance. The following are the main problems that the paper attempts to solve: 1. **Complexity of tabular data**: Tabular data usually contains a mixture of numerical and categorical features, and the relationships between these features are relatively complex. Traditional machine - learning models such as Gradient Boosting Decision Trees (GBDT) perform well on tabular data, but the performance of deep - learning models on this type of data has not yet reached the optimal state. 2. **Dependence on labeled data**: Deep - learning models usually rely on a large amount of labeled data for training, but in practical applications, labeled data may be difficult to obtain or costly. Therefore, the paper proposes using self - supervised learning methods, enabling the model to learn from unlabeled data, thereby reducing the dependence on labeled data. 3. **Effectiveness of feature representation**: The paper attempts to find the most effective methods for representing categorical and numerical features. There are a wide variety of features in tabular data. How to effectively input these features into the Transformer model and let the model capture the complex relationships between features is an important research direction. 4. **Generalization ability of the model**: Tabular data sets are usually much smaller than those in the fields of computer vision or natural language processing, lacking large - scale general - purpose data sets. Therefore, the paper hopes to improve the generalization ability of the model through self - supervised learning, enabling it to perform well in different tasks without a large amount of task - specific labeled data. 5. **Interpretability of the model**: Deep - learning models are often considered to lack interpretability, which limits their applications in certain fields. The paper hopes to enhance the interpretability of the model by improving the model structure and training methods, enabling it to be trusted and used in more application scenarios. ### Solutions To address the above challenges, the paper proposes the following solutions: - **TabTransformer model**: Use the TabTransformer model based on the Transformer architecture. This model is specifically optimized for tabular data and can capture the complex relationships between features through the self - attention mechanism. - **Self - supervised learning method**: Adopt a self - supervised learning method. By creating surrogate supervised tasks, the model can be pre - trained on unlabeled data, thereby improving the generalization ability and performance of the model. - **Feature representation optimization**: Experiment with different input settings to find the optimal methods for representing categorical and numerical features to better adapt to the characteristics of tabular data. - **Comparative experiments**: Conduct comparative experiments between the TabTransformer model and traditional machine - learning models (such as MLP, GBDT) and the supervised - learning TabTransformer model to evaluate the effectiveness of the self - supervised learning method. Through these methods, the paper hopes to provide new ideas and technical means for the deep - learning application of tabular data and promote the further development of this field.

Deep Learning with Tabular Data: A Self-supervised Approach

A Survey on Deep Tabular Learning

Transfer Learning with Deep Tabular Models

XTab: Cross-table Pretraining for Tabular Transformers

Transformers with Stochastic Competition for Tabular Data Modelling

Tabular Transformers for Modeling Multivariate Time Series

TabGSL: Graph Structure Learning for Tabular Data Prediction

TabDPT: Scaling Tabular Foundation Models

Revisiting Deep Learning Models for Tabular Data

Tabular Data: Is Attention All You Need?

Time Sequence Deep Learning Model for Ubiquitous Tabular Data with Unique 3D Tensors Manipulation

Why In-Context Learning Transformers are Tabular Data Classifiers

PyTorch Tabular: A Framework for Deep Learning with Tabular Data

PTab: Using the Pre-trained Language Model for Modeling Tabular Data

From Supervised to Generative: A Novel Paradigm for Tabular Deep Learning with Large Language Models

Deep Feature Embedding for Tabular Data

Rethinking Data Augmentation for Tabular Data in Deep Learning

MambaTab: A Plug-and-Play Model for Learning Tabular Data

Retrieval & Fine-Tuning for In-Context Tabular Models

TabM: Advancing Tabular Deep Learning with Parameter-Efficient Ensembling

UniTabE: A Universal Pretraining Protocol for Tabular Foundation Model in Data Science