Abstract:Data organized in tabular format is ubiquitous in real-world applications, and users often craft tables with biased feature definitions and flexibly set prediction targets of their interests. Thus, a rapid development of a robust, effective, dataset-versatile, user-friendly tabular prediction approach is highly desired. While Gradient Boosting Decision Trees (GBDTs) and existing deep neural networks (DNNs) have been extensively utilized by professional users, they present several challenges for casual users, particularly: (i) the dilemma of model selection due to their different dataset preferences, and (ii) the need for heavy hyperparameter searching, failing which their performances are deemed inadequate. In this paper, we delve into this question: Can we develop a deep learning model that serves as a sure bet solution for a wide range of tabular prediction tasks, while also being user-friendly for casual users? We delve into three key drawbacks of deep tabular models, encompassing: (P1) lack of rotational variance property, (P2) large data demand, and (P3) over-smooth solution. We propose ExcelFormer, addressing these challenges through a semi-permeable attention module that effectively constrains the influence of less informative features to break the DNNs' rotational invariance property (for P1), data augmentation approaches tailored for tabular data (for P2), and attentive feedforward network to boost the model fitting capability (for P3). These designs collectively make ExcelFormer a sure bet solution for diverse tabular datasets. Extensive and stratified experiments conducted on real-world datasets demonstrate that our model outperforms previous approaches across diverse tabular data prediction tasks, and this framework can be friendly to casual users, offering ease of use without the heavy hyperparameter tuning. The codes are available at https://github.com/whatashot/excelformer.

SDTR: Soft Decision Tree Regressor for Tabular Data

NCART: Neural Classification and Regression Tree for Tabular Data

Vanilla Gradient Descent for Oblique Decision Trees

Deep Trees for (Un)structured Data: Tractability, Performance, and Interpretability

Distilling a Neural Network Into a Soft Decision Tree

Distributional Adaptive Soft Regression Trees

A generalized decision tree ensemble based on the NeuralNetworks architecture: Distributed Gradient Boosting Forest (DGBF)

Can a Deep Learning Model Be a Sure Bet for Tabular Prediction?

Can Differentiable Decision Trees Enable Interpretable Reward Learning from Human Feedback?

Escaping the Forest: Sparse Interpretable Neural Networks for Tabular Data

Tabular Data: Is Attention All You Need?

Learn Decision Trees with Deep Visual Primitives

Learning Deep Nearest Neighbor Representations Using Differentiable Boundary Trees

ExcelFormer: A neural network surpassing GBDTs on tabular data

ST-Tree with Interpretability for Multivariate Time Series Classification

Differentiable Tree Search Network

Improving the Validity of Decision Trees as Explanations

When are Deep Networks really better than Decision Forests at small sample sizes, and how?

Tabular Data: Deep Learning is Not All You Need

Learning Binary Decision Trees by Argmin Differentiation

TabR: Tabular Deep Learning Meets Nearest Neighbors in 2023