Abstract:Tabular data is foundational to predictive modeling in various crucial industries, including healthcare, finance, retail, sustainability, etc. Despite the progress made in specialized models, there is an increasing demand for universal models that can transfer knowledge, generalize from limited data, and follow human instructions. These are challenges that current tabular deep learning approaches have not fully tackled. Here we introduce Generative Tabular Learning (GTL), a novel framework that integrates the advanced functionalities of large language models (LLMs)-such as prompt-based zero-shot generalization and in-context learning-into tabular deep learning. GTL capitalizes on the pre-training of LLMs on diverse tabular data, enhancing their understanding of domain-specific knowledge, numerical sequences, and statistical dependencies critical for accurate predictions. Our empirical study spans 384 public datasets, rigorously analyzing GTL's convergence and scaling behaviors and assessing the impact of varied data templates. The GTL-enhanced LLaMA-2 model demonstrates superior zero-shot and in-context learning capabilities across numerous classification and regression tasks. Notably, it achieves this without fine-tuning, outperforming traditional methods and rivaling state-of-the-art models like GPT-4 in certain cases. Through GTL, we not only foster a deeper integration of LLMs' sophisticated abilities into tabular data comprehension and application but also offer a new training resource and a test bed for LLMs to enhance their ability to comprehend tabular data. To facilitate reproducible research, we release our code, data, and model checkpoints at <a class="link-external link-https" href="https://github.com/microsoft/Industrial-Foundation-Models" rel="external noopener nofollow">this https URL</a>.

What problem does this paper attempt to address?

The problem that this paper attempts to solve is the lack of generality and adaptability in existing deep - learning methods for tabular data. Specifically, although significant progress has been made on specific tasks, these methods still have limitations in the following aspects: 1. **Knowledge transfer ability**: Existing models perform poorly when transferred to new datasets, especially when the amount of data is limited. 2. **Zero - shot learning and context - learning ability**: Although some research has attempted to introduce zero - shot learning and context - learning, the application scope of these methods is limited and the effect is not ideal. 3. **Understanding of tabular data characteristics**: When processing tabular data, existing large - language models (LLMs) often have difficulty understanding numerical features and statistical dependencies, which limits their application effectiveness on tabular data. To overcome these problems, the paper proposes a new paradigm - Generative Tabular Learning (GTL). GTL enhances the understanding of tabular data by continuing to pre - train large - language models (LLMs) and enables them to have better zero - shot learning and context - learning abilities. Specifically, the main contributions of GTL include: - **Generative learning paradigm**: GTL integrates the advanced functions of LLMs (such as prompt - based zero - shot generalization and context - learning) into tabular deep - learning. By continuing to pre - train on diverse tabular data, it enhances the model's understanding of domain - specific knowledge, numerical sequences, and statistical dependencies. - **Data construction pipeline**: GTL develops a comprehensive data construction pipeline that converts tabular data instances into an instruction - oriented language format, supporting further research and model training. - **Experimental verification**: Through a large number of experiments on 384 public tabular datasets, the effectiveness of GTL has been proven. The experimental results show that GTL performs excellently in zero - shot and context - learning tasks and even exceeds the state - of - the - art models (such as GPT - 4) in some cases. In conclusion, this paper aims to improve the generality, adaptability, and prediction ability of deep - learning models for tabular data through the GTL framework.

From Supervised to Generative: A Novel Paradigm for Tabular Deep Learning with Large Language Models

From supervised to generative: A novel paradigm for tabular deep learning with large language models

Scaling Generative Tabular Learning for Large Language Models

Unleashing the Potential of Large Language Models for Predictive Tabular Tasks in Data Science

Generating Realistic Tabular Data with Large Language Models

Large language models (LLMs) on tabular data: Prediction, generation, and understanding-a survey

Large Language Models(LLMs) on Tabular Data: Prediction, Generation, and Understanding -- A Survey

Large Scale Transfer Learning for Tabular Data via Language Modeling

TabLLM: Few-shot Classification of Tabular Data with Large Language Models

Table Meets LLM: Can Large Language Models Understand Structured Table Data? A Benchmark and Empirical Study

TableGPT2: A Large Multimodal Model with Tabular Data Integration

Supervised Knowledge Makes Large Language Models Better In-context Learners

Towards Foundation Models for Learning on Tabular Data

TableGPT: Towards Unifying Tables, Nature Language and Commands into One GPT

Language Modeling on Tabular Data: A Survey of Foundations, Techniques and Evolution

UniPredict: Large Language Models are Universal Tabular Classifiers

LaTable: Towards Large Tabular Models

Large language models on tabular data--a survey

Bridging the Gap: Deciphering Tabular Data Using Large Language Model

MALLM-GAN: Multi-Agent Large Language Model as Generative Adversarial Network for Synthesizing Tabular Data