Abstract:Tabular data, a prevalent data type across various domains, presents unique challenges due to its heterogeneous nature and complex structural relationships. Achieving high predictive performance and robustness in tabular data analysis holds significant promise for numerous applications. Influenced by recent advancements in natural language processing, particularly transformer architectures, new methods for tabular data modeling have emerged. Early techniques concentrated on pre-training transformers from scratch, often encountering scalability issues. Subsequently, methods leveraging pre-trained language models like BERT have been developed, which require less data and yield enhanced performance. The recent advent of large language models, such as GPT and LLaMA, has further revolutionized the field, facilitating more advanced and diverse applications with minimal fine-tuning. Despite the growing interest, a comprehensive survey of language modeling techniques for tabular data remains absent. This paper fills this gap by providing a systematic review of the development of language modeling for tabular data, encompassing: (1) a categorization of different tabular data structures and data types; (2) a review of key datasets used in model training and tasks used for evaluation; (3) a summary of modeling techniques including widely-adopted data processing methods, popular architectures, and training objectives; (4) the evolution from adapting traditional Pre-training/Pre-trained language models to the utilization of large language models; (5) an identification of persistent challenges and potential future research directions in language modeling for tabular data analysis. GitHub page associated with this survey is available at: <a class="link-external link-https" href="https://github.com/lanxiang1017/Language-Modeling-on-Tabular-Data-Survey.git" rel="external noopener nofollow">this https URL</a>.

From supervised to generative: A novel paradigm for tabular deep learning with large language models

From Supervised to Generative: A Novel Paradigm for Tabular Deep Learning with Large Language Models

Generating Realistic Tabular Data with Large Language Models

Unleashing the Potential of Large Language Models for Predictive Tabular Tasks in Data Science

TabLLM: Few-shot Classification of Tabular Data with Large Language Models

Large Language Models(LLMs) on Tabular Data: Prediction, Generation, and Understanding -- A Survey

UniPredict: Large Language Models are Universal Tabular Classifiers

Large Scale Transfer Learning for Tabular Data via Language Modeling

LaTable: Towards Large Tabular Models

MALLM-GAN: Multi-Agent Large Language Model as Generative Adversarial Network for Synthesizing Tabular Data

TabuLa: Harnessing Language Models for Tabular Data Synthesis

Towards Foundation Models for Learning on Tabular Data

Making Pre-trained Language Models Great on Tabular Prediction

Gradient Boosting Trees and Large Language Models for Tabular Data Few-Shot Learning

Language Models are Realistic Tabular Data Generators

Large Language Models Can Automatically Engineer Features for Few-Shot Tabular Learning

Transfer Learning with Deep Tabular Models

Language Modeling on Tabular Data: A Survey of Foundations, Techniques and Evolution

Automated Data Visualization from Natural Language via Large Language Models: An Exploratory Study

Bridging the Gap: Deciphering Tabular Data Using Large Language Model

TableGPT2: A Large Multimodal Model with Tabular Data Integration