Abstract:Tabular data, a prevalent data type across various domains, presents unique challenges due to its heterogeneous nature and complex structural relationships. Achieving high predictive performance and robustness in tabular data analysis holds significant promise for numerous applications. Influenced by recent advancements in natural language processing, particularly transformer architectures, new methods for tabular data modeling have emerged. Early techniques concentrated on pre-training transformers from scratch, often encountering scalability issues. Subsequently, methods leveraging pre-trained language models like BERT have been developed, which require less data and yield enhanced performance. The recent advent of large language models, such as GPT and LLaMA, has further revolutionized the field, facilitating more advanced and diverse applications with minimal fine-tuning. Despite the growing interest, a comprehensive survey of language modeling techniques for tabular data remains absent. This paper fills this gap by providing a systematic review of the development of language modeling for tabular data, encompassing: (1) a categorization of different tabular data structures and data types; (2) a review of key datasets used in model training and tasks used for evaluation; (3) a summary of modeling techniques including widely-adopted data processing methods, popular architectures, and training objectives; (4) the evolution from adapting traditional Pre-training/Pre-trained language models to the utilization of large language models; (5) an identification of persistent challenges and potential future research directions in language modeling for tabular data analysis. GitHub page associated with this survey is available at: <a class="link-external link-https" href="https://github.com/lanxiang1017/Language-Modeling-on-Tabular-Data-Survey.git" rel="external noopener nofollow">this https URL</a>.

Large language models on tabular data--a survey

Large language models (LLMs) on tabular data: Prediction, generation, and understanding-a survey

Large Language Models(LLMs) on Tabular Data: Prediction, Generation, and Understanding -- A Survey

Language Modeling on Tabular Data: A Survey of Foundations, Techniques and Evolution

Large Language Model for Table Processing: A Survey

Large Language Models (LLMs) on Tabular Data: Prediction, Generation, and Understanding—A Survey. arXiv 2024

Unleashing the Potential of Large Language Models for Predictive Tabular Tasks in Data Science

Bridging the Gap: Deciphering Tabular Data Using Large Language Model

Large Language Models for Time Series: A Survey

Datasets for Large Language Models: A Comprehensive Survey

Rethinking Tabular Data Understanding with Large Language Models

Natural Language Interfaces for Tabular Data Querying and Visualization: A Survey

Generating Realistic Tabular Data with Large Language Models

Large Scale Transfer Learning for Tabular Data via Language Modeling

From supervised to generative: A novel paradigm for tabular deep learning with large language models

Empowering Time Series Analysis with Large Language Models: A Survey

History, Development, and Principles of Large Language Models-An Introductory Survey

A Survey of Large Language Models

Efficient Large Language Models: A Survey