Abstract:Language models, such as GPT-3.5 and ChatGPT, demonstrate remarkable abilities to follow diverse human instructions and perform a wide range of tasks. However, when probing language models using a range of basic table-understanding tasks, we observe that today's language models are still sub-optimal in many table-related tasks, likely because they are pre-trained predominantly on \emph{one-dimensional} natural-language texts, whereas relational tables are \emph{two-dimensional} objects. In this work, we propose a new "\emph{table-tuning}" paradigm, where we continue to train/fine-tune language models like GPT-3.5 and ChatGPT, using diverse table-tasks synthesized from real tables as training data, with the goal of enhancing language models' ability to understand tables and perform table tasks. We show that our resulting Table-GPT models demonstrate (1) better \emph{table-understanding} capabilities, by consistently outperforming the vanilla GPT-3.5 and ChatGPT, on a wide-range of table tasks, including holdout unseen tasks, and (2) strong \emph{generalizability}, in its ability to respond to diverse human instructions to perform new table-tasks, in a manner similar to GPT-3.5 and ChatGPT.

What problem does this paper attempt to address?

The paper primarily addresses the limitations of current large language models (such as GPT-3.5 and ChatGPT) in handling tabular data by proposing a new approach. The authors observed that although these models perform excellently on natural language tasks, they still have shortcomings in understanding tabular data. This is mainly because they are pre-trained on one-dimensional natural language text, whereas tabular data is inherently two-dimensional. To solve this problem, the paper proposes a new model called "Table-GPT," which enhances the language model's ability to understand tables through a method called "table-tuning." Specifically, this method involves continuing the training of existing language models (e.g., GPT-3.5 or ChatGPT) using a variety of tabular tasks synthesized from real tables as training data. The goal is to improve the model's ability to understand and execute table-related tasks. The main contributions of the paper are as follows: 1. **Proposed a new "table-tuning" paradigm**: Utilizing a large number of diverse tabular tasks synthesized from real tables to continue training the language model, enhancing its ability to handle tabular data. 2. **Developed various data augmentation techniques**: Including task-level, table-level, instruction-level, and completion-level data augmentation techniques to ensure the generalization ability of the Table-GPT model and avoid overfitting. 3. **Demonstrated the strong performance of the Table-GPT model**: It significantly outperformed the non-tuned GPT-3.5 and ChatGPT models on various known and unknown tabular tasks and adapted well to new tasks, similar to ChatGPT's flexibility in handling natural language tasks. In summary, this paper aims to improve the ability of language models to handle tabular data and experimentally validates the effectiveness and superiority of the proposed Table-GPT model.

Table-GPT: Table-tuned GPT for Diverse Table Tasks

TableGPT: Towards Unifying Tables, Nature Language and Commands into One GPT

Table-LLM-Specialist: Language Model Specialists for Tables using Iterative Generator-Validator Fine-tuning

TableGPT: Few-shot Table-to-Text Generation with Table Structure Reconstruction and Content Matching

TableGPT2: A Large Multimodal Model with Tabular Data Integration

Large Language Models are Complex Table Parsers

Table Meets LLM: Can Large Language Models Understand Structured Table Data? A Benchmark and Empirical Study

TableBench: A Comprehensive and Complex Benchmark for Table Question Answering

Struc-Bench: Are Large Language Models Really Good at Generating Complex Structured Data?

TableLlama: Towards Open Large Generalist Models for Tables

Bridge the Gap between Language models and Tabular Understanding

CT-Eval: Benchmarking Chinese Text-to-Table Performance in Large Language Models

Bridging the Gap: Deciphering Tabular Data Using Large Language Model

Generating Tables from the Parametric Knowledge of Language Models

A comprehensive capability analysis of gpt-3 and gpt-3.5 series models

gTBLS: Generating Tables from Text by Conditional Question Answering

Scaling Generative Tabular Learning for Large Language Models

TAPEX: Table Pre-training Via Learning a Neural SQL Executor

From Supervised to Generative: A Novel Paradigm for Tabular Deep Learning with Large Language Models