Abstract:A myriad of different Large Language Models (LLMs) face a common challenge in contextually analyzing table question-answering tasks. These challenges are engendered from (1) finite context windows for large tables, (2) multi-faceted discrepancies amongst tokenization patterns against cell boundaries, and (3) various limitations stemming from data confidentiality in the process of using external models such as gpt-3.5-turbo. We propose a cooperative game dubbed "HiddenTables" as a potential resolution to this challenge. In essence, "HiddenTables" is played between the code-generating LLM "Solver" and the "Oracle" which evaluates the ability of the LLM agents to solve Table QA tasks. This game is based on natural language schemas and importantly, ensures the security of the underlying data. We provide evidential experiments on a diverse set of tables that demonstrate an LLM's collective inability to generalize and perform on complex queries, handle compositional dependencies, and align natural language to programmatic commands when concrete table schemas are provided. Unlike encoder-based models, we have pushed the boundaries of "HiddenTables" to not be limited by the number of rows - therefore we exhibit improved efficiency in prompt and completion tokens. Our infrastructure has spawned a new dataset "PyQTax" that spans across 116,671 question-table-answer triplets and provides additional fine-grained breakdowns & labels for varying question taxonomies. Therefore, in tandem with our academic contributions regarding LLMs' deficiency in TableQA tasks, "HiddenTables" is a tactile manifestation of how LLMs can interact with massive datasets while ensuring data security and minimizing generation costs.

What problem does this paper attempt to address?

### Problems the Paper Attempts to Solve This paper aims to address some common challenges faced by large language models (LLMs) when handling table question-answering tasks. Specifically, these challenges include: 1. **Limited Context Window**: For large tables, the context window of LLMs is limited and cannot process the entire table at once. 2. **Discrepancies Between Tokenization Patterns and Cell Boundaries**: Different tokenization patterns may lead to misunderstandings of the table content. 3. **Data Confidentiality Restrictions**: When using external models (such as gpt-3.5-turbo), data confidentiality is a significant issue, as the raw data cannot be freely exposed. To address these challenges, the authors propose a collaborative game called "HiddenTables." In this game, there are two roles: an LLM "Solver" that generates code and an "Oracle" that evaluates the code. The Solver relies solely on the table schema and instructions provided by the Oracle to generate code, while the Oracle ensures data security and evaluates whether the Solver's generated code correctly answers the user's question. In this way, HiddenTables not only addresses the aforementioned challenges but also generates a new dataset called "PyQTax," which contains 116,671 question-table-answer-code quadruples, covering various types of tables and questions. This provides a rich resource for future research. ### Main Contributions 1. **Proposed a New Collaborative Game Framework**: HiddenTables serves as an alternative method to effectively solve table question-answering tasks without exposing complete data. 2. **Improved Code Generation Capability**: Through program commands, LLMs can better demonstrate their problem-solving approach, thereby enhancing the interpretability of the answers. 3. **Showcased LLMs' Performance with Partial Information**: Experiments show that even without providing complete data, LLMs can gradually improve accuracy after several rounds of feedback. 4. **Generated a New Dataset**: The PyQTax dataset provides a rich resource for future research, covering various types of tables and questions. In summary, this paper not only offers a new technical solution but also makes significant contributions to data privacy and security.

HiddenTables & PyQTax: A Cooperative Game and Dataset For TableQA to Ensure Scale and Data Privacy Across a Myriad of Taxonomies

TQA-Bench: Evaluating LLMs for Multi-Table Question Answering with Scalable Context and Symbolic Extension

Benchmarking Table Comprehension In The Wild

Tree-of-Table: Unleashing the Power of LLMs for Enhanced Large-Scale Table Understanding

Table Meets LLM: Can Large Language Models Understand Structured Table Data? A Benchmark and Empirical Study

Uncovering Limitations of Large Language Models in Information Seeking from Tables

TableBench: A Comprehensive and Complex Benchmark for Table Question Answering

CABINET: Content Relevance based Noise Reduction for Table Question Answering

Bridging the Gap: Deciphering Tabular Data Using Large Language Model

CRT-QA: A Dataset of Complex Reasoning Question Answering over Tabular Data

Piece of Table: A Divide-and-Conquer Approach for Selecting Sub-Tables in Table Question Answering

TableGPT: Towards Unifying Tables, Nature Language and Commands into One GPT

HiTab: A Hierarchical Table Dataset for Question Answering and Natural Language Generation

Large Language Models are few(1)-shot Table Reasoners

MultiTabQA: Generating Tabular Answers for Multi-Table Question Answering

Rethinking Tabular Data Understanding with Large Language Models

DocTabQA: Answering Questions from Long Documents Using Tables

Large Language Models are Complex Table Parsers

Accurate and Regret-aware Numerical Problem Solver for Tabular Question Answering

TabSQLify: Enhancing Reasoning Capabilities of LLMs Through Table Decomposition