Abstract:The advent of large language models (LLMs) has unlocked great opportunities in complex data management tasks, particularly in question answering (QA) over complicated multi-table relational data. Despite significant progress, systematically evaluating LLMs on multi-table QA remains a critical challenge due to the inherent complexity of analyzing heterogeneous table structures and potential large scale of serialized relational data. Existing benchmarks primarily focus on single-table QA, failing to capture the intricacies of reasoning across multiple relational tables, as required in real-world domains such as finance, healthcare, and e-commerce. To address this gap, we present TQA-Bench, a new multi-table QA benchmark designed to evaluate the capabilities of LLMs in tackling complex QA tasks over relational data. Our benchmark incorporates diverse relational database instances sourced from real-world public datasets and introduces a flexible sampling mechanism to create tasks with varying multi-table context lengths, ranging from 8K to 64K tokens. To ensure robustness and reliability, we integrate symbolic extensions into the evaluation framework, enabling the assessment of LLM reasoning capabilities beyond simple data retrieval or probabilistic pattern matching. We systematically evaluate a range of LLMs, both open-source and closed-source, spanning model scales from 7 billion to 70 billion parameters. Our extensive experiments reveal critical insights into the performance of LLMs in multi-table QA, highlighting both challenges and opportunities for advancing their application in complex, data-driven environments. Our benchmark implementation and results are available at <a class="link-external link-https" href="https://github.com/Relaxed-System-Lab/TQA-Bench" rel="external noopener nofollow">this https URL</a>.

DateLogicQA: Benchmarking Temporal Biases in Large Language Models

Towards Benchmarking and Improving the Temporal Reasoning Capability of Large Language Models

Test of Time: A Benchmark for Evaluating LLMs on Temporal Reasoning

MenatQA: A New Dataset for Testing the Temporal Comprehension and Reasoning Abilities of Large Language Models

Living in the Moment: Can Large Language Models Grasp Co-Temporal Reasoning?

Unveiling Divergent Inductive Biases of LLMs on Temporal Data

TimeBench: A Comprehensive Evaluation of Temporal Reasoning Abilities in Large Language Models

Quantifying Bias in Agentic Large Language Models: A Benchmarking Approach

BabelBench: An Omni Benchmark for Code-Driven Analysis of Multimodal and Multistructured Data

Remember This Event That Year? Assessing Temporal Information and Reasoning in Large Language Models

Temporal Blind Spots in Large Language Models

STBench: Assessing the Ability of Large Language Models in Spatio-Temporal Analysis

TQA-Bench: Evaluating LLMs for Multi-Table Question Answering with Scalable Context and Symbolic Extension

LogicBench: Towards Systematic Evaluation of Logical Reasoning Ability of Large Language Models

EconLogicQA: A Question-Answering Benchmark for Evaluating Large Language Models in Economic Sequential Reasoning

Towards Robust Temporal Reasoning of Large Language Models via a Multi-Hop QA Dataset and Pseudo-Instruction Tuning

Cognitive Bias in Decision-Making with LLMs

Analyzing Social Biases in Japanese Large Language Models

LTLBench: Towards Benchmarks for Evaluating Temporal Logic Reasoning in Large Language Models

Is Your LLM Outdated? Evaluating LLMs at Temporal Generalization