Abstract:Large Language Models (LLMs), excel in natural language understanding, but their capability for complex mathematical reasoning with an amalgamation of structured tables and unstructured text is uncertain. This study explores LLMs' mathematical reasoning on four financial tabular question-answering datasets: TATQA, FinQA, ConvFinQA, and Multihiertt. Through extensive experiments with various models and prompting techniques, we assess how LLMs adapt to complex tables and mathematical tasks. We focus on sensitivity to table complexity and performance variations with an increasing number of arithmetic reasoning steps. The results provide insights into LLMs' capabilities and limitations in handling complex mathematical scenarios for semi-structured tables. Ultimately, we introduce a novel prompting technique tailored to semi-structured documents, matching or outperforming other baselines in performance while providing a nuanced understanding of LLMs abilities for such a task.

What problem does this paper attempt to address?

This paper mainly explores the mathematical reasoning ability of large language models (LLMs) in processing complex financial documents that contain structured tables and unstructured texts. The study evaluates how LLMs adapt to complex tables and tasks that require multi-step arithmetic reasoning through extensive experiments on four financial table question answering datasets (TATQA, FinQA, ConvFinQA, and Multihiertt). The authors analyze the performance changes of the models as the complexity of the tables and the number of reasoning steps increase, in order to understand the ability and limitations of LLMs in dealing with complex mathematical scenarios in semi-structured tables. The paper also proposes a novel prompting technique called EEDP (Elicit→Extract→Decompose→Predict), which is designed for semi-structured documents. This technique performs either at par or surpasses existing baseline methods and provides a deep understanding of LLMs' capabilities in this task. The research finds that although LLMs perform well in natural language understanding and a range of NLP tasks, their ability is still limited when it comes to complex tables and problems involving multi-step mathematical operations. The paper also categorizes different types of errors, such as incorrect evidence extraction, reasoning errors, and calculation errors, to gain a deeper understanding of the challenges and limitations of the models in various aspects of mathematical reasoning. In conclusion, this paper aims to fill the gap in the evaluation of LLMs' abilities in handling complex mathematical problems involving semi-structured tables and proposes an improved approach to enhance their performance. It also points out areas that future research may need to focus on, such as the models' handling of domain knowledge and limitations in processing semi-structured data.

Evaluating LLMs' Mathematical Reasoning in Financial Document Question Answering

LLM Reasoning Engine: Specialized Training for Enhanced Mathematical Reasoning

DocMath-Eval: Evaluating Math Reasoning Capabilities of LLMs in Understanding Long and Specialized Documents

Evaluating LLMs' Mathematical and Coding Competency through Ontology-guided Interventions

Can LLMs Master Math? Investigating Large Language Models on Math Stack Exchange

LLMs Are Not Intelligent Thinkers: Introducing Mathematical Topic Tree Benchmark for Comprehensive Evaluation of LLMs

Large Language Models for Mathematical Reasoning: Progresses and Challenges

Zero-Shot Question Answering over Financial Documents using Large Language Models

Mathify: Evaluating Large Language Models on Mathematical Problem Solving Tasks

Multilingual Mathematical Reasoning: Advancing Open-Source LLMs in Hindi and English

Can Language Models Rival Mathematics Students? Evaluating Mathematical Reasoning through Textual Manipulation and Human Experiments

MARIO Eval: Evaluate Your Math LLM with your Math LLM--A mathematical dataset evaluation toolkit

Numerical Reasoning for Financial Reports

Do Large Language Models Truly Grasp Mathematics? An Empirical Exploration From A Psychological Perspective

Can LLMs Compute with Reasons?

FinLLMs: A Framework for Financial Reasoning Dataset Generation with Large Language Models

From Calculation to Adjudication: Examining LLM judges on Mathematical Reasoning Tasks

Evaluating Mathematical Reasoning Beyond Accuracy