VerityMath: Advancing Mathematical Reasoning by Self-Verification Through Unit Consistency

Vernon Toh Yan Han,Ratish Puduppully,Nancy F. Chen
2024-07-21
Abstract:Large Language Models (LLMs), combined with program-based solving techniques, are increasingly demonstrating proficiency in mathematical reasoning. For example, closed-source models such as OpenAI GPT-4 and Claude show excellent results in solving math word problems. However, progress in math word problem-solving for open-source LLMs is limited, and the challenges these models face are not well-studied. In this paper, we study the performance of strong open-source LLMs, including Llama 2 (7B), Code Llama (7B), and Mistral (7B) on math word problems using program-based solving techniques. Specifically, we analyze the outputs of these models when applied to math word problems and identify a category of problems that pose a significant challenge, particularly those involving quantities spanning multiple units. To address this issue, we propose a systematic approach by defining the units for each quantity and ensuring the consistency of these units during mathematical operations. We developed Unit Consistency Programs (UCPs), an annotated dataset of math word problems, each paired with programs containing unit specifications and unit verification routines. We fine-tuned Llama 2 (7B), Code Llama (7B), and Mistral (7B) models with UCPs to produce theirVerityMath variants. Our findings indicate that our approach, which incorporates unit consistency, currently slightly underperforms compared to an approach that does not. To understand the reasons behind this, we conduct an in-depth error analysis and suggest options for future improvements. Our code and dataset are available at <a class="link-external link-https" href="https://github.com/vernontoh/VerityMath" rel="external noopener nofollow">this https URL</a>.
Computation and Language,Programming Languages
What problem does this paper attempt to address?
The problem this paper attempts to address is the challenge faced by open-source large language models (LLMs) in solving mathematical word problems, particularly those involving multiple units. Specifically, the paper examines the performance of strong open-source LLMs, including Llama 2 (7B), Code Llama (7B), and Mistral (7B), in solving mathematical word problems and finds that these models exhibit significant difficulties when dealing with mathematical problems involving multiple units. To tackle this issue, the authors propose a systematic approach to improve the models' performance by defining the units of each quantity and ensuring the consistency of these units in mathematical operations. To this end, they developed "Unit Consistency Programs" (UCPs), which is an annotated dataset of mathematical word problems along with unit specifications and unit verification programs. By fine-tuning the models using UCPs, the authors hope to enhance the models' reasoning and verification capabilities in solving complex mathematical problems. Summary: - **Problem**: Open-source large language models perform poorly in solving mathematical word problems involving multiple units. - **Solution**: Introduce "Unit Consistency Programs" (UCPs) to improve the models' reasoning abilities by defining units and verifying unit consistency.