Solving Math Word Problems by Combining Language Models With Symbolic Solvers

Joy He-Yueya,Gabriel Poesia,Rose E. Wang,Noah D. Goodman

2023-04-16

Abstract:Automatically generating high-quality step-by-step solutions to math word problems has many applications in education. Recently, combining large language models (LLMs) with external tools to perform complex reasoning and calculation has emerged as a promising direction for solving math word problems, but prior approaches such as Program-Aided Language model (PAL) are biased towards simple procedural problems and less effective for problems that require declarative reasoning. We propose an approach that combines an LLM that can incrementally formalize word problems as a set of variables and equations with an external symbolic solver that can solve the equations. Our approach achieves comparable accuracy to the original PAL on the GSM8K benchmark of math word problems and outperforms PAL by an absolute 20% on ALGEBRA, a new dataset of more challenging word problems extracted from Algebra textbooks. Our work highlights the benefits of using declarative and incremental representations when interfacing with an external tool for solving complex math word problems. Our data and prompts are publicly available at <a class="link-external link-https" href="https://github.com/joyheyueya/declarative-math-word-problem" rel="external noopener nofollow">this https URL</a>.

Computation and Language,Artificial Intelligence

What problem does this paper attempt to address?

The problem that this paper attempts to solve is how to automatically generate high - quality step - by - step solutions for math word problems. Specifically, the authors propose a method that combines large - language models (LLMs) with external symbolic solvers to improve the ability to solve math word problems. This method pays special attention to problems that require declarative reasoning, not just simple procedural problems. In this way, the paper aims to overcome the limitations of existing methods in dealing with complex math word problems and provide more effective solutions.

The main contributions of the paper are as follows:
1. **Combining LLM with Symbolic Solvers**: By having the LLM gradually formalize math word problems into variables and systems of equations, and then using an external symbolic solver to solve these systems of equations, the errors of the LLM in performing arithmetic operations are avoided.
2. **Introducing a New Dataset**: To evaluate the performance of different methods on more complex problems, the authors constructed a new dataset named ALGEBRA, which contains 222 more challenging math word problems extracted from algebra textbooks.
3. **Performance Improvement**: The experimental results show that this method achieves an accuracy comparable to the original PAL method in the GSM8K benchmark test, and has an absolute accuracy improvement of 20% over the PAL method on the ALGEBRA dataset.

Through these contributions, the paper demonstrates the effectiveness of using declarative and incremental formalization methods combined with external tools in solving complex math word problems.