Abstract:Large language models (LLMs) can solve an increasing number of complex reasoning tasks while making surprising mistakes in basic numerical understanding and processing (such as 9.11 > 9.9). The latter ability is essential for tackling complex arithmetic and mathematical problems and serves as a foundation for most reasoning tasks, but previous work paid little attention to it or only discussed several restricted tasks (like integer addition). In this paper, we comprehensively investigate the numerical understanding and processing ability (NUPA) of LLMs. Firstly, we introduce a benchmark covering four common numerical representations and 17 distinct numerical tasks in four major categories, resulting in 41 meaningful combinations in total. These tasks are derived from primary and secondary education curricula, encompassing nearly all everyday numerical understanding and processing scenarios, and the rules of these tasks are very simple and clear. Through the benchmark, we find that current LLMs fail frequently in many of the tasks. To study the problem, we train small models with existing and potential techniques for enhancing NUPA (such as special tokenizers, PEs, and number formats), comprehensively evaluating their effectiveness using our testbed. We also finetune practical-scale LLMs on our proposed NUPA tasks and find that 1) naive finetuning can improve NUPA a lot on many but not all tasks, and 2) surprisingly, techniques designed to enhance NUPA prove ineffective for finetuning pretrained models. We further explore the impact of chain-of-thought techniques on NUPA. Our work takes a preliminary step towards understanding and improving NUPA of LLMs. Our benchmark and code are released at <a class="link-external link-https" href="https://github.com/GraphPKU/number_cookbook" rel="external noopener nofollow">this https URL</a>.

A Neural Model of Number Comparison with Surprisingly Robust Generalization

A neural model of how the brain represents and compares multi-digit numbers: spatial and categorical processes

Human Behavioral Benchmarking: Numeric Magnitude Comparison Effects in Large Language Models

The Challenge of Modeling the Acquisition of Mathematical Concepts

A visual sense of number emerges from the dynamics of a recurrent on-center off-surround neural network

Evaluating Visual Number Discrimination in Deep Neural Networks

On Numerosity of Deep Convolutional Neural Networks.

On Numerosity of Deep Convolutional Neural Networks

On Numerosity of Deep Neural Networks

Language Models Encode Numbers Using Digit Representations in Base 10

NumeroLogic: Number Encoding for Enhanced LLMs' Numerical Reasoning

Exploring Early Number Abilities With Multimodal Transformers

Number Cookbook: Number Understanding of Language Models and How to Improve It

Trained deep neural network models of the ventral visual pathway encode numerosity with robustness to object and scene identity

Cognitive Deficit of Deep Learning in Numerosity

Distinct Representations of Subtraction and Multiplication in the Neural Systems for Numerosity and Language.

Children's comparison of different-length numbers: Managing different attributes in multidigit number processing

Estimating Numbers without Regression

Neural representational precision of distance predicts arithmetic performance

A Comprehensive Comparison of Neural Networks as Cognitive Models of Inflection

Contrastive Learning and Abstract Concepts: The Case of Natural Numbers