Abstract:Code has been shown to be effective in enhancing the mathematical reasoning abilities of large language models due to its precision and accuracy. Previous works involving continued mathematical pretraining often include code that utilizes math-related packages, which are primarily designed for fields such as engineering, machine learning, signal processing, or module testing, rather than being directly focused on mathematical reasoning. In this paper, we introduce a novel method for generating mathematical code accompanied with corresponding reasoning steps for continued pretraining. Our approach begins with the construction of a high-quality mathematical continued pretraining dataset by incorporating math-related web data, code using mathematical packages, math textbooks, and synthetic data. Next, we construct reasoning steps by extracting LaTeX expressions, the conditions needed for the expressions, and the results of the expressions from the previously collected dataset. Based on this extracted information, we generate corresponding code to accurately capture the mathematical reasoning process. Appending the generated code to each reasoning step results in data consisting of paired natural language reasoning steps and their corresponding code. Combining this data with the original dataset results in a 19.2B-token high-performing mathematical pretraining corpus, which we name MathCode-Pile. Training several popular base models with this corpus significantly improves their mathematical abilities, leading to the creation of the MathCoder2 family of models. All of our data processing and training code is open-sourced, ensuring full transparency and easy reproducibility of the entire data collection and training pipeline. The code is released at <a class="link-external link-https" href="https://github.com/mathllm/MathCoder2" rel="external noopener nofollow">this https URL</a> .

Pre-Calc: Learning to Use the Calculator Improves Numeracy in Language Models

Pre-training and Evaluation of Numeracy-Oriented Language Model.

Arithmetic-Based Pretraining -- Improving Numeracy of Pretrained Language Models

MathCoder2: Better Math Reasoning from Continued Pretraining on Model-translated Mathematical Code

Measuring and Improving BERT's Mathematical Abilities by Predicting the Order of Reasoning

Laying Anchors: Semantically Priming Numerals in Language Modeling

Injecting Numerical Reasoning Skills into Language Models

Number Cookbook: Number Understanding of Language Models and How to Improve It

Overcoming Barriers to Skill Injection in Language Modeling: Case Study in Arithmetic

Reverse That Number! Decoding Order Matters in Arithmetic Learning

An Effective Framework to Help Large Language Models Handle Numeric-involved Long-context Tasks

Evaluating Transformer Language Models on Arithmetic Operations Using Number Decomposition

MathBERT: A Pre-trained Language Model for General NLP Tasks in Mathematics Education

Reflection of Thought: Inversely Eliciting Numerical Reasoning in Language Models via Solving Linear Systems

Masked Measurement Prediction: Learning to Jointly Predict Quantities and Units from Textual Context

Investigating the Limitations of Transformers with Simple Arithmetic Tasks

MATHSENSEI: A Tool-Augmented Large Language Model for Mathematical Reasoning

Mathify: Evaluating Large Language Models on Mathematical Problem Solving Tasks

AutoMathText: Autonomous Data Selection with Language Models for Mathematical Texts

FERMAT: An Alternative to Accuracy for Numerical Reasoning

NumeroLogic: Number Encoding for Enhanced LLMs' Numerical Reasoning