Abstract:Large language models (LLMs) have demonstrated impressive versatility across numerous tasks, yet their generalization capabilities remain poorly understood. To investigate these behaviors, arithmetic tasks serve as important venues. In previous studies, seemingly unrelated mysteries still exist -- (1) models with appropriate positional embeddings can correctly perform longer unseen arithmetic operations such as addition, but their effectiveness varies in more complex tasks like multiplication; (2) models perform well for longer unseen cases in modular addition under specific moduli (e.g., modulo 100) but struggle under very close moduli (e.g., modulo 101), regardless of the positional encoding used. We believe previous studies have been treating the symptoms rather than addressing the root cause -- they have paid excessive attention to improving model components, while overlooking the differences in task properties that may be the real drivers. This is confirmed by our unified theoretical framework for different arithmetic scenarios. For example, unlike multiplication, the digital addition task has the property of translation invariance which naturally aligns with the relative positional encoding, and this combination leads to successful generalization of addition to unseen longer domains. The discrepancy in operations modulo 100 and 101 arises from the base. Modulo 100, unlike 101, is compatible with the decimal system (base 10), such that unseen information in digits beyond the units digit and the tens digit is actually not needed for the task. Extensive experiments with GPT-like models validate our theoretical predictions. These findings deepen our understanding of the generalization mechanisms, and facilitate more data-efficient model training and objective-oriented AI alignment.

Improving Arithmetic Reasoning Ability of Large Language Models through Relation Tuples, Verification and Dynamic Feedback

Teaching-Inspired Integrated Prompting Framework: A Novel Approach for Enhancing Reasoning in Large Language Models

Improving LLM Reasoning through Scaling Inference Computation with Collaborative Verification

Multi-tool Integration Application for Math Reasoning Using Large Language Model

When Do Program-of-Thought Works for Reasoning?

Stepwise Verification and Remediation of Student Reasoning Errors with Large Language Model Tutors

How well do Large Language Models perform in Arithmetic tasks?

Unraveling Arithmetic in Large Language Models: The Role of Algebraic Structures

GraphReason: Enhancing Reasoning Capabilities of Large Language Models through A Graph-Based Verification Approach

Interpreting and Improving Large Language Models in Arithmetic Calculation

Enhancing Zero-Shot Chain-of-Thought Reasoning in Large Language Models through Logic

Meaningful Learning: Enhancing Abstract Reasoning in Large Language Models via Generic Fact Guidance

MARIO: MAth Reasoning with code Interpreter Output -- A Reproducible Pipeline

Logic Contrastive Reasoning with Lightweight Large Language Model for Math Word Problems

MAF: Multi-Aspect Feedback for Improving Reasoning in Large Language Models

Self-training Language Models for Arithmetic Reasoning

Meaningful Learning: Advancing Abstract Reasoning in Large Language Models Via Generic Fact Guidance.

Relating the Seemingly Unrelated: Principled Understanding of Generalization for Generative Models in Arithmetic Reasoning Tasks

Evaluating Mathematical Reasoning of Large Language Models: A Focus on Error Identification and Correction

Learning Multi-Step Reasoning by Solving Arithmetic Tasks