Abstract:Large language models (LLMs) have demonstrated impressive versatility across numerous tasks, yet their generalization capabilities remain poorly understood. To investigate these behaviors, arithmetic tasks serve as important venues. In previous studies, seemingly unrelated mysteries still exist -- (1) models with appropriate positional embeddings can correctly perform longer unseen arithmetic operations such as addition, but their effectiveness varies in more complex tasks like multiplication; (2) models perform well for longer unseen cases in modular addition under specific moduli (e.g., modulo 100) but struggle under very close moduli (e.g., modulo 101), regardless of the positional encoding used. We believe previous studies have been treating the symptoms rather than addressing the root cause -- they have paid excessive attention to improving model components, while overlooking the differences in task properties that may be the real drivers. This is confirmed by our unified theoretical framework for different arithmetic scenarios. For example, unlike multiplication, the digital addition task has the property of translation invariance which naturally aligns with the relative positional encoding, and this combination leads to successful generalization of addition to unseen longer domains. The discrepancy in operations modulo 100 and 101 arises from the base. Modulo 100, unlike 101, is compatible with the decimal system (base 10), such that unseen information in digits beyond the units digit and the tens digit is actually not needed for the task. Extensive experiments with GPT-like models validate our theoretical predictions. These findings deepen our understanding of the generalization mechanisms, and facilitate more data-efficient model training and objective-oriented AI alignment.

A New Uniform Or Operation Model Based On Generalized S-Norm

New Universal Logic Operation Models And Neurons On Generalized Interval

M-norm useful in artificial intelligence

Mathematical Modeling in Construction of Logical Carrying Networks

A New Neural Network Model And Its Application

Robust Optimization of Uncertain Structures Based on Normalized Violation Degree of Interval Constraint

A Universal Logic Operator for Interpretable Deep Convolution Networks

Supermodular Approximation of Norms and Applications

Modularity conditions between overlap (grouping) function and uni-nullnorm or null-uninorm

On Operator Expressions of Monotone Boolean Networks

On Uniform Functions on Configuration Spaces of Large Scale Interacting Systems

From Generalization Analysis to Optimization Designs for State Space Models

Modularity characterization on general 2-uninorms and overlap or grouping functions

Submodular Norms with Applications To Online Facility Location and Stochastic Probing

Numerically Probing the Universal Operator Growth Hypothesis

Uncertain Logic

Relating the Seemingly Unrelated: Principled Understanding of Generalization for Generative Models in Arithmetic Reasoning Tasks

Uniformity_and_nonuniformity

A Universal Machine for Biform Theory Graphs

Generalized convergence in measure theorems of Sugeno integrals

Genericity and Universality for Operator Ideals