Abstract:Large language models (LLMs) achieve good performance on challenging reasoning benchmarks, yet could also make basic reasoning mistakes. This contrasting behavior is puzzling when it comes to understanding the mechanisms behind LLMs' reasoning capabilities. One hypothesis is that the increasingly high and nearly saturated performance on common reasoning benchmarks could be due to the memorization of similar problems. In this paper, we systematically investigate this hypothesis with a quantitative measurement of memorization in reasoning tasks, using a dynamically generated logical reasoning benchmark based on Knights and Knaves (K&K) puzzles. We found that LLMs could interpolate the training puzzles (achieving near-perfect accuracy) after fine-tuning, yet fail when those puzzles are slightly perturbed, suggesting that the models heavily rely on memorization to solve those training puzzles. On the other hand, we show that while fine-tuning leads to heavy memorization, it also consistently improves generalization performance. In-depth analyses with perturbation tests, cross difficulty-level transferability, probing model internals, and fine-tuning with wrong answers suggest that the LLMs learn to reason on K&K puzzles despite training data memorization. This phenomenon indicates that LLMs exhibit a complex interplay between memorization and genuine reasoning abilities. Finally, our analysis with per-sample memorization score sheds light on how LLMs switch between reasoning and memorization in solving logical puzzles. Our code and data are available at <a class="link-external link-https" href="https://memkklogic.github.io" rel="external noopener nofollow">this https URL</a>.

The Buffer Mechanism for Multi-Step Information Reasoning in Language Models

Buffer of Thoughts: Thought-Augmented Reasoning with Large Language Models

Towards a Mechanistic Interpretation of Multi-Step Reasoning Capabilities of Language Models

INFORM : Information Entropy Based Multi-Step Reasoning FOR Large Language Models

A Mechanistic Analysis of a Transformer Trained on a Symbolic Multi-Step Reasoning Task

Optimizing Chain-of-Thought Reasoning: Tackling Arranging Bottleneck via Plan Augmentation

Patience Is The Key to Large Language Model Reasoning

Conceptual and Unbiased Reasoning in Language Models

Large Language Models Still Face Challenges in Multi-Hop Reasoning with External Knowledge

On Memorization of Large Language Models in Logical Reasoning

Learning Multi-Step Reasoning by Solving Arithmetic Tasks

Disentangling Memory and Reasoning Ability in Large Language Models

Chain-of-Thought Hub: A Continuous Effort to Measure Large Language Models' Reasoning Performance

Uncovering the Interpretation of Large Language Models

Understanding Hidden Computations in Chain-of-Thought Reasoning

Meta-Reasoning: Semantics-Symbol Deconstruction for Large Language Models

Language Models Are Greedy Reasoners: A Systematic Formal Analysis of Chain-of-Thought

Break the Chain: Large Language Models Can be Shortcut Reasoners

From Complex to Simple: Unraveling the Cognitive Tree for Reasoning with Small Language Models

Parrot Mind: Towards Explaining the Complex Task Reasoning of Pretrained Large Language Models with Template-Content Structure

Multi-tool Integration Application for Math Reasoning Using Large Language Model