Abstract:In this study, we propose an axiomatic system to define and quantify the precise memorization and in-context reasoning effects used by the large language model (LLM) for language generation. These effects are formulated as non-linear interactions between tokens/words encoded by the LLM. Specifically, the axiomatic system enables us to categorize the memorization effects into foundational memorization effects and chaotic memorization effects, and further classify in-context reasoning effects into enhanced inference patterns, eliminated inference patterns, and reversed inference patterns. Besides, the decomposed effects satisfy the sparsity property and the universal matching property, which mathematically guarantee that the LLM's confidence score can be faithfully decomposed into the memorization effects and in-context reasoning effects. Experiments show that the clear disentanglement of memorization effects and in-context reasoning effects enables a straightforward examination of detailed inference patterns encoded by LLMs.

What problem does this paper attempt to address?

### Problems the Paper Attempts to Solve This paper aims to define and quantify the precise memory effects and contextual reasoning effects used by large language models (LLMs) during the language generation process. Specifically, the authors propose an axiomatic system to classify memory effects (including basic memory effects and chaotic memory effects) and contextual reasoning effects (including enhanced reasoning patterns, eliminated reasoning patterns, and reversed reasoning patterns). Through this classification, researchers can analyze and interpret the specific reasoning patterns that LLMs rely on when generating language in detail. ### Summary of Main Content 1. **Introduction**: - Research Background: For a long time, there has been a discussion on whether large language models infer based on memory knowledge and reasoning logic. - Existing Research: Many empirical studies suggest that the reasoning ability of LLMs can be tested through specific tasks or their memory ability through common-sense questions. - Research Motivation: There is currently no clear method to mathematically define and quantify the memory effects and reasoning effects of LLMs, as there is no clear boundary between memory and reasoning. 2. **Methods**: - **Axiomatic System**: An axiomatic system is proposed to define and quantify memory effects and reasoning effects. - **Decomposition Method**: The confidence scores of LLMs are decomposed into basic memory effects, chaotic memory effects, and contextual reasoning effects. - **Specific Classification**: - **Basic Memory Effects**: When the input prompt contains only the question, the interaction effects encoded by the LLM can be regarded as memory effects. - **Contextual Reasoning Effects**: When premises are added, the changes in interaction effects are defined as contextual reasoning effects, further divided into enhanced reasoning patterns, eliminated reasoning patterns, and reversed reasoning patterns. - **Chaotic Memory Effects**: In addition to contextual reasoning effects, adding premises introduces additional effects unrelated to reasoning, referred to as chaotic memory effects. 3. **Experimental Results**: - **Validation of Decomposition Effects**: Experiments show that the clear separation of memory effects and contextual reasoning effects allows for an intuitive examination of the detailed reasoning patterns encoded by LLMs. - **Specific Findings**: Although the final output of LLMs appears reasonable, they actually use many incorrect interactions for reasoning. - **Comparison of Different Models**: The proportions of contextual reasoning effects and chaotic memory effects were tested on three LLMs (OPT-1.3B, LLaMA-7B, and GPT-3.5-Turbo), finding that all models encoded significant contextual reasoning effects, while chaotic memory effects were weaker. 4. **Theoretical Guarantees**: - **Sparsity Property**: The decomposed effects still satisfy the sparsity property, meaning only a few significant interaction effects exist. - **Universal Matching Property**: The decomposed effects can faithfully explain the output changes of LLMs on different masked samples. 5. **Visualization Analysis**: - **Distribution of Different Orders**: By visualizing the memory effects and reasoning effects of different orders, the reasoning patterns of LLMs under different complexities are demonstrated. - **Analysis of Specific Samples**: The memory effects and reasoning effects on individual samples are shown, further validating the effectiveness of the method. ### Conclusion This study successfully defines and quantifies the memory effects and contextual reasoning effects of large language models during the language generation process by proposing an axiomatic system. This not only provides a new perspective for understanding the internal mechanisms of LLMs but also offers a theoretical foundation for further optimization and improvement of LLMs.

Quantifying In-Context Reasoning Effects and Memorization Effects in LLMs

Concise and Organized Perception Facilitates Large Language Models for Deductive Reasoning.

On Memorization of Large Language Models in Logical Reasoning

Disentangling Memory and Reasoning Ability in Large Language Models

Understanding Memorisation in LLMs: Dynamics, Influencing Factors, and Implications

A Multi-Perspective Analysis of Memorization in Large Language Models

Large Language Models Are In-Context Semantic Reasoners Rather Than Symbolic Reasoners

Summing Up the Facts: Additive Mechanisms Behind Factual Recall in LLMs

Understanding and Patching Compositional Reasoning in LLMs

Schrodinger's Memory: Large Language Models

Unveiling Factual Recall Behaviors of Large Language Models through Knowledge Neurons

Benchmarking Chinese Commonsense Reasoning of LLMs: From Chinese-Specifics to Reasoning-Memorization Correlations

Rethinking LLM Memorization through the Lens of Adversarial Compression

LLMs for Relational Reasoning: How Far are We?

SoK: Memorization in General-Purpose Large Language Models

MemLLM: Finetuning LLMs to Use An Explicit Read-Write Memory

Concise and Organized Perception Facilitates Reasoning in Large Language Models

An evaluation on large language model outputs: Discourse and memorization

CLR-Fact: Evaluating the Complex Logical Reasoning Capability of Large Language Models over Factual Knowledge

Reasoning or a Semblance of it? A Diagnostic Study of Transitive Reasoning in LLMs

An Investigation of Neuron Activation as a Unified Lens to Explain Chain-of-Thought Eliciting Arithmetic Reasoning of LLMs