Abstract:Memory is the foundation of all human activities; without memory, it would be nearly impossible for people to perform any task in daily life. With the development of Large Language Models (LLMs), their language capabilities are becoming increasingly comparable to those of humans. But do LLMs have memory? Based on current performance, LLMs do appear to exhibit memory. So, what is the underlying mechanism of this memory? Previous research has lacked a deep exploration of LLMs' memory capabilities and the underlying theory. In this paper, we use Universal Approximation Theorem (UAT) to explain the memory mechanism in LLMs. We also conduct experiments to verify the memory capabilities of various LLMs, proposing a new method to assess their abilities based on these memory ability. We argue that LLM memory operates like Schrödinger's memory, meaning that it only becomes observable when a specific memory is queried. We can only determine if the model retains a memory based on its output in response to the query; otherwise, it remains indeterminate. Finally, we expand on this concept by comparing the memory capabilities of the human brain and LLMs, highlighting the similarities and differences in their operational mechanisms.

What problem does this paper attempt to address?

### Problems the Paper Attempts to Solve This paper aims to explore whether large language models (LLMs) possess memory functions and to delve into their memory mechanisms. Specifically, the paper attempts to answer the following core questions: 1. **Do LLMs have memory capabilities?** - Current research indicates that LLMs do exhibit some form of memory capability. However, how this memory is realized and how it differs from human memory are questions that remain inadequately answered. 2. **What is the memory mechanism of LLMs?** - The paper uses the Universal Approximation Theorem (UAT) to explain the memory mechanism of LLMs. The authors suggest that the memory of LLMs is similar to Schrödinger's memory, which becomes observable only when a specific memory is queried. Without a query, this memory state remains indeterminate. 3. **How to evaluate the memory capability of LLMs?** - The paper proposes a new method to evaluate the memory capability of LLMs and validates the memory performance of different LLMs through experiments. These experiments include memory tests of Chinese and English poetry to assess the accuracy of the models' memory under different conditions. 4. **What are the similarities and differences between the memory of LLMs and human memory?** - By comparing the memory mechanisms of LLMs and the human brain, the paper explores the functional similarities and differences between the two. The authors believe that both LLMs and the human brain can be understood as processes that dynamically generate outputs based on inputs. ### Main Contributions of the Paper 1. **Explanation of LLMs' memory capability**: - Using the UAT theory, the paper explains how LLMs dynamically generate memory based on inputs. 2. **Proposing a new evaluation method**: - The paper proposes an objective method to evaluate the memory capability of LLMs, measuring the overall performance of the model through memory capability assessment. 3. **Comparison of LLMs and human memory**: - The paper logically compares the memory and reasoning capabilities of LLMs and the human brain from a functional perspective, highlighting their similarities in dynamically generating outputs. ### Experimental Results - **Chinese Poetry Memory Test**: - On the CN Poems dataset, Qwen2-1.5B-Instruct and bloom-1b4-zh performed the best, remembering 1,938 and 1,932 poems respectively. - **English Poetry Memory Test**: - On the ENG Poems dataset, almost all models were able to remember all the poems, showing excellent performance. - **Impact of Output Length**: - The experimental results show that as the length of the output text increases, the difficulty of the model's memory content also increases. ### Conclusion Through theoretical analysis and experimental validation, the paper demonstrates that LLMs possess memory capabilities, and this memory mechanism can be explained through the UAT theory. Additionally, the paper proposes an objective evaluation method to measure the memory capability of LLMs and further explores the similarities and differences between the memory mechanisms of LLMs and the human brain.

Schrodinger's Memory: Large Language Models

Aspects of human memory and Large Language Models

Memory, Consciousness and Large Language Model

$\text{Memory}^3$: Language Modeling with Explicit Memory

Empowering Working Memory for Large Language Model Agents

A Survey on the Memory Mechanism of Large Language Model based Agents

MEMORYLLM: Towards Self-Updatable Large Language Models

RET-LLM: Towards a General Read-Write Memory for Large Language Models

Disentangling Memory and Reasoning Ability in Large Language Models

RecallM: An Adaptable Memory Mechanism with Temporal Understanding for Large Language Models

Understanding Memorisation in LLMs: Dynamics, Influencing Factors, and Implications

Large Language Models Are Semi-Parametric Reinforcement Learning Agents

Augmenting Language Models with Long-Term Memory

Enhancing Large Language Model with Self-Controlled Memory Framework

Needle in the Haystack for Memory Based Large Language Models

Memory Augmented Large Language Models are Computationally Universal

Think-in-Memory: Recalling and Post-thinking Enable LLMs with Long-Term Memory

Detecting Memorization in Large Language Models