Abstract:Large language models (LLMs) have demonstrated the world with the sparks of artificial general intelligence (AGI). One opinion, especially from some startups working on LLMs, argues that an LLM with nearly unlimited context length can realize AGI. However, they might be too optimistic about the long-context capability of (existing) LLMs -- (1) Recent literature has shown that their effective context length is significantly smaller than their claimed context length; and (2) Our reasoning-in-a-haystack experiments further demonstrate that simultaneously finding the relevant information from a long context and conducting (simple) reasoning is nearly impossible. In this paper, we envision a pathway from LLMs to AGI through the integration of \emph{memory}. We believe that AGI should be a system where LLMs serve as core processors. In addition to raw data, the memory in this system would store a large number of important conclusions derived from reasoning processes. Compared with retrieval-augmented generation (RAG) that merely processing raw data, this approach not only connects semantically related information closer, but also simplifies complex inferences at the time of querying. As an intermediate stage, the memory will likely be in the form of natural language descriptions, which can be directly consumed by users too. Ultimately, every agent/person should have its own large personal model, a deep neural network model (thus \emph{AI-native}) that parameterizes and compresses all types of memory, even the ones cannot be described by natural languages. Finally, we discuss the significant potential of AI-native memory as the transformative infrastructure for (proactive) engagement, personalization, distribution, and social in the AGI era, as well as the incurred privacy and security challenges with preliminary solutions.

Memory and attention in deep learning

How the Brain Formulates Memory: A Spatio-Temporal Model Research Frontier.

Attention-Augmented Machine Memory

Memory Networks: Towards Fully Biologically Plausible Learning

Memoria: Resolving Fateful Forgetting Problem through Human-Inspired Memory Architecture

Ordered Memory.

Improving Image Similarity Learning by Adding External Memory

Survey on Memory-Augmented Neural Networks: Cognitive Insights to AI Applications

Multigrid Neural Memory

Stable Hadamard Memory: Revitalizing Memory-Augmented Agents for Reinforcement Learning

Learning to Generate with Memory

Structured Memory for Neural Turing Machines

One-shot Learning with Memory-Augmented Neural Networks

Memory-Augmented Theory of Mind Network

Concept learning through deep reinforcement learning with memory-augmented neural networks

Schrodinger's Memory: Large Language Models

Schematic Memory Persistence and Transience for Efficient and Robust Continual Learning

Memory Management in Deep Learning: a Survey

AI-native Memory: A Pathway from LLMs Towards AGI

Memorization in deep learning: A survey