Abstract:The statistical study of human memory requires large-scale experiments, involving many stimuli conditions and test subjects. While this approach has proven to be quite fruitful for meaningless material such as random lists of words, naturalistic stimuli, like narratives, have until now resisted such a large-scale study, due to the quantity of manual labor required to design and analyze such experiments. In this work, we develop a pipeline that uses large language models (LLMs) both to design naturalistic narrative stimuli for large-scale recall and recognition memory experiments, as well as to analyze the results. We performed online memory experiments with a large number of participants and collected recognition and recall data for narratives of different sizes. We found that both recall and recognition performance scale linearly with narrative length; however, for longer narratives people tend to summarize the content rather than recalling precise details. To investigate the role of narrative comprehension in memory, we repeated these experiments using scrambled versions of the narratives. Although recall performance declined significantly, recognition remained largely unaffected. Recalls in this condition seem to follow the original narrative order rather than the actual scrambled presentation, pointing to a contextual reconstruction of the story in memory. Finally, using LLM text embeddings, we construct a simple measure for each clause based on semantic similarity to the whole narrative, that shows a strong correlation with recall probability. Overall, our work demonstrates the power of LLMs in accessing new regimes in the study of human memory, as well as suggesting novel psychologically informed benchmarks for LLM performance.

Random Tree Model of Meaningful Memory

Large-scale study of human memory for meaningful narratives

Coherent Narrative Summarization with A Cognitive Model

From Isolated Conversations to Hierarchical Schemas: Dynamic Tree Memory Representation for LLMs

`Keep it Together': Enforcing Cohesion in Extractive Summaries by Simulating Human Memory

The Goldilocks Principle: Reading Children's Books with Explicit Memory Representations

Imagined versus Remembered Stories: Quantifying Differences in Narrative Flow

Fine-Grained Modeling of Narrative Context: A Coherence Perspective via Retrospective Questions

Stories, Statistics, and Memory

Neural substrates of narrative comprehension and memory

Ordered Memory.

Memory-based Extractive Summarization

Enhancing Long-Term Memory using Hierarchical Aggregate Tree for Retrieval Augmented Generation

Beyond Memorization: The Challenge of Random Memory Access in Language Models

A Model of Semantic Completion in Generative Episodic Memory

Modeling Discourse Cohesion for Discourse Parsing Via Memory Network.

NarrativeXL: A Large-scale Dataset For Long-Term Memory Models

Quantifying the narrative flow of imagined versus autobiographical stories

Abstractive Text Summarization with Hierarchical Multi-scale Abstraction Modeling and Dynamic Memory.

Machine-learning as a validated tool to characterize individual differences in free recall of naturalistic events

Tree2Tree Learning with Memory Unit