Memory Encoding Model

Huzheng Yang,James Gee,Jianbo Shi

2023-08-02

Abstract:We explore a new class of brain encoding model by adding memory-related information as input. Memory is an essential brain mechanism that works alongside visual stimuli. During a vision-memory cognitive task, we found the non-visual brain is largely predictable using previously seen images. Our Memory Encoding Model (Mem) won the Algonauts 2023 visual brain competition even without model ensemble (single model score 66.8, ensemble score 70.8). Our ensemble model without memory input (61.4) can also stand a 3rd place. Furthermore, we observe periodic delayed brain response correlated to 6th-7th prior image, and hippocampus also showed correlated activity timed with this periodicity. We conjuncture that the periodic replay could be related to memory mechanism to enhance the working memory.

Computer Vision and Pattern Recognition

What problem does this paper attempt to address?

The paper attempts to address the question of how the brain encodes and decodes information in visual memory tasks, particularly in non-visual brain regions. Specifically, the authors propose a new brain encoding model (Memory Encoding Model, Mem) by introducing memory-related information as input to improve the prediction capability of whole-brain activity. The main contributions of the paper include:

1. **Exploring the role of memory in visual tasks**: Traditional brain encoding models mainly predict brain responses based on the current image, neglecting the influence of memory. This paper significantly improves the model's prediction performance by incorporating memory-related information such as previously seen images and behavioral responses.

2. **Discovering periodic delayed responses**: The authors observed that in both visual and non-visual brain regions, the brain exhibits periodic delayed responses to images seen 6-7 frames earlier. This periodic response aligns with the mechanisms of working memory and may help enhance working memory.

3. **Improving prediction capability in non-visual brain regions**: Traditional models perform well in predicting early visual cortex activity but poorly in non-visual brain regions. The proposed model significantly improves the prediction capability in non-visual brain regions by incorporating memory-related information.

4. **Hypothesis validation**: The authors experimentally validated their hypothesis that the hippocampus periodically reads and regenerates memory, which may computationally expand the capacity of working memory.

In summary, by introducing memory-related inputs, this paper not only improves the prediction capability of whole-brain activity but also reveals the important role of memory in visual tasks, providing a new perspective for understanding the working mechanisms of the brain.