Abstract:When we perceive a scene, our brain processes various types of visual information simultaneously, ranging from sensory features, such as line orientations and colors, to categorical features, such as objects and their arrangements. Whereas the role of sensory and categorical visual representations in predicting subsequent memory has been studied using isolated objects, their impact on memory for complex scenes remains largely unknown. To address this gap, we conducted an fMRI study in which female and male participants encoded pictures of familiar scenes (e.g., an airport picture) and later recalled them, while rating the vividness of their visual recall. Outside the scanner, participants had to distinguish each seen scene (e.g., an airport picture) from three similar lures (e.g., 3 airport pictures). We modeled the sensory and categorical visual features of multiple scenes using both early and late layers of a deep convolutional neural network. Then, we applied representational similarity analysis to determine which brain regions represented stimuli in accordance with the sensory and categorical models. We found that categorical, but not sensory, representations predicted subsequent memory. In line with the previous result, only for the categorical model, the average recognition performance of each scene exhibited a positive correlation with the average visual dissimilarity between the item in question and its respective lures. These results strongly suggest that even in memory tests that ostensibly rely solely on visual cues (such as force-choice visual recognition with similar distractors), memory decisions for scenes may be primarily influenced by categorical rather than sensory representations. Significance Statement Our memory for real-world scenes often comprises a tableau of complex visual features, but recent findings challenge the view that our memories of such stimuli rely on purely visual information. Instead, it appears that our memory for scenes is heavily influenced by higher-level categorical information. Analyzing cortical representations in regions responsive to both categorical and sensory features, we discovered that only the former can reliably predict memory outcomes. Moreover, the distinctiveness of scenes in terms of their categoric features among similar examples is positively associated with our ability to accurately recognize previously encountered scenes. In essence, this study sheds light on how our brains rely on categorical information to recognize natural scenes.

Predicting the memorability of scene pictures: Improved accuracy through one's own experience

Differential Neural Correlates Underlie Judgment of Learning and Subsequent Memory Performance

The disconnect between metamemory and memory for emotional images

Picture-perfect is not perfect for metamemory: Testing the perceptual fluency hypothesis with degraded images.

Neural correlates of retrospective memory confidence during face–name associative learning

Visual recognition memory of scenes is driven by categorical, not sensory, visual representations.

Misleading emotions: judgments of learning overestimate recognition of negative and positive emotional images

Memory for artwork is predictable

I remember it now, so I'll remember it later: Working memory strength guides predictions for long-term memory performance

Visual continuous recognition reveals behavioral and neural differences for short- and long-term scene memory

What guides the judgment of learning: Memory or heuristics? An event-related potential study

Increased metamemory accuracy with practice does not require practice with metamemory

Image memorability influences memory for where the item was seen but not when

Highly memorable images are more readily perceived.

Recognition memory decisions made with short- and long-term retrieval

The formation of episodic autobiographical memory is predicted by mental imagery, self-reference, and anticipated details

The mismeasure of memory: When retrieval fluency is misleading as a metamnemonic index.

Item-specific and relational encoding are effective at reducing the illusion of competence

Defining Image Memorability using the Visual Memory Schema

The Resiliency of Memorability: A Predictor of Memory Separate from Attention and Priming

Does context matter for memory? Testing the effectiveness of learning by imagining situated interactions with objects