Abstract:When we perceive a scene, our brain processes various types of visual information simultaneously, ranging from sensory features, such as line orientations and colors, to categorical features, such as objects and their arrangements. Whereas the role of sensory and categorical visual representations in predicting subsequent memory has been studied using isolated objects, their impact on memory for complex scenes remains largely unknown. To address this gap, we conducted an fMRI study in which female and male participants encoded pictures of familiar scenes (e.g., an airport picture) and later recalled them, while rating the vividness of their visual recall. Outside the scanner, participants had to distinguish each seen scene (e.g., an airport picture) from three similar lures (e.g., 3 airport pictures). We modeled the sensory and categorical visual features of multiple scenes using both early and late layers of a deep convolutional neural network. Then, we applied representational similarity analysis to determine which brain regions represented stimuli in accordance with the sensory and categorical models. We found that categorical, but not sensory, representations predicted subsequent memory. In line with the previous result, only for the categorical model, the average recognition performance of each scene exhibited a positive correlation with the average visual dissimilarity between the item in question and its respective lures. These results strongly suggest that even in memory tests that ostensibly rely solely on visual cues (such as force-choice visual recognition with similar distractors), memory decisions for scenes may be primarily influenced by categorical rather than sensory representations. Significance Statement Our memory for real-world scenes often comprises a tableau of complex visual features, but recent findings challenge the view that our memories of such stimuli rely on purely visual information. Instead, it appears that our memory for scenes is heavily influenced by higher-level categorical information. Analyzing cortical representations in regions responsive to both categorical and sensory features, we discovered that only the former can reliably predict memory outcomes. Moreover, the distinctiveness of scenes in terms of their categoric features among similar examples is positively associated with our ability to accurately recognize previously encountered scenes. In essence, this study sheds light on how our brains rely on categorical information to recognize natural scenes.

Perceptual comparisons induce lasting and generalizing changes to face memory reports

Visible Persistence Speeds Up Visual Search

The Neural Mechanisms of Percept–memory Comparison in Visual Working Memory

Perceptual encoding benefit of visual memorability on visual memory formation

Prior familiarity enhances recognition memory of faces, not just images of faces, when accompanied by conceptual information

Memory retrieval effects as a function of differences in phenomenal experience

Coarse-to-fine Encoding of Spatial Frequency Information into Visual Short-Term Memory for Faces but Impartial Decay.

Memory matching features bias the ensemble perception of facial identity

Greater Visual Working Memory Capacity for Visually Matched Stimuli When They Are Perceived as Meaningful

"Memory compression" effects in visual working memory are contingent on explicit long-term memory

Adaptive Memory Distortions Are Predicted by Feature Representations in Parietal Cortex

It is not in the details: Self-related shapes are rapidly classified but their features are not better remembered

Memory reports are biased by all relevant contents of working memory

Better face-name recall is associated with better face recognition ability

The Effects of Similarity on High-Level Visual Working Memory Processing

Adaptively triggered comparisons enhance perceptual category learning: evidence from face learning

Perceptual memory for highly familiar people's body shape: manipulation of images of the self and friend

Assessing the interaction between working memory and perception through time

Visual recognition memory of scenes is driven by categorical, not sensory, visual representations.

Subsequent Memory Effects in Cortical Pattern Similarity Differ by Semantic Class

The Effects of Categorical Similarity and Feature-Space Proximity on Visual Working Memory Processing