Abstract:When we perceive a scene, our brain processes various types of visual information simultaneously, ranging from sensory features, such as line orientations and colors, to categorical features, such as objects and their arrangements. Whereas the role of sensory and categorical visual representations in predicting subsequent memory has been studied using isolated objects, their impact on memory for complex scenes remains largely unknown. To address this gap, we conducted an fMRI study in which female and male participants encoded pictures of familiar scenes (e.g., an airport picture) and later recalled them, while rating the vividness of their visual recall. Outside the scanner, participants had to distinguish each seen scene (e.g., an airport picture) from three similar lures (e.g., 3 airport pictures). We modeled the sensory and categorical visual features of multiple scenes using both early and late layers of a deep convolutional neural network. Then, we applied representational similarity analysis to determine which brain regions represented stimuli in accordance with the sensory and categorical models. We found that categorical, but not sensory, representations predicted subsequent memory. In line with the previous result, only for the categorical model, the average recognition performance of each scene exhibited a positive correlation with the average visual dissimilarity between the item in question and its respective lures. These results strongly suggest that even in memory tests that ostensibly rely solely on visual cues (such as force-choice visual recognition with similar distractors), memory decisions for scenes may be primarily influenced by categorical rather than sensory representations. Significance Statement Our memory for real-world scenes often comprises a tableau of complex visual features, but recent findings challenge the view that our memories of such stimuli rely on purely visual information. Instead, it appears that our memory for scenes is heavily influenced by higher-level categorical information. Analyzing cortical representations in regions responsive to both categorical and sensory features, we discovered that only the former can reliably predict memory outcomes. Moreover, the distinctiveness of scenes in terms of their categoric features among similar examples is positively associated with our ability to accurately recognize previously encountered scenes. In essence, this study sheds light on how our brains rely on categorical information to recognize natural scenes.

Brain mechanisms underlying cue-based memorizing during free viewing of movie Memento

Watching Movies Unfold, a Frame-by-Frame Analysis of the Associated Neural Dynamics

Hippocampal-Medial Prefrontal Event Segmentation and Integration Contribute to Episodic Memory Formation

Rapid Memory Reactivation at Movie Event Boundaries Promotes Episodic Encoding

They saw a movie: Long-term memory for an extended audiovisual narrative

Neural correlates of episodic memory in the Memento cohort

Long-term memory for movie details: selective decay for verbal information at one week

What Happened When? Cerebral Processing of Modified Structure and Content in Episodic Cueing

Mapping patterns of thought onto brain activity during movie-watching

Mega-scale movie-fields in the mouse visuo-hippocampal network

A core set of neural states underlying memory reactivation of naturalistic events in the posterior medial cortex

Effective connectivity of the anterior hippocampus predicts recollection confidence during natural memory retrieval

Reconstructive nature of temporal memory for movie scenes

Visual continuous recognition reveals behavioral and neural differences for short- and long-term scene memory

Temporal dynamics of competition between statistical learning and episodic memory in intracranial recordings of human visual cortex

Visual recognition memory of scenes is driven by categorical, not sensory, visual representations.

A shared neural code for perceiving and remembering social interactions in the human superior temporal sulcus

Motifs of human hippocampal and cortical high frequency oscillations structure processing and memory of naturalistic stimuli

Correlation between Inter-Blink Interval and Episodic Encoding during Movie Watching

Persistent schema-dependent hippocampal-neocortical connectivity during memory encoding and postencoding rest in humans

Movie viewing elicits rich and reliable brain state dynamics