Brain Diffusion for Visual Exploration: Cortical Discovery using Large Scale Generative Models

Andrew F. Luo,Margaret M. Henderson,Leila Wehbe,Michael J. Tarr
2023-11-29
Abstract:A long standing goal in neuroscience has been to elucidate the functional organization of the brain. Within higher visual cortex, functional accounts have remained relatively coarse, focusing on regions of interest (ROIs) and taking the form of selectivity for broad categories such as faces, places, bodies, food, or words. Because the identification of such ROIs has typically relied on manually assembled stimulus sets consisting of isolated objects in non-ecological contexts, exploring functional organization without robust a priori hypotheses has been challenging. To overcome these limitations, we introduce a data-driven approach in which we synthesize images predicted to activate a given brain region using paired natural images and fMRI recordings, bypassing the need for category-specific stimuli. Our approach -- Brain Diffusion for Visual Exploration ("BrainDiVE") -- builds on recent generative methods by combining large-scale diffusion models with brain-guided image synthesis. Validating our method, we demonstrate the ability to synthesize preferred images with appropriate semantic specificity for well-characterized category-selective ROIs. We then show that BrainDiVE can characterize differences between ROIs selective for the same high-level category. Finally we identify novel functional subdivisions within these ROIs, validated with behavioral data. These results advance our understanding of the fine-grained functional organization of human visual cortex, and provide well-specified constraints for further examination of cortical organization using hypothesis-driven methods.
Computer Vision and Pattern Recognition
What problem does this paper attempt to address?
### Problems the Paper Aims to Solve This paper aims to address the issue of functional organization in neuroscience through a method called "Brain Diffusion for Visual Exploration" (BrainDiVE). Specifically: 1. **Functional Organization Research**: - Traditional research methods rely on manually selected stimulus sets by researchers, which often include isolated objects rather than ecological scenes, leading to a lack of robust prior assumptions when exploring brain functional organization. - To overcome this limitation, the authors propose a data-driven approach that uses natural images and functional magnetic resonance imaging (fMRI) recordings to synthesize images that activate specific brain regions, thereby avoiding the need for specific category stimuli. 2. **Generating High Semantic Specificity Images**: - By combining large-scale diffusion models with brain-guided image synthesis techniques, BrainDiVE can generate images with appropriate semantic specificity that can activate known functional regions. - Experiments demonstrate that BrainDiVE can synthesize images for selective regions targeting specific high-level categories (such as faces, places, bodies, text, etc.) and can reveal differences between these regions. 3. **Identifying Fine-Grained Functional Subdivisions**: - BrainDiVE can also identify new functional subregions within existing functional areas and validate these results through behavioral data, further advancing the understanding of fine-grained functional organization in the human visual cortex. In summary, by proposing the BrainDiVE method, this paper aims to utilize generative models and brain signals to synthesize images that activate specific brain regions, thereby providing new tools and techniques for neuroscience research.