Brain3D: Generating 3D Objects from fMRI

Yuankun Yang,Li Zhang,Ziyang Xie,Zhiyuan Yuan,Jianfeng Feng,Xiatian Zhu,Yu-Gang Jiang

2024-08-28

Abstract:Understanding the hidden mechanisms behind human's visual perception is a fundamental question in neuroscience. To that end, investigating into the neural responses of human mind activities, such as functional Magnetic Resonance Imaging (fMRI), has been a significant research vehicle. However, analyzing fMRI signals is challenging, costly, daunting, and demanding for professional training. Despite remarkable progress in fMRI analysis, existing approaches are limited to generating 2D images and far away from being biologically meaningful and practically useful. Under this insight, we propose to generate visually plausible and functionally more comprehensive 3D outputs decoded from brain signals, enabling more sophisticated modeling of fMRI data. Conceptually, we reformulate this task as a {\em fMRI conditioned 3D object generation} problem. We design a novel 3D object representation learning method, Brain3D, that takes as input the fMRI data of a subject who was presented with a 2D image, and yields as output the corresponding 3D object images. The key capabilities of this model include tackling the noises with high-level semantic signals and a two-stage architecture design for progressive high-level information integration. Extensive experiments validate the superior capability of our model over previous state-of-the-art 3D object generation methods. Importantly, we show that our model captures the distinct functionalities of each region of human vision system as well as their intricate interplay relationships, aligning remarkably with the established discoveries in neuroscience. Further, preliminary evaluations indicate that Brain3D can successfully identify the disordered brain regions in simulated scenarios, such as V1, V2, V3, V4, and the medial temporal lobe (MTL) within the human visual system. Our data and code will be available at <a class="link-external link-https" href="https://brain-3d.github.io/" rel="external noopener nofollow">this https URL</a>.

Computer Vision and Pattern Recognition

What problem does this paper attempt to address?

### Problems the Paper Aims to Solve

This paper aims to generate three-dimensional (3D) objects from functional magnetic resonance imaging (fMRI) signals to better understand and simulate the human brain's 3D visual processing capabilities. Specifically:

1. **Overcoming Existing Technological Limitations**:
- Current methods for analyzing fMRI signals are challenging, costly, and require specialized training.
- Existing methods are mainly limited to generating two-dimensional images and lack biological significance and practicality.

2. **Proposing a New Generation Method**:
- A new method named Brain3D is proposed, which can generate visually plausible and functionally comprehensive 3D outputs from fMRI data.
- This method redefines the task as a 3D object generation problem based on fMRI conditions.

3. **Model Design and Validation**:
- A novel 3D object representation learning method is designed to handle high-noise environments with advanced semantic signals.
- The model adopts a two-stage architecture design to achieve gradual information integration.
- Extensive experiments validate the superior performance of this model in 3D object generation.

4. **Biological Consistency**:
- The model can capture the unique functions of various regions of the human visual system and their complex interactions, which are highly consistent with existing neuroscience research findings.
- Preliminary evaluations indicate that Brain3D can successfully identify damaged brain regions in simulated scenarios, such as V1, V2, V3, V4, and the medial temporal lobe (MTL).

Through these studies, the paper hopes to develop a comprehensive computational framework that combines artificial visual models with biological vision, thereby better understanding the human 3D visual processing mechanism and providing useful tools for clinical fMRI evaluation.