Abstract:Understanding the hidden mechanisms behind human's visual perception is a fundamental question in neuroscience. To that end, investigating into the neural responses of human mind activities, such as functional Magnetic Resonance Imaging (fMRI), has been a significant research vehicle. However, analyzing fMRI signals is challenging, costly, daunting, and demanding for professional training. Despite remarkable progress in fMRI analysis, existing approaches are limited to generating 2D images and far away from being biologically meaningful and practically useful. Under this insight, we propose to generate visually plausible and functionally more comprehensive 3D outputs decoded from brain signals, enabling more sophisticated modeling of fMRI data. Conceptually, we reformulate this task as a {\em fMRI conditioned 3D object generation} problem. We design a novel 3D object representation learning method, Brain3D, that takes as input the fMRI data of a subject who was presented with a 2D image, and yields as output the corresponding 3D object images. The key capabilities of this model include tackling the noises with high-level semantic signals and a two-stage architecture design for progressive high-level information integration. Extensive experiments validate the superior capability of our model over previous state-of-the-art 3D object generation methods. Importantly, we show that our model captures the distinct functionalities of each region of human vision system as well as their intricate interplay relationships, aligning remarkably with the established discoveries in neuroscience. Further, preliminary evaluations indicate that Brain3D can successfully identify the disordered brain regions in simulated scenarios, such as V1, V2, V3, V4, and the medial temporal lobe (MTL) within the human visual system. Our data and code will be available at <a class="link-external link-https" href="https://brain-3d.github.io/" rel="external noopener nofollow">this https URL</a>.

Multi-pooling 3D Convolutional Neural Network for fMRI Classification of Visual Brain States

Multi-view SoftPool Attention Convolutional Networks for 3D Model Classification.

A Multichannel 2D Convolutional Neural Network Model for Task-Evoked fMRI Data Classification

3D Convolution neural network with multiscale spatial and temporal cues for motor imagery EEG classification

A Densely Connected Multi-Branch 3D Convolutional Neural Network for Motor Imagery EEG Decoding

3D Convolutional Neural Networks for Classification of Functional Connectomes

A Multi-Branch 3D Convolutional Neural Network for EEG-Based Motor Imagery Classification.

Multi-Center Brain Imaging Classification Using a Novel 3D CNN Approach.

A spatial-frequency-temporal 3D convolutional neural network for motor imagery EEG signal classification

Three‐class classification of brain magnetic resonance images using average‐pooling convolutional neural network

Automatic Recognition of fMRI-derived Functional Networks using 3D Convolutional Neural Networks.

Decoding Categories from Human Brain Activity in the Human Visual Cortex Using the Triplet Network

Temporal-frequency-phase feature classification using 3D-convolutional neural networks for motor imagery and movement

Brain Decoding Using fMRI Images for Multiple Subjects through Deep Learning

Alzheimer's Disease Classification Based on Combination of Multi-Model Convolutional Networks

Decoding Visual fMRI Stimuli from Human Brain Based on Graph Convolutional Neural Network

A parallel multi-scale time-frequency block convolutional neural network based on channel attention module for motor imagery classification

Reconstruction of Natural Images from Human fMRI Using a Three-Stage Multi-Level Deep Fusion Model

MI3DNet: A Compact CNN for Motor Imagery EEG Classification with Visualizable Dense Layer Parameters.

Decoding visual stimuli in human brain by using Anatomical Pattern Analysis on fMRI images

Brain3D: Generating 3D Objects from fMRI