Abstract:Understanding the hidden mechanisms behind human's visual perception is a fundamental question in neuroscience. To that end, investigating into the neural responses of human mind activities, such as functional Magnetic Resonance Imaging (fMRI), has been a significant research vehicle. However, analyzing fMRI signals is challenging, costly, daunting, and demanding for professional training. Despite remarkable progress in fMRI analysis, existing approaches are limited to generating 2D images and far away from being biologically meaningful and practically useful. Under this insight, we propose to generate visually plausible and functionally more comprehensive 3D outputs decoded from brain signals, enabling more sophisticated modeling of fMRI data. Conceptually, we reformulate this task as a {\em fMRI conditioned 3D object generation} problem. We design a novel 3D object representation learning method, Brain3D, that takes as input the fMRI data of a subject who was presented with a 2D image, and yields as output the corresponding 3D object images. The key capabilities of this model include tackling the noises with high-level semantic signals and a two-stage architecture design for progressive high-level information integration. Extensive experiments validate the superior capability of our model over previous state-of-the-art 3D object generation methods. Importantly, we show that our model captures the distinct functionalities of each region of human vision system as well as their intricate interplay relationships, aligning remarkably with the established discoveries in neuroscience. Further, preliminary evaluations indicate that Brain3D can successfully identify the disordered brain regions in simulated scenarios, such as V1, V2, V3, V4, and the medial temporal lobe (MTL) within the human visual system. Our data and code will be available at <a class="link-external link-https" href="https://brain-3d.github.io/" rel="external noopener nofollow">this https URL</a>.

Brainformer: Mimic Human Visual Brain Functions to Machine Vision Models via fMRI

Mmformer: Multimodal Medical Transformer for Incomplete Multimodal Learning of Brain Tumor Segmentation

Unidirectional brain-computer interface: Artificial neural network encoding natural images to fMRI response in the visual cortex

Embedding Human Brain Function via Transformer

Brain3D: Generating 3D Objects from fMRI

Transformer3: A Pure Transformer Framework for fMRI-Based Representations of Human Brain Function

BRACTIVE: A Brain Activation Approach to Human Visual Brain Learning

A Generic Framework for Embedding Human Brain Function with Temporally Correlated Autoencoder.

3D Brainformer: 3D Fusion Transformer for Brain Tumor Segmentation

Reconstruction of Natural Images from Human fMRI Using a Three-Stage Multi-Level Deep Fusion Model

Deep image reconstruction from human brain activity

Neuro-Vision to Language: Enhancing Brain Recording-based Visual Reconstruction and Language Interaction

Multimodal Contrastive Learning for Brain-Machine Fusion: from Brain-in-the-loop Modeling to Brain-out-of-the-loop Application

Decoding human brain activity with deep learning

NT-ViT: Neural Transcoding Vision Transformers for EEG-to-fMRI Synthesis

CvFormer: Cross-view transFormers with Pre-training for fMRI Analysis of Human Brain

Natural Image Reconstruction From fMRI Using Deep Learning: A Survey

A Visual Encoding Model Based on Deep Neural Networks and Transfer Learning for Brain Activity Measured by Functional Magnetic Resonance Imaging

NeuroCine: Decoding Vivid Video Sequences from Human Brain Activties

Quantum-Brain: Quantum-Inspired Neural Network Approach to Vision-Brain Understanding

Functional Brain-to-Brain Transformation with No Shared Data