Abstract:Understanding the hidden mechanisms behind human's visual perception is a fundamental question in neuroscience. To that end, investigating into the neural responses of human mind activities, such as functional Magnetic Resonance Imaging (fMRI), has been a significant research vehicle. However, analyzing fMRI signals is challenging, costly, daunting, and demanding for professional training. Despite remarkable progress in fMRI analysis, existing approaches are limited to generating 2D images and far away from being biologically meaningful and practically useful. Under this insight, we propose to generate visually plausible and functionally more comprehensive 3D outputs decoded from brain signals, enabling more sophisticated modeling of fMRI data. Conceptually, we reformulate this task as a {\em fMRI conditioned 3D object generation} problem. We design a novel 3D object representation learning method, Brain3D, that takes as input the fMRI data of a subject who was presented with a 2D image, and yields as output the corresponding 3D object images. The key capabilities of this model include tackling the noises with high-level semantic signals and a two-stage architecture design for progressive high-level information integration. Extensive experiments validate the superior capability of our model over previous state-of-the-art 3D object generation methods. Importantly, we show that our model captures the distinct functionalities of each region of human vision system as well as their intricate interplay relationships, aligning remarkably with the established discoveries in neuroscience. Further, preliminary evaluations indicate that Brain3D can successfully identify the disordered brain regions in simulated scenarios, such as V1, V2, V3, V4, and the medial temporal lobe (MTL) within the human visual system. Our data and code will be available at <a class="link-external link-https" href="https://brain-3d.github.io/" rel="external noopener nofollow">this https URL</a>.

The Algonauts Project 2023 Challenge: How the Human Brain Makes Sense of Natural Scenes

The Algonauts Project: A Platform for Communication between the Sciences of Biological and Artificial Intelligence

The Algonauts Project 2023 Challenge: UARK-UAlbany Team Solution

A massive 7T fMRI dataset to bridge cognitive neuroscience and artificial intelligence

Exploration and Comparison of Deep Learning Architectures to Predict Brain Response to Realistic Pictures

Brain3D: Generating 3D Objects from fMRI

Unidirectional brain-computer interface: Artificial neural network encoding natural images to fMRI response in the visual cortex

Biological Blueprints for Next Generation AI Systems

Natural scene reconstruction from fMRI signals using generative latent diffusion

Neural networks and neuroscience-inspired computer vision

Generalizability analysis of deep learning predictions of human brain responses to augmented and semantically novel visual stimuli

Empirically Identifying and Computationally Modeling the Brain-Behavior Relationship for Human Scene Categorization

Probing neural representations of scene perception in a hippocampally dependent task using artificial neural networks

The Dynamic Sensorium competition for predicting large-scale mouse visual cortex activity from videos

NeuroCine: Decoding Vivid Video Sequences from Human Brain Activties

Retrospective for the Dynamic Sensorium Competition for predicting large-scale mouse primary visual cortex activity from videos

The Neural Representation Benchmark and its Evaluation on Brain and Machine

Decoding dynamic visual scenes across the brain hierarchy

Neural Foundations of Mental Simulation: Future Prediction of Latent Representations on Dynamic Scenes

BrainFounder: Towards Brain Foundation Models for Neuroimage Analysis

Catalyzing next-generation Artificial Intelligence through NeuroAI