Abstract:In neuroscience, all kinds of computation models were designed to answer the open question of how sensory stimuli are encoded by neurons and conversely, how sensory stimuli can be decoded from neuronal activities. Especially, functional Magnetic Resonance Imaging (fMRI) studies have made many great achievements with the rapid development of deep network computation. However, comparing with the goal of decoding orientation, position and object category from human fMRI in visual cortex, accurate reconstruction of image stimuli is a still challenging work. Current prevailing methods were composed of two independent steps, (1) decoding intermediate features from human fMRI and (2) reconstruction using the decoded intermediate features. The new concept of 'capsule' and 'capsule' based neural network were proposed recently. The 'capsule' represented a kind of structure containing a group of neurons to perform better feature representation. Especially, the high-level capsule's features in the capsule network (CapsNet) contains various features of image stimuli such as semantic class, orientation, location, scale and so on, and these features can better represent the processed information inherited in the fMRI data collected in visual cortex. In this paper, a novel CapsNet architecture based visual reconstruction (CNAVR) computation model is developed to reconstruct image stimuli from human fMRI. The CNAVR is composed of linear encoding computation from capsule's features to fMRI data and inverse reconstruction computation. In the first part, we trained the CapsNet model to obtain the non-linear mappings from images to high-level capsule's features, and from high-level capsule's features to images again in an end-to-end manner. In the second part, we trained the non-linear mapping from fMRI data of selected voxels to high-level capsule's features. For a new image stimulus, we can use the method to predict the corresponding high-level capsule's features using fMRI data, and reconstruct image stimuli with the trained reconstruction part in the CapsNet. We evaluated the proposed CNAVR method on the open dataset of handwritten digital images, and exceeded about 10% than the accuracy of all existing state-of-the-art methods on the structural similarity index (SSIM). In addition, we explained the selected voxels in specific interpretable image features to prove the effectivity and generalization of the CNAVR method.

Reconstruction of 3D Images from Human Activity by a Compound Reconstruction Model

3D Contrast Image Reconstruction From Human Brain Activity

Rethinking Visual Reconstruction: Experience-Based Content Completion Guided by Visual Cues

Reconstruction of Natural Images from Human fMRI Using a Three-Stage Multi-Level Deep Fusion Model

Alleviating the Semantic Gap for Generalized Fmri-to-image Reconstruction.

Visual Image Reconstruction from fMRI Activation Using Multi-scale Support Vector Machine Decoders.

Deep image reconstruction from human brain activity

Reconstructing Natural Images from Human Fmri by Alternating Encoding and Decoding with Shared Autoencoder Regularization

Pre-training Deep Neural Network for Decoding Multiple Brain Regions Can Enhance Image Reconstruction

A novel DRL-guided sparse voxel decoding model for reconstructing perceived images from brain activity

F-score Feature Selection Based Bayesian Reconstruction of Visual Image from Human Brain Activity

Visual Image Reconstructed Without Semantics from Human Brain Activity Using Linear Image Decoders and Nonlinear Noise Suppression

Reconstruction of natural images from evoked brain activity with a dictionary-based invertible encoding procedure

Reconstructing Visual Stimulus Images from EEG Signals Based on Deep Visual Representation Model

Foreground-attention in Neural Decoding: Guiding Loop-Enc-Dec to Reconstruct Visual Stimulus Images from Fmri

Reconstructing seen images from human brain activity via guided stochastic search

Deep Natural Image Reconstruction from Human Brain Activity Based on Conditional Progressively Growing Generative Adversarial Networks

Reconstructing seen image from brain activity by visually-guided cognitive representation and adversarial learning

Accurate Reconstruction of Image Stimuli From Human Functional Magnetic Resonance Imaging Based on the Decoding Model With Capsule Network Architecture

Decoding Visual Experience and Mapping Semantics through Whole-Brain Analysis Using fMRI Foundation Models

MindLDM: Reconstruct Visual Stimuli from Fmri Using Latent Diffusion Model