Abstract:In neuroscience, all kinds of computation models were designed to answer the open question of how sensory stimuli are encoded by neurons and conversely, how sensory stimuli can be decoded from neuronal activities. Especially, functional Magnetic Resonance Imaging (fMRI) studies have made many great achievements with the rapid development of deep network computation. However, comparing with the goal of decoding orientation, position and object category from human fMRI in visual cortex, accurate reconstruction of image stimuli is a still challenging work. Current prevailing methods were composed of two independent steps, (1) decoding intermediate features from human fMRI and (2) reconstruction using the decoded intermediate features. The new concept of 'capsule' and 'capsule' based neural network were proposed recently. The 'capsule' represented a kind of structure containing a group of neurons to perform better feature representation. Especially, the high-level capsule's features in the capsule network (CapsNet) contains various features of image stimuli such as semantic class, orientation, location, scale and so on, and these features can better represent the processed information inherited in the fMRI data collected in visual cortex. In this paper, a novel CapsNet architecture based visual reconstruction (CNAVR) computation model is developed to reconstruct image stimuli from human fMRI. The CNAVR is composed of linear encoding computation from capsule's features to fMRI data and inverse reconstruction computation. In the first part, we trained the CapsNet model to obtain the non-linear mappings from images to high-level capsule's features, and from high-level capsule's features to images again in an end-to-end manner. In the second part, we trained the non-linear mapping from fMRI data of selected voxels to high-level capsule's features. For a new image stimulus, we can use the method to predict the corresponding high-level capsule's features using fMRI data, and reconstruct image stimuli with the trained reconstruction part in the CapsNet. We evaluated the proposed CNAVR method on the open dataset of handwritten digital images, and exceeded about 10% than the accuracy of all existing state-of-the-art methods on the structural similarity index (SSIM). In addition, we explained the selected voxels in specific interpretable image features to prove the effectivity and generalization of the CNAVR method.

Natural Image Reconstruction from fMRI Based on Node–Edge Interaction and Multi–Scale Constraint

Fast and Lightweight Network Improves Serial Brain Section Stitching

Deep Natural Image Reconstruction from Human Brain Activity Based on Conditional Progressively Growing Generative Adversarial Networks

Functional diversity of visual cortex improves constraint-free natural image reconstruction from human brain activity

Deep image reconstruction from human brain activity

Jointly Fusing Multi-Scale Spatial-Logical Brain Networks: A Neural Decoding Method

Reconstruction of Natural Images from Human fMRI Using a Three-Stage Multi-Level Deep Fusion Model

Natural Image Reconstruction From fMRI Using Deep Learning: A Survey

Reconstructing Natural Images from Human Fmri by Alternating Encoding and Decoding with Shared Autoencoder Regularization

Multiscale Voxel Based Decoding For Enhanced Natural Image Reconstruction From Brain Activity

Reconstructing controllable faces from brain activity with hierarchical multiview representations

Analysis of Functional Areas of Human Brain Based on Reconstructed Images of DMFG-generated Countermeasure Network

Mind-bridge: Reconstructing Visual Images Based on Diffusion Model from Human Brain Activity

Accurate Reconstruction of Image Stimuli From Human Functional Magnetic Resonance Imaging Based on the Decoding Model With Capsule Network Architecture

Visual Image Reconstruction from fMRI Activation Using Multi-scale Support Vector Machine Decoders.

Brain-optimized inference improves reconstructions of fMRI brain activity

Reconstruction of natural images from evoked brain activity with a dictionary-based invertible encoding procedure

Pre-training Deep Neural Network for Decoding Multiple Brain Regions Can Enhance Image Reconstruction

Unidirectional brain-computer interface: Artificial neural network encoding natural images to fMRI response in the visual cortex

Siamese Reconstruction Network: Accurate Image Reconstruction from Human Brain Activity by Learning to Compare

Seeing through the Brain: Image Reconstruction of Visual Perception from Human Brain Signals