Abstract:In neuroscience, all kinds of computation models were designed to answer the open question of how sensory stimuli are encoded by neurons and conversely, how sensory stimuli can be decoded from neuronal activities. Especially, functional Magnetic Resonance Imaging (fMRI) studies have made many great achievements with the rapid development of deep network computation. However, comparing with the goal of decoding orientation, position and object category from human fMRI in visual cortex, accurate reconstruction of image stimuli is a still challenging work. Current prevailing methods were composed of two independent steps, (1) decoding intermediate features from human fMRI and (2) reconstruction using the decoded intermediate features. The new concept of 'capsule' and 'capsule' based neural network were proposed recently. The 'capsule' represented a kind of structure containing a group of neurons to perform better feature representation. Especially, the high-level capsule's features in the capsule network (CapsNet) contains various features of image stimuli such as semantic class, orientation, location, scale and so on, and these features can better represent the processed information inherited in the fMRI data collected in visual cortex. In this paper, a novel CapsNet architecture based visual reconstruction (CNAVR) computation model is developed to reconstruct image stimuli from human fMRI. The CNAVR is composed of linear encoding computation from capsule's features to fMRI data and inverse reconstruction computation. In the first part, we trained the CapsNet model to obtain the non-linear mappings from images to high-level capsule's features, and from high-level capsule's features to images again in an end-to-end manner. In the second part, we trained the non-linear mapping from fMRI data of selected voxels to high-level capsule's features. For a new image stimulus, we can use the method to predict the corresponding high-level capsule's features using fMRI data, and reconstruct image stimuli with the trained reconstruction part in the CapsNet. We evaluated the proposed CNAVR method on the open dataset of handwritten digital images, and exceeded about 10% than the accuracy of all existing state-of-the-art methods on the structural similarity index (SSIM). In addition, we explained the selected voxels in specific interpretable image features to prove the effectivity and generalization of the CNAVR method.

Reconstructing Natural Images from Human Fmri by Alternating Encoding and Decoding with Shared Autoencoder Regularization

Rethinking Visual Reconstruction: Experience-Based Content Completion Guided by Visual Cues

Reconstruction of Natural Images from Human fMRI Using a Three-Stage Multi-Level Deep Fusion Model

Reconstruction of natural images from evoked brain activity with a dictionary-based invertible encoding procedure

Reconstructing seen image from brain activity by visually-guided cognitive representation and adversarial learning

Deep Natural Image Reconstruction from Human Brain Activity Based on Conditional Progressively Growing Generative Adversarial Networks

An fMRI-based visual decoding framework combined with two-stage learning and asynchronous iterative update strategy

Foreground-attention in Neural Decoding: Guiding Loop-Enc-Dec to Reconstruct Visual Stimulus Images from Fmri

Natural Image Reconstruction From fMRI Using Deep Learning: A Survey

Visual Image Reconstructed Without Semantics from Human Brain Activity Using Linear Image Decoders and Nonlinear Noise Suppression

Reconstructing controllable faces from brain activity with hierarchical multiview representations

Accurate Reconstruction of Image Stimuli From Human Functional Magnetic Resonance Imaging Based on the Decoding Model With Capsule Network Architecture

Deep image reconstruction from human brain activity

Visual Image Reconstruction from fMRI Activation Using Multi-scale Support Vector Machine Decoders.

Reconstructing Retinal Visual Images from 3T fMRI Data Enhanced by Unsupervised Learning

Multiscale Voxel Based Decoding For Enhanced Natural Image Reconstruction From Brain Activity

Images Structure Reconstruction from Fmri by Unsupervised Learning Based on VAE

Reconstructing Visual Stimulus Images from EEG Signals Based on Deep Visual Representation Model

Pre-training Deep Neural Network for Decoding Multiple Brain Regions Can Enhance Image Reconstruction

Functional diversity of visual cortex improves constraint-free natural image reconstruction from human brain activity

Siamese Reconstruction Network: Accurate Image Reconstruction from Human Brain Activity by Learning to Compare