Abstract:Reconstructing visual experience from brain responses measured by functional magnetic resonance imaging (fMRI) is a challenging yet important research topic in brain decoding, especially it has proved more difficult to decode visually similar stimuli, such as faces. Although face attributes are known as the key to face recognition, most existing methods generally ignore how to decode facial attributes more precisely in perceived face reconstruction, which often leads to indistinguishable reconstructed faces. To solve this problem, we propose a novel neural decoding framework called VSPnet (voxel2style2pixel) by establishing hierarchical encoding and decoding networks with disentangled latent representations as media, so that to recover visual stimuli more elaborately. And we design a hierarchical visual encoder (named HVE) to pre-extract features containing both high-level semantic knowledge and low-level visual details from stimuli. The proposed VSPnet consists of two networks: Multi-branch cognitive encoder and style-based image generator. The encoder network is constructed by multiple linear regression branches to map brain signals to the latent space provided by the pre-extracted visual features and obtain representations containing hierarchical information consistent to the corresponding stimuli. We make the generator network inspired by StyleGAN to untangle the complexity of fMRI representations and generate images. And the HVE network is composed of a standard feature pyramid over a ResNet backbone. Extensive experimental results on the latest public datasets have demonstrated the reconstruction accuracy of our proposed method outperforms the state-of-the-art approaches and the identifiability of different reconstructed faces has been greatly improved. In particular, we achieve feature editing for several facial attributes in fMRI domain based on the multiview ( i.e. , visual stimuli and evoked fMRI) latent representations.

A Multi-View Fmri Model with Voxel Inner State for Vision Encoding

A Visual Encoding Model Based on Deep Neural Networks and Transfer Learning for Brain Activity Measured by Functional Magnetic Resonance Imaging

A Multimodal Visual Encoding Model Aided by Introducing Verbal Semantic Information

Effective and Efficient ROI-wise Visual Encoding Using an End-to-End CNN Regression Model and Selective Optimization

A visual encoding model based on deep neural networks and transfer learning

Decoding Visual Experience and Mapping Semantics through Whole-Brain Analysis Using fMRI Foundation Models

Encoding brain network response to free viewing of videos

Reconstructing controllable faces from brain activity with hierarchical multiview representations

VTVBrain: A Two-stage Brain Encoding Model for Decoding Key Neural Responses in Multimodal Contexts

An fMRI-based visual decoding framework combined with two-stage learning and asynchronous iterative update strategy

Structurally-constrained encoding framework using a multi-voxel reduced-rank latent model for human natural vision

Neural encoding and interpretation for high-level visual cortices based on fMRI using image caption features

Variational Autoencoder: An Unsupervised Model for Modeling and Decoding fMRI Activity in Visual Cortex

Encoding functional brain interactions from computational visual features

GaborNet Visual Encoding: A Lightweight Region-Based Visual Encoding Model With Good Expressiveness and Biological Interpretability

Retinotopy Inspired Brain Encoding Model and the All-for-One Training Recipe

‘When’ and ‘what’ did you see? A novel fMRI-based visual decoding framework

An Encoding Framework With Brain Inner State for Natural Image Identification

Upgrading Voxel-wise Encoding Model via Integrated Integration over Features and Brain Networks

Visual Encoding and Decoding of the Human Brain Based on Shared Features

Survey of encoding and decoding of visual stimulus via FMRI: an image analysis perspective