Abstract:The sensory cortex is characterized by general organizational principles such as topography and hierarchy. However, measured brain activity given identical input exhibits substantially different patterns across individuals. Although anatomical and functional alignment methods have been proposed in functional magnetic resonance imaging (fMRI) studies, it remains unclear whether and how hierarchical and fine-grained representations can be converted between individuals while preserving the encoded perceptual content. In this study, we trained a method of functional alignment called neural code converter that predicts a target subject's brain activity pattern from a source subject given the same stimulus, and analyzed the converted patterns by decoding hierarchical visual features and reconstructing perceived images. The converters were trained on fMRI responses to identical sets of natural images presented to pairs of individuals, using the voxels on the visual cortex that covers from V1 through the ventral object areas without explicit labels of the visual areas. We decoded the converted brain activity patterns into the hierarchical visual features of a deep neural network using decoders pre-trained on the target subject and then reconstructed images via the decoded features. Without explicit information about the visual cortical hierarchy, the converters automatically learned the correspondence between visual areas of the same levels. Deep neural network feature decoding at each layer showed higher decoding accuracies from corresponding levels of visual areas, indicating that hierarchical representations were preserved after conversion. The visual images were reconstructed with recognizable silhouettes of objects even with relatively small numbers of data for converter training. The decoders trained on pooled data from multiple individuals through conversions led to a slight improvement over those trained on a single individual. These results demonstrate that the hierarchical and fine-grained representation can be converted by functional alignment, while preserving sufficient visual information to enable inter-individual visual image reconstruction.

Deep image reconstruction from human brain activity

Deep Natural Image Reconstruction from Human Brain Activity Based on Conditional Progressively Growing Generative Adversarial Networks

Rethinking Visual Reconstruction: Experience-Based Content Completion Guided by Visual Cues

Reconstruction of Natural Images from Human fMRI Using a Three-Stage Multi-Level Deep Fusion Model

Semantics-Guided Hierarchical Feature Encoding Generative Adversarial Network for Visual Image Reconstruction From Brain Activity

Inter-individual deep image reconstruction via hierarchical neural code conversion

Visual Image Reconstructed Without Semantics from Human Brain Activity Using Linear Image Decoders and Nonlinear Noise Suppression

Seeing through the Brain: Image Reconstruction of Visual Perception from Human Brain Signals

Decoding human brain activity with deep learning

Reconstructing seen image from brain activity by visually-guided cognitive representation and adversarial learning

Reconstructing Perceptive Images from Brain Activity by Shape-Semantic GAN

Multiscale Voxel Based Decoding For Enhanced Natural Image Reconstruction From Brain Activity

Visual Image Reconstruction from fMRI Activation Using Multi-scale Support Vector Machine Decoders.

Siamese Reconstruction Network: Accurate Image Reconstruction from Human Brain Activity by Learning to Compare

Natural Image Reconstruction From fMRI Using Deep Learning: A Survey

Natural Image Reconstruction from fMRI Based on Node–Edge Interaction and Multi–Scale Constraint

More Than Meets the Eye: Self-Supervised Depth Reconstruction From Brain Activity

MindLDM: Reconstruct Visual Stimuli from Fmri Using Latent Diffusion Model

Improving visual image reconstruction from human brain activity using latent diffusion models via multiple decoded inputs

Reconstructing Natural Images from Human Fmri by Alternating Encoding and Decoding with Shared Autoencoder Regularization

Analysis of Functional Areas of Human Brain Based on Reconstructed Images of DMFG-generated Countermeasure Network