Abstract:The sensory cortex is characterized by general organizational principles such as topography and hierarchy. However, measured brain activity given identical input exhibits substantially different patterns across individuals. Although anatomical and functional alignment methods have been proposed in functional magnetic resonance imaging (fMRI) studies, it remains unclear whether and how hierarchical and fine-grained representations can be converted between individuals while preserving the encoded perceptual content. In this study, we trained a method of functional alignment called neural code converter that predicts a target subject's brain activity pattern from a source subject given the same stimulus, and analyzed the converted patterns by decoding hierarchical visual features and reconstructing perceived images. The converters were trained on fMRI responses to identical sets of natural images presented to pairs of individuals, using the voxels on the visual cortex that covers from V1 through the ventral object areas without explicit labels of the visual areas. We decoded the converted brain activity patterns into the hierarchical visual features of a deep neural network using decoders pre-trained on the target subject and then reconstructed images via the decoded features. Without explicit information about the visual cortical hierarchy, the converters automatically learned the correspondence between visual areas of the same levels. Deep neural network feature decoding at each layer showed higher decoding accuracies from corresponding levels of visual areas, indicating that hierarchical representations were preserved after conversion. The visual images were reconstructed with recognizable silhouettes of objects even with relatively small numbers of data for converter training. The decoders trained on pooled data from multiple individuals through conversions led to a slight improvement over those trained on a single individual. These results demonstrate that the hierarchical and fine-grained representation can be converted by functional alignment, while preserving sufficient visual information to enable inter-individual visual image reconstruction.

Generic decoding of seen and imagined objects using hierarchical visual features

Multimodal deep neural decoding reveals highly resolved spatiotemporal profile of visual object representation in humans

Hierarchical Neural Representation of Dreamed Objects Revealed by Brain Decoding with Deep Neural Network Features

Decoding Generic Visual Representations From Human Brain Activity using Machine Learning

Inter-individual deep image reconstruction via hierarchical neural code conversion

Brain decoding: toward real-time reconstruction of visual perception

Long short‐term memory‐based neural decoding of object categories evoked by natural images

Decoding Visual Experience and Mapping Semantics through Whole-Brain Analysis Using fMRI Foundation Models

‘When’ and ‘what’ did you see? A novel fMRI-based visual decoding framework

MindGPT: Interpreting What You See with Non-invasive Brain Recordings

Decoding Visual Neural Representations by Multimodal Learning of Brain-Visual-Linguistic Features

Toward Generalizing Visual Brain Decoding to Unseen Subjects

Category Decoding of Visual Stimuli From Human Brain Activity Using a Bidirectional Recurrent Neural Network to Simulate Bidirectional Information Flows in Human Visual Cortices

Decoding human brain activity with deep learning

Convolutional neural network-based encoding and decoding of visual object recognition in space and time

Decoding dynamic visual scenes across the brain hierarchy

Neural Encoding and Decoding with Deep Learning for Dynamic Natural Vision

Decoding Visual fMRI Stimuli from Human Brain Based on Graph Convolutional Neural Network

From Sight to Insight: A Multi-task Approach with the Visual Language Decoding Model

Visual Encoding and Decoding of the Human Brain Based on Shared Features

Brain-aligning of semantic vectors improves neural decoding of visual stimuli