Abstract:Brain decoding, a pivotal field in neuroscience, aims to reconstruct stimuli from acquired brain signals, primarily utilizing functional magnetic resonance imaging (fMRI). Currently, brain decoding is confined to a per-subject-per-model paradigm, limiting its applicability to the same individual for whom the decoding model is trained. This constraint stems from three key challenges: 1) the inherent variability in input dimensions across subjects due to differences in brain size; 2) the unique intrinsic neural patterns, influencing how different individuals perceive and process sensory information; 3) limited data availability for new subjects in real-world scenarios hampers the performance of decoding models. In this paper, we present a novel approach, MindBridge, that achieves cross-subject brain decoding by employing only one model. Our proposed framework establishes a generic paradigm capable of addressing these challenges by introducing biological-inspired aggregation function and novel cyclic fMRI reconstruction mechanism for subject-invariant representation learning. Notably, by cycle reconstruction of fMRI, MindBridge can enable novel fMRI synthesis, which also can serve as pseudo data augmentation. Within the framework, we also devise a novel reset-tuning method for adapting a pretrained model to a new subject. Experimental results demonstrate MindBridge's ability to reconstruct images for multiple subjects, which is competitive with dedicated subject-specific models. Furthermore, with limited data for a new subject, we achieve a high level of decoding accuracy, surpassing that of subject-specific models. This advancement in cross-subject brain decoding suggests promising directions for wider applications in neuroscience and indicates potential for more efficient utilization of limited fMRI data in real-world scenarios. Project page: https://littlepure2333.github.io/MindBridge

BrainCLIP: Bridging Brain and Visual-Linguistic Representation Via CLIP for Generic Natural Visual Stimulus Decoding

CLIP-MUSED: CLIP-Guided Multi-Subject Visual Neural Information Semantic Decoding

NeuroClips: Towards High-fidelity and Smooth fMRI-to-Video Reconstruction

BrainChat: Decoding Semantic Information from fMRI using Vision-language Pretrained Models

Decoding Visual Experience and Mapping Semantics through Whole-Brain Analysis Using fMRI Foundation Models

Neuro-Vision to Language: Enhancing Brain Recording-based Visual Reconstruction and Language Interaction

How Much Can CLIP Benefit Vision-and-Language Tasks?

MindSemantix: Deciphering Brain Visual Experiences with a Brain-Language Model

Decoding Visual Neural Representations by Multimodal Learning of Brain-Visual-Linguistic Features

Identifying Shared Decodable Concepts in the Human Brain Using Image-Language Foundation Models

MindLDM: Reconstruct Visual Stimuli from Fmri Using Latent Diffusion Model

Neural-MCRL: Neural Multimodal Contrastive Representation Learning for EEG-based Visual Decoding

MindBridge: A Cross-Subject Brain Decoding Framework

NeuroCLIP: Neuromorphic Data Understanding by CLIP and SNN

BrainVis: Exploring the Bridge between Brain and Visual Signals via Image Reconstruction

A dual‐channel language decoding from brain activity with progressive transfer training

NeuroCine: Decoding Vivid Video Sequences from Human Brain Activties

From Sight to Insight: A Multi-task Approach with the Visual Language Decoding Model

SpaceCLIP: A Vision-Language Pretraining Framework With Spatial Reconstruction On Text

CRIS: CLIP-Driven Referring Image Segmentation