MindBridge: A Cross-Subject Brain Decoding Framework

Shizun Wang,Songhua Liu,Zhenxiong Tan,Xinchao Wang

2024-04-11

Abstract:Brain decoding, a pivotal field in neuroscience, aims to reconstruct stimuli from acquired brain signals, primarily utilizing functional magnetic resonance imaging (fMRI). Currently, brain decoding is confined to a per-subject-per-model paradigm, limiting its applicability to the same individual for whom the decoding model is trained. This constraint stems from three key challenges: 1) the inherent variability in input dimensions across subjects due to differences in brain size; 2) the unique intrinsic neural patterns, influencing how different individuals perceive and process sensory information; 3) limited data availability for new subjects in real-world scenarios hampers the performance of decoding models. In this paper, we present a novel approach, MindBridge, that achieves cross-subject brain decoding by employing only one model. Our proposed framework establishes a generic paradigm capable of addressing these challenges by introducing biological-inspired aggregation function and novel cyclic fMRI reconstruction mechanism for subject-invariant representation learning. Notably, by cycle reconstruction of fMRI, MindBridge can enable novel fMRI synthesis, which also can serve as pseudo data augmentation. Within the framework, we also devise a novel reset-tuning method for adapting a pretrained model to a new subject. Experimental results demonstrate MindBridge's ability to reconstruct images for multiple subjects, which is competitive with dedicated subject-specific models. Furthermore, with limited data for a new subject, we achieve a high level of decoding accuracy, surpassing that of subject-specific models. This advancement in cross-subject brain decoding suggests promising directions for wider applications in neuroscience and indicates potential for more efficient utilization of limited fMRI data in real-world scenarios. Project page:

Computer Vision and Pattern Recognition,Artificial Intelligence

What problem does this paper attempt to address?

The problem that this paper attempts to solve is to achieve cross - subject image reconstruction in the field of brain decoding. Current brain - decoding methods are usually limited by the need to train an independent model for each subject, which not only limits their applicability among different individuals, but also leads to an increase in model storage and training costs. The main challenges mentioned in the paper include:

1. **Size Variability**: Due to the inherent differences in brain size and structure, there are significant differences in the fMRI signal sizes of different subjects, which requires flexible methods to handle these changes.
2. **Diverse Neural Responses**: Each subject's brain processes stimuli in a unique way because of differences in their experiences, biases, and cognitive patterns, which complicates the unified interpretation of brain signals.
3. **Scarcity of Data for New Subjects**: In practical applications, obtaining a large amount of fMRI data for new subjects is both time - consuming and expensive, which seriously hinders the training and adaptation of brain - decoding models for new subjects.

To overcome these challenges, the paper proposes a new framework named "MindBridge", aiming to achieve cross - subject brain decoding through a single model. MindBridge introduces biologically - inspired aggregation functions and a novel cyclic fMRI reconstruction mechanism to learn subject - invariant representations. In addition, the paper also proposes a new fine - tuning method - reset - tuning, which is used to quickly adapt the pre - trained model to new subjects, effectively solving the problem of data scarcity. Experimental results show that MindBridge can not only achieve effective image reconstruction among multiple subjects, but also performs better than models trained for specific subjects when data is limited. This progress makes broader applications possible, especially in the efficient use of limited fMRI data in practical scenarios.