Abstract:Previous brain decoding research primarily involves single-subject studies, reconstructing stimuli via fMRI activity from the same subject. Our study aims to introduce a generalization technique for cross-subject brain decoding, facilitated by exploring data alignment methods. We utilized the NSD dataset, a comprehensive 7T fMRI vision experiment involving multiple subjects exposed to 9841 images, 982 of which were viewed by all. Our approach involved training a decoding model on one subject, aligning others' data to this space, and testing the decoding on the second subject. We compared ridge regression, hyper alignment, and anatomical alignment techniques for fMRI data alignment. We established that cross-subject brain decoding is feasible, even using around 10% of the total data, or 982 common images, with comparable performance to single-subject decoding. Ridge regression was the best method for functional alignment. Through subject alignment, we achieved superior brain decoding and a potential 90% reduction in scan time. This could pave the way for more efficient experiments and further advancements in the field, typically requiring an exorbitant 20-hour scan time per subject.

What problem does this paper attempt to address?

The main problem that this paper attempts to solve is to achieve cross - subject brain decoding, that is, to decode the brain activity of one subject through the functional magnetic resonance imaging (fMRI) activity of another subject, thereby reconstructing visual stimuli. Specifically, the research aims to align the brain activities of different subjects into a common space through simple alignment techniques to achieve fine - grained cross - subject brain decoding. ### Main Problems and Background 1. **Limitations of Single - Subject Brain Decoding**: - Current brain - decoding research mainly focuses on a single subject, that is, reconstructing the perceived stimuli from the fMRI activity of a certain subject. This method requires training a model separately for each subject, resulting in high data collection and computational costs. 2. **The Need for Cross - Subject Brain Decoding**: - The goal of cross - subject brain decoding is to develop a general - purpose technology that can use a model trained with the data of one subject to decode the brain activities of other subjects, thereby reducing the time and cost of data collection. 3. **The Challenge of Data Alignment**: - Due to the differences in brain structures and functions among different subjects, how to align these differences into a common space is a key challenge. ### Research Methods To achieve cross - subject brain decoding, the researchers used the following methods: - **Data Set**: The research used the Natural Scenes Dataset (NSD), which is a 7T fMRI data set of multiple subjects when viewing natural images. - **Alignment Techniques**: Three different alignment techniques were explored: - **Anatomical Alignment**: Alignment based on anatomical structures. - **HyperAlignment**: A functional alignment method in high - dimensional space. - **Ridge Regression**: A functional alignment method based on linear combinations. ### Main Findings 1. **Feasibility of Cross - Subject Brain Decoding**: - The research shows that even with a small amount of shared data (about 10%), fine - grained cross - subject brain decoding can be achieved, and the performance is close to that of single - subject decoding. 2. **The Best Alignment Method**: - Ridge Regression performs the best in decoding fine - grained information, better than other alignment methods. 3. **Significantly Reducing Scan Time**: - By obtaining only the data used for alignment, the scan time can be reduced by 90%, thus greatly improving the experimental efficiency. ### Conclusion This research shows the potential of cross - subject brain decoding, especially in reducing scan time and data requirements. Ridge Regression, as an effective alignment method, performs excellently in fine - grained brain decoding and provides a new direction for future research.

Through their eyes: multi-subject Brain Decoding with simple alignment techniques

Aligning brain functions boosts the decoding of visual semantics in novel subjects

MindBridge: A Cross-Subject Brain Decoding Framework

MindFormer: Semantic Alignment of Multi-Subject fMRI for Brain Decoding

See Through Their Minds: Learning Transferable Neural Representation from Cross-Subject fMRI

Group-level Brain Decoding with Deep Learning

Across-subject ensemble-learning alleviates the need for large samples for fMRI decoding

Group‐level brain decoding with deep learning

Neural decoding from stereotactic EEG: accounting for electrode variability across subjects

MindTuner: Cross-Subject Visual Decoding with Visual Fingerprint and Semantic Correction

Multi-subject data augmentation for target subject semantic decoding with deep multi-view adversarial learning

Toward Generalizing Visual Brain Decoding to Unseen Subjects

Graph-Based Decoding Model for Functional Alignment of Unaligned fMRI Data

Enhancing Cross-Subject Fmri-to-video Decoding with Global-Local Functional Alignment

Brain Dialogue Interface (BDI): A User-Friendly fMRI Model for Interactive Brain Decoding

Deep Neural Networks and Brain Alignment: Brain Encoding and Decoding (Survey)

Decoding Visual Experience and Mapping Semantics through Whole-Brain Analysis Using fMRI Foundation Models

Deep learning of fMRI big data: a novel approach to subject-transfer decoding

MindShot: Brain Decoding Framework Using Only One Image

Towards Neural Foundation Models for Vision: Aligning EEG, MEG, and fMRI Representations for Decoding, Encoding, and Modality Conversion