Abstract:Objective: Modeling dynamic neuronal activity within brain networks enables the precise tracking of rapid temporal fluctuations across different brain regions. However, current approaches in computational neuroscience fall short of capturing and representing the spatiotemporal dynamics within each brain network. We developed a novel weakly supervised spatiotemporal dense prediction model capable of generating personalized 4D dynamic brain networks from fMRI data, providing a more granular representation of brain activity over time. Methods: We developed a model that leverages the vision transformer (ViT) as its backbone, jointly encoding spatial and temporal information from fMRI inputs using two different configurations: space-time and sequential encoders. The model generates 4D brain network maps that evolve over time, capturing dynamic changes in both spatial and temporal dimensions. In the absence of ground-truth data, we used spatially constrained windowed independent component analysis (ICA) components derived from fMRI data as weak supervision to guide the training process. The model was evaluated using large-scale resting-state fMRI datasets, and statistical analyses were conducted to assess the effectiveness of the generated dynamic maps using various metrics. Results: Our model effectively produced 4D brain maps that captured both inter-subject and temporal variations, offering a dynamic representation of evolving brain networks. Notably, the model demonstrated the ability to produce smooth maps from noisy priors, effectively denoising the resulting brain dynamics. Additionally, statistically significant differences were observed in the temporally averaged brain maps, as well as in the summation of absolute temporal gradient maps, between patients with schizophrenia and healthy controls. For example, within the Default Mode Network (DMN), significant differences emerged in the temporally averaged space-time configurations, particularly in the thalamus, where healthy controls exhibited higher activity levels compared to subjects with schizophrenia. These findings highlight the model's potential for differentiating between clinical populations. Conclusion: The proposed spatiotemporal dense prediction model offers an effective approach for generating dynamic brain maps by capturing significant spatiotemporal variations in brain activity. Leveraging weak supervision through ICA components enables the model to learn dynamic patterns without direct ground-truth data, making it a robust and efficient tool for brain mapping. Significance: This work presents an important new approach for dynamic brain mapping, potentially opening up new opportunities for studying brain dynamics within specific networks. By framing the problem as a spatiotemporal dense prediction task in computer vision, we leverage the spatiotemporal ViT architecture combined with weakly supervised learning techniques to efficiently and effectively estimate these maps.

NT-ViT: Neural Transcoding Vision Transformers for EEG-to-fMRI Synthesis

MSVTNet: Multi-Scale Vision Transformer Neural Network for EEG-Based Motor Imagery Decoding

Neuro-Vision to Language: Enhancing Brain Recording-based Visual Reconstruction and Language Interaction

Fusing Pretrained ViTs with TCNet for Enhanced EEG Regression

Towards Neural Foundation Models for Vision: Aligning EEG, MEG, and fMRI Representations for Decoding, Encoding, and Modality Conversion

Brainformer: Mimic Human Visual Brain Functions to Machine Vision Models via fMRI

st-DenseViT: A Weakly Supervised Spatiotemporal Vision Transformer for Dense Prediction of Dynamic Brain Networks

MultiCrossViT: Multimodal Vision Transformer for Schizophrenia Prediction using Structural MRI and Functional Network Connectivity Data

Gray Matters: An Efficient Vision Transformer GAN Framework for Predicting Functional Network Connectivity Biomarkers from Brain Structure

NeuroBOLT: Resting-state EEG-to-fMRI Synthesis with Multi-dimensional Feature Mapping

GET: A Generative EEG Transformer for Continuous Context-Based Neural Signals

Brain Structural Connectivity Guided Vision Transformers for Identification of Functional Connectivity Characteristics in Preterm Neonates

FMViT: A multiple-frequency mixing Vision Transformer

Unidirectional brain-computer interface: Artificial neural network encoding natural images to fMRI response in the visual cortex

A multimodal vision transformer for interpretable fusion of functional and structural neuroimaging data

NeuroGen: Activation optimized image synthesis for discovery neuroscience

Large Transformers are Better EEG Learners

Brain decoding: toward real-time reconstruction of visual perception

ViT2EEG: Leveraging Hybrid Pretrained Vision Transformers for EEG Data

See Through Their Minds: Learning Transferable Neural Representation from Cross-Subject fMRI

PTNet: A High-Resolution Infant MRI Synthesizer Based on Transformer