Abstract:Introduction: Decoding brain activities is one of the most popular topics in neuroscience in recent years. And deep learning has shown high performance in fMRI data classification and regression, but its requirement for large amounts of data conflicts with the high cost of acquiring fMRI data. Methods: In this study, we propose an end-to-end temporal contrastive self-supervised learning algorithm, which learns internal spatiotemporal patterns within fMRI and allows the model to transfer learning to datasets of small size. For a given fMRI signal, we segmented it into three sections: the beginning, middle, and end. We then utilized contrastive learning by taking the end-middle (i.e., neighboring) pair as the positive pair, and the beginning-end (i.e., distant) pair as the negative pair. Results: We pretrained the model on 5 out of 7 tasks from the Human Connectome Project (HCP) and applied it in a downstream classification of the remaining two tasks. The pretrained model converged on data from 12 subjects, while a randomly initialized model required 100 subjects. We then transferred the pretrained model to a dataset containing unpreprocessed whole-brain fMRI from 30 participants, achieving an accuracy of 80.2 ± 4.7%, while the randomly initialized model failed to converge. We further validated the model's performance on the Multiple Domain Task Dataset (MDTB), which contains fMRI data of 26 tasks from 24 participants. Thirteen tasks of fMRI were selected as inputs, and the results showed that the pre-trained model succeeded in classifying 11 of the 13 tasks. When using the 7 brain networks as input, variations of the performance were observed, with the visual network performed as well as whole brain inputs, while the limbic network almost failed in all 13 tasks. Discussion: Our results demonstrated the potential of self-supervised learning for fMRI analysis with small datasets and unpreprocessed data, and for analysis of the correlation between regional fMRI activity and cognitive tasks.

Self-Supervised Transformers for fMRI representation

Self-Supervised Pre-training Tasks for an fMRI Time-series Transformer in Autism Detection

TF-Unet:An Automatic Cardiac MRI Image Segmentation Method

Self-Supervised Pretraining on Paired Sequences of fMRI Data for Transfer Learning to Brain Decoding Tasks

Transformer3: A Pure Transformer Framework for fMRI-Based Representations of Human Brain Function

SwiFT: Swin 4D fMRI Transformer

Modelling Spatio-Temporal Features of Task FMRI Data Via Spatio-Temporal Fusion-Transformer

Disentangling Spatial-Temporal Functional Brain Networks via Twin-Transformers

3D Brainformer: 3D Fusion Transformer for Brain Tumor Segmentation

A transformer model for learning spatiotemporal contextual representation in fMRI data

Frequency Learning Via Multi-Scale Fourier Transformer for MRI Reconstruction.

Function MRI Representation Learning via Self-supervised Transformer for Automated Brain Disorder Analysis.

Unsupervised brain imaging 3D anomaly detection and segmentation with transformers

Unsupervised Brain Anomaly Detection and Segmentation with Transformers

Transformer Model for Functional Near-Infrared Spectroscopy Classification

Community-Aware Transformer for Autism Prediction in fMRI Connectome

Embedding Human Brain Function via Transformer

Self-Supervised Transformer Model Training for a Sleep-EEG Foundation Model

Self-supervised pretraining improves the performance of classification of task functional magnetic resonance imaging

Swin fMRI Transformer Predicts Early Neurodevelopmental Outcomes from Neonatal fMRI

Classifying ASD based on time-series fMRI using spatial-temporal transformer