Abstract:In neuroimaging analysis, fMRI can well assess the function changes for brain diseases with no obvious structural lesions. To date, most deep-learning-based fMRI studies have employed functional connectivity (FC) as the basic feature for disease classification. However, FC is calculated on time series of predefined regions of interest and neglects detailed information contained in each voxel. Another drawback of using FC is the limited sample size for the training of deep models. The low representation ability of FC leads to poor performance in clinical practice, especially when dealing with multimodal medical data involving multiple types of visual signals and textual records for brain diseases. To overcome this bottleneck problem in the fMRI feature modality, we propose BrainFormer, an end-to-end functional interaction learning method for brain disease classification with single fMRI volume. Unlike traditional deep learning methods that construct convolution and transformers on FC, BrainFormer learns the functional interaction from fMRI signals, by modeling the local cues within each voxel with 3D convolutions and capturing the global correlations among distant regions with specially designed global attention mechanisms from shallow layers to deep layers. Meanwhile, BrainFormer can deal with multimodal medical data including fMRI volume, structural MRI, FC features and phenotypic data to achieve more comprehensive brain disease diagnosis. We evaluate BrainFormer on five independent multi-site datasets on autism, Alzheimer's disease, depression, attention deficit hyperactivity disorder and headache disorders. The results demonstrate its effectiveness and generalizability for multiple brain diseases diagnosis with multimodal features. BrainFormer may promote precision of neuroimaging-based diagnosis in clinical practice and motivate future studies on fMRI analysis.

Incomplete multi-modal brain image fusion for epilepsy classification

Multimodal Triplet Attention Network for Brain Disease Diagnosis

MMF-NNs: Multi-modal Multi-granularity Fusion Neural Networks for brain networks and its application to epilepsy identification

Multimodal Brain Network Jointly Construction and Fusion for Diagnosis of Epilepsy

Multimodal Fusion of Brain Imaging Data: Methods and Applications

Improved Classification Performance of EEG-fNIRS Multimodal Brain-Computer Interface Based on Multi-Domain Features and Multi-Level Progressive Learning

Multi-source Feature Learning for Joint Analysis of Incomplete Multiple Heterogeneous Neuroimaging Data

Multi-modal feature fusion with multi-head self-attention for epileptic EEG signals

Spatially-Constrained Fisher Representation for Brain Disease Identification with Incomplete Multi-Modal Neuroimages.

Multi-graph Fusion for Functional Neuroimaging Biomarker Detection

Multimodal Brain Disease Classification with Functional Interaction Learning from Single fMRI Volume

Multi-modal bioelectrical signal fusion analysis based on different acquisition devices and scene settings: Overview, challenges, and novel orientation

Incomplete Multimodal Learning for Complex Brain Disorders Prediction

Multimodal Magnetic Resonance Image and Electroencephalogram Constrained Fusion Algorithm Using Deep Learning

Fusing Structural and Functional Connectivities using Disentangled VAE for Detecting MCI

Coherent Pattern in Multi-Layer Brain Networks: Application to Epilepsy Identification

Incomplete Learning of Multi-Modal Connectome for Brain Disorder Diagnosis Via Modal-Mixup and Deep Supervision.

Brain Structure-Function Fusing Representation Learning using Adversarial Decomposed-VAE for Analyzing MCI

Multimodal Fusion of EEG and Fmri for Epilepsy Detection

Simultaneous EEG-fNIRS Data Classification Through Selective Channel Representation and Spectrogram Imaging

Multimodal MRI Volumetric Data Fusion With Convolutional Neural Networks