Abstract:Cross-modal fusion of different types of neuroimaging data has shown great promise for predicting the progression of Alzheimer's Disease(AD). However, most existing methods applied in neuroimaging can not efficiently fuse the functional and structural information from multi-modal neuroimages. In this work, a novel cross-modal transformer generative adversarial network(CT-GAN) is proposed to fuse functional information contained in resting-state functional magnetic resonance imaging (rs-fMRI) and structural information contained in Diffusion Tensor Imaging (DTI). The developed bi-attention mechanism can match functional information to structural information efficiently and maximize the capability of extracting complementary information from rs-fMRI and DTI. By capturing the deep complementary information between structural features and functional features, the proposed CT-GAN can detect the AD-related brain connectivity, which could be used as a bio-marker of AD. Experimental results show that the proposed model can not only improve classification performance but also detect the AD-related brain connectivity effectively.

What problem does this paper attempt to address?

The problem that this paper attempts to solve is the deep fusion of multi - modal brain network structure - function in Alzheimer's disease (AD). Specifically, most of the existing neuroimaging methods cannot efficiently fuse the functional and structural information from multi - modal neuroimaging. To overcome this problem, the authors propose a new cross - modal Transformer Generative Adversarial Network (CT - GAN) for fusing the functional information in resting - state functional magnetic resonance imaging (rs - fMRI) and the structural information in diffusion tensor imaging (DTI). By capturing the deep complementary information between structural and functional features, the proposed CT - GAN can detect brain connections related to AD, which can be used as biomarkers for AD. ### Main Contributions 1. **Propose a new cross - modal Transformer Generative Adversarial Network (CT - GAN)**: This model can efficiently fuse the functional and structural information in rs - fMRI and DTI. 2. **Introduce the Dual - Attention Mechanism (Bi - Attention Mechanism)**: This mechanism can effectively match the functional and structural information and maximize the ability to extract complementary information. 3. **Improve the classification performance**: The experimental results show that the proposed model can not only improve the classification performance but also effectively detect brain connections related to AD. ### Technical Details - **Model Architecture**: - **Generator**: It contains four modules: the Convolutional Neural Network (CNN) module is used to extract functional information from rs - fMRI; the Graph Convolutional Network (GCN) module is used to extract structural information from DTI; the F2S - Attention module is used to convert functional information into structural information; the S2F - Attention module is used to convert structural information into functional information. - **Decoders**: Two decoders respectively decode the multi - modal connectivity matrix into the corresponding structural connectivity (SC) and functional connectivity (FC). - **Discriminators**: Two discriminators are respectively used to judge whether SC and FC are generated by the model or output by the software template. - **Classifier**: Predict the AD stage according to the multi - modal connectivity. - **Loss Functions**: - **Adversarial Loss**: It is used to make the generated SC and FC matrices as close as possible to the actual SC and FC matrices. - **Classification Loss**: It is used to train the generator and classifier to improve the accuracy of AD stage prediction. - **Pair - wise Connectivity Reconstruction Loss**: It is used to impose additional topological constraints between the generator and decoder to minimize the differences between the generated SC and FC matrices and the actual SC and FC matrices. ### Experimental Results - **Dataset**: Use the ADNI public dataset, which contains DTI and rs - fMRI data of 268 subjects. - **Performance Evaluation**: Evaluate the model performance through three binary classification experiments (AD vs. NC, LMCI vs. NC, EMCI vs. NC), and use the detection accuracy (ACC), sensitivity (SEN) and specificity (SPEC) as evaluation indicators. - **Results**: The experimental results show that the proposed multi - modal fusion model is superior to other existing multi - modal fusion models in terms of the accuracy of predicting the AD stage. ### Conclusion This study proposes a new CT - GAN model, which realizes the efficient fusion between rs - fMRI and DTI through the dual - attention mechanism, improves the accuracy of AD prediction, and discovers brain connections related to AD. Although this study mainly focuses on AD, the proposed model can be easily extended and applied to other neurodegenerative diseases.

Cross-Modal Transformer GAN: A Brain Structure-Function Deep Fusing Framework for Alzheimer's Disease