Abstract:The MultiViT model combines structural and functional neuroimaging data for the prediction of schizophrenia and integrates vision transformers with cross‐attention layers in order to preserve mutual information. The pipeline generates highly interpretable cross‐attention‐based brain saliency maps and emphasizes functional network connectivity patterns related to the disorder. Multimodal neuroimaging is an emerging field that leverages multiple sources of information to diagnose specific brain disorders, especially when deep learning‐based AI algorithms are applied. The successful combination of different brain imaging modalities using deep learning remains a challenging yet crucial research topic. The integration of structural and functional modalities is particularly important for the diagnosis of various brain disorders, where structural information plays a crucial role in diseases such as Alzheimer's, while functional imaging is more critical for disorders such as schizophrenia. However, the combination of functional and structural imaging modalities can provide a more comprehensive diagnosis. In this work, we present MultiViT, a novel diagnostic deep learning model that utilizes vision transformers and cross‐attention mechanisms to effectively fuse information from 3D gray matter maps derived from structural MRI with functional network connectivity matrices obtained from functional MRI using the ICA algorithm. MultiViT achieves an AUC of 0.833, outperforming both our unimodal and multimodal baselines, enabling more accurate classification and diagnosis of schizophrenia. In addition, using vision transformer's unique attentional maps in combination with cross‐attentional mechanisms and brain function information, we identify critical brain regions in 3D gray matter space associated with the characteristics of schizophrenia. Our research not only significantly improves the accuracy of AI‐based automated imaging diagnostics for schizophrenia, but also pioneers a rational and advanced data fusion approach by replacing complex, high‐dimensional fMRI information with functional network connectivity, integrating it with representative structural data from 3D gray matter images, and further providing interpretative biomarker localization in a 3D structural space.

Cross-Modal Synthesis of Structural MRI and Functional Connectivity Networks via Conditional ViT-GANs

Gray Matters: An Efficient Vision Transformer GAN Framework for Predicting Functional Network Connectivity Biomarkers from Brain Structure

MultiCrossViT: Multimodal Vision Transformer for Schizophrenia Prediction using Structural MRI and Functional Network Connectivity Data

A multimodal vision transformer for interpretable fusion of functional and structural neuroimaging data

Cross-Modal Transformer GAN: A Brain Structure-Function Deep Fusing Framework for Alzheimer's Disease

Cross-Modality Translation with Generative Adversarial Networks to Unveil Alzheimer's Disease Biomarkers

A new VAE-GAN model to synthesize arterial spin labeling images from structural MRI

Deep Non-Linear Embedding Deformation Network for Cross-Modal Brain MRI Synthesis.

Cross2SynNet: cross-device-cross-modal synthesis of routine brain MRI sequences from CT with brain lesion

Gray Matters: ViT-GAN Framework for Identifying Schizophrenia Biomarkers Linking Structural MRI and Functional Connectivity

Style transfer in conditional GANs for cross-modality synthesis of brain magnetic resonance images

Functional network connectivity (FNC)-based generative adversarial network (GAN) and its applications in classification of mental disorders.

Can functional connectivity be used to refine structural connectivity strength by combining neural computational model and generative adversarial network?

MRI to PET Cross-Modality Translation using Globally and Locally Aware GAN (GLA-GAN) for Multi-Modal Diagnosis of Alzheimer's Disease

Alzheimer's Disease Prediction via Brain Structural-Functional Deep Fusing Network

DCNN-GAN: Reconstructing Realistic Image from fMRI

MRI-Trans-GAN: 3D MRI Cross-Modality Translation

Reconstructing Rapid Natural Vision with fMRI-Conditional Video Generative Adversarial Network

EC-GAN: Inferring Brain Effective Connectivity via Generative Adversarial Networks

Integrated Brain Connectivity Analysis with fMRI, DTI, and sMRI Powered by Interpretable Graph Neural Networks

Multi-Constraint Transferable Generative Adversarial Networks for Cross-Modal Brain Image Synthesis