Abstract:Multimodal neuroimaging modeling has become a widely used approach but confronts considerable challenges due to heterogeneity, which encompasses variability in data types, scales, and formats across modalities. This variability necessitates the deployment of advanced computational methods to integrate and interpret these diverse datasets within a cohesive analytical framework. In our research, we amalgamate functional magnetic resonance imaging (fMRI), diffusion tensor imaging (DTI), and structural MRI (sMRI) into a cohesive framework. This integration capitalizes on the unique strengths of each modality and their inherent interconnections, aiming for a comprehensive understanding of the brain's connectivity and anatomical characteristics. Utilizing the Glasser atlas for parcellation, we integrate imaging-derived features from various modalities-functional connectivity from fMRI, structural connectivity from DTI, and anatomical features from sMRI-within consistent regions. Our approach incorporates a masking strategy to differentially weight neural connections, thereby facilitating a holistic amalgamation of multimodal imaging data. This technique enhances interpretability at connectivity level, transcending traditional analyses centered on singular regional attributes. The model is applied to the Human Connectome Project's Development study to elucidate the associations between multimodal imaging and cognitive functions throughout youth. The analysis demonstrates improved predictive accuracy and uncovers crucial anatomical features and essential neural connections, deepening our understanding of brain structure and function. This study not only advances multi-modal neuroimaging analytics by offering a novel method for the integrated analysis of diverse imaging modalities but also improves the understanding of intricate relationship between the brain's structural and functional networks and cognitive development.

Predicting Diverse Functional Connectivity from Structural Connectivity Based on Multi-contexts Discriminator GAN

Regional Brain Fusion: Graph Convolutional Network for Alzheimer's Disease Prediction and Analysis

DiffGAN-F2S: Symmetric and Efficient Denoising Diffusion GANs for Structural Connectivity Prediction from Brain fMRI

Alzheimer's Disease Prediction via Brain Structural-Functional Deep Fusing Network

Ensemble Manifold Regularized Multi-Modal Graph Convolutional Network for Cognitive Ability Prediction

Multigroup Recognition of Dementia Patients with Dynamic Brain Connectivity under Multimodal Cortex Parcellation

Fusing Structural and Functional Connectivities using Disentangled VAE for Detecting MCI

Ensemble manifold based regularized multi-modal graph convolutional network for cognitive ability prediction

Predicting Human Intrinsic Functional Connectivity From Structural Connectivity: An Artificial Neural Network Approach

Brain Imaging-to-Graph Generation using Adversarial Hierarchical Diffusion Models for MCI Causality Analysis

Multiscale Functional Connectome Abnormality Predicts Cognitive Outcomes in Subcortical Ischemic Vascular Disease

A Heterogeneous Graph Neural Network Fusing Functional and Structural Connectivity for MCI Diagnosis

Integrated Brain Connectivity Analysis with fMRI, DTI, and sMRI Powered by Interpretable Graph Neural Networks

Multi-Scale FC-Based Multi-Order GCN: A Novel Model for Predicting Individual Behavior From fMRI

Cross-Modal Transformer GAN: A Brain Structure-Function Deep Fusing Framework for Alzheimer's Disease

Interpretable AI for relating brain structural and functional connectomes

Triplet Graph Convolutional Network for Multi-scale Analysis of Functional Connectivity Using Functional MRI.

Characterization Multimodal Connectivity of Brain Network by Hypergraph GAN for Alzheimer's Disease Analysis

BG-GAN: Generative AI Enables Representing Brain Structure-Function Connections for Alzheimer's Disease

EC-GAN: Inferring Brain Effective Connectivity via Generative Adversarial Networks