Abstract:Recently, it has become progressively more evident that classic diagnostic labels are unable to reliably describe the complexity and variability of several clinical phenotypes. This is particularly true for a broad range of neuropsychiatric illnesses (e.g., depression, anxiety disorders, behavioral phenotypes). Patient heterogeneity can be better described by grouping individuals into novel categories based on empirically derived sections of intersecting continua that span across and beyond traditional categorical borders. In this context, neuroimaging data carry a wealth of spatiotemporally resolved information about each patient's brain. However, they are usually heavily collapsed a priori through procedures which are not learned as part of model training, and consequently not optimized for the downstream prediction task. This is because every individual participant usually comes with multiple whole-brain 3D imaging modalities often accompanied by a deep genotypic and phenotypic characterization, hence posing formidable computational challenges. In this paper we design a deep learning architecture based on generative models rooted in a modular approach and separable convolutional blocks to a) fuse multiple 3D neuroimaging modalities on a voxel-wise level, b) convert them into informative latent embeddings through heavy dimensionality reduction, c) maintain good generalizability and minimal information loss. As proof of concept, we test our architecture on the well characterized Human Connectome Project database demonstrating that our latent embeddings can be clustered into easily separable subject strata which, in turn, map to different phenotypical information which was not included in the embedding creation process. This may be of aid in predicting disease evolution as well as drug response, hence supporting mechanistic disease understanding and empowering clinical trials.

Multimodal Fusion of Brain Imaging Data: Methods and Applications

Advances in multimodal data fusion in neuroimaging: Overview, challenges, and novel orientation

A review of multivariate methods for multimodal fusion of brain imaging data

Editorial: Multimodal Brain Image Fusion: Methods, Evaluations, and Applications

Multimodal MRI Volumetric Data Fusion With Convolutional Neural Networks

Deep Learning Based Multimodal Biomedical Data Fusion: an Overview and Comparative Review

Multimodal medical image fusion algorithm in the era of big data

Research Progress and Practical Application of Multimodal Neuroimaging Technology

Multimodal Neuroimaging Computing: the Workflows, Methods, and Platforms.

Multimodal Magnetic Resonance Imaging for Brain Disorders: Advances and Perspectives

Deep Multimodal Data Fusion

Multi-modal bioelectrical signal fusion analysis based on different acquisition devices and scene settings: Overview, challenges, and novel orientation

Fusing DTI and fMRI data: A survey of methods and applications

Advancing healthcare through multimodal data fusion: a comprehensive review of techniques and applications

Multimodal Contrastive Learning for Brain-Machine Fusion: from Brain-in-the-loop Modeling to Brain-out-of-the-loop Application

Advances in multimodality data fusion in neuroimaging

Interpretable Multimodal Fusion Networks Reveal Mechanisms of Brain Cognition

Multimodal and multicontrast image fusion via deep generative models

A Review of Multivariate Methods in Brain Imaging Data Fusion

Multimodal neuroimaging computing: a review of the applications in neuropsychiatric disorders

A Comprehensive Survey on Deep Learning Multi-Modal Fusion: Methods, Technologies and Applications