Abstract:With a wide array of multi-modal, multi-protocol, and multi-scale biomedical data available for disease diagnosis and prognosis, there is a need for quantitative tools to combine such varied channels of information, especially imaging and non-imaging data (e.g. spectroscopy, proteomics). The major problem in such quantitative data integration lies in reconciling the large spread in the range of dimensionalities and scales across the different modalities. The primary goal of quantitative data integration is to build combined meta-classifiers; however these efforts are thwarted by challenges in (1) homogeneous representation of the data channels, (2) fusing the attributes to construct an integrated feature vector, and (3) the choice of learning strategy for training the integrated classifier. In this paper, we seek to (a) define the characteristics that guide the 4 independent methods for quantitative data fusion that use the idea of a meta-space for building integrated multi-modal, multi-scale meta-classifiers, and (b) attempt to understand the key components which allowed each method to succeed. These methods include (1) Generalized Embedding Concatenation (GEC), (2) Consensus Embedding (CE), (3) Semi-Supervised Multi-Kernel Graph Embedding (SeSMiK), and (4) Boosted Embedding Combination (BEC). In order to evaluate the optimal scheme for fusing imaging and non-imaging data, we compared these 4 schemes for the problems of combining (a) multi-parametric MRI with spectroscopy for prostate cancer (CaP) diagnosis in vivo, and (b) histological image with proteomic signatures (obtained via mass spectrometry) for predicting prognosis in CaP patients. The kernel combination approach (SeSMiK) marginally outperformed the embedding combination schemes. Additionally, intelligent weighting of the data channels (based on their relative importance) appeared to outperform unweighted strategies. All 4 strategies easily outperformed a naïve decision fusion approach, suggesting that data integration methods will play an important role in the rapidly emerging field of integrated diagnostics and personalized healthcare.

Multimodal data integration and cross-modal querying via orchestrated approximate message passing

Simultaneous Covariance Inference for Multimodal Integrative Analysis.

Multimodal integration-a statistical view

CMCI: A Robust Multimodal Fusion Method for Spiking Neural Networks

Multimodal Neuroimaging Data Integration and Pathway Analysis

Jointly Optimizing Sensing Pipelines for Multimodal Mixed Reality Interaction

Integration of Multimodal Data for Deciphering Brain Disorders.

MIDAS: a deep generative model for mosaic integration and knowledge transfer of single-cell multimodal data

Integrating Heterogeneous Datasets by Using Multimodal Deep Learning

Multimodal Data Fusion in High-Dimensional Heterogeneous Datasets via Generative Models

MULTI-MODAL DATA FUSION SCHEMES FOR INTEGRATED CLASSIFICATION OF IMAGING AND NON-IMAGING BIOMEDICAL DATA

Orthogonalized Kernel Debiased Machine Learning for Multimodal Data Analysis

Optimal Multimodal Fusion for Multimedia Data Analysis

Similarity-Based Multimodal Regression

Multimodal joint deconvolution and integrative signature selection in proteomics

Incomplete Multimodal Learning for Complex Brain Disorders Prediction

Deep Fusion Of Heterogeneous Sensor Data

Deep Multimodal Data Fusion

Quantifying & Modeling Multimodal Interactions: An Information Decomposition Framework

Integrating Multimodal Data for Joint Generative Modeling of Complex Dynamics

Statistical Inferences for Complex Dependence of Multimodal Imaging Data