Abstract:Multi-modal imaging analyses of dosed tissue samples can provide more comprehensive insight into the effects of a therapeutically active compound on a target tissue compared to single-modal imaging. For example, simultaneous spatial mapping of pharmaceutical compounds and endogenous macromolecule receptors is difficult to achieve in a single imaging experiment. Herein, we present a multi-modal workflow combining imaging mass spectrometry with immunohistochemistry (IHC) fluorescence imaging and brightfield microscopy imaging. Imaging mass spectrometry enables direct mapping of pharmaceutical compounds and metabolites, IHC fluorescence imaging can visualize large proteins, and brightfield microscopy imaging provides tissue morphology information. Single-cell resolution images are generally difficult to acquire using imaging mass spectrometry, but are readily acquired with IHC fluorescence and brightfield microscopy imaging. Spatial sharpening of mass spectrometry images would thus allow for higher fidelity co-registration with higher resolution microscopy images. Imaging mass spectrometry spatial resolution can be predicted to a finer value via a computational image fusion workflow, which models the relationship between the intensity values in the mass spectrometry image and the features of a high spatial resolution microscopy image. As a proof of concept, our multi-modal workflow was applied to brain tissue extracted from a Sprague Dawley rat dosed with a kratom alkaloid, corynantheidine. Four candidate mathematical models including linear regression, partial least squares regression (PLS), random forest regression, and two-dimensional convolutional neural network (2-D CNN), were tested. The random forest and 2-D CNN models most accurately predicted the intensity values at each pixel as well as the overall patterns of the mass spectrometry images, while also providing the best spatial resolution enhancements. Herein, image fusion enabled predicted mass spectrometry images of corynantheidine, GABA, and glutamine to approximately 2.5 μm spatial resolutions, a significant improvement compared to the original images acquired at 25 μm spatial resolution. The predicted mass spectrometry images were then co-registered with an H&E image and IHC fluorescence image of the μ- opioid receptor to assess co-localization of corynantheidine with brain cells. Our study also provides insight into the different evaluation parameters to consider when utilizing image fusion for biological applications.

Multimodal Image Fusion Offers Better Spatial Resolution for Mass Spectrometry Imaging

Hyperspectral and Multispectral Image Fusion Based on Deep Attention Network.

Enhancing Spatial Resolution in Tandem Mass Spectrometry Ion/Ion Reaction Imaging Experiments through Image Fusion

Multimodal Image Fusion Workflow Incorporating MALDI Imaging Mass Spectrometry and Microscopy for the Study of Small Pharmaceutical Compounds

Multispectral and Hyperspectral Image Fusion by MS/HS Fusion Net.

Adaptive Multimodal Image Fusion with a Deep Pyramidal Residual Learning Network

Coupled Matrix Factorization Constrained Deep Hyperspectral and Multispectral Image Fusion

A multi-modal image fusion workflow incorporating MALDI imaging mass spectrometry and microscopy for the study of small pharmaceutical compounds

Multimodal Super-Resolution Reconstruction of Infrared and Visible Images Via Deep Learning

A super-resolution strategy for mass spectrometry imaging via transfer learning

Multimodal Medical Image Fusion: The Perspective of Deep Learning

Multimodal medical image fusion using convolutional neural network and extreme learning machine

Hyperspectral and Multispectral Image Fusion via Deep Two-Branches Convolutional Neural Network

Medical Image Segmentation Based on Multi-Modal Convolutional Neural Network: Study on Image Fusion Schemes

CIRF: Coupled Image Reconstruction and Fusion Strategy for Deep Learning Based Multi-Modal Image Fusion

A model-guided deep convolutional sparse coding network for hyperspectral and multispectral image fusion

Multimodal medical image fusion and classification using deep learning techniques

MDC-RHT: Multi-Modal Medical Image Fusion via Multi-Dimensional Dynamic Convolution and Residual Hybrid Transformer

Application of Multimodal Fusion Deep Learning Model in Disease Recognition

MSI-DTrans: A Multi-Focus Image Fusion Using Multilayer Semantic Interaction and Dynamic Transformer

$M^{2}$Fusion: Bayesian-based Multimodal Multi-level Fusion on Colorectal Cancer Microsatellite Instability Prediction