Abstract:Introduction: Tissue-based sampling and diagnosis are defined as the extraction of information from certain limited spaces and its diagnostic significance of a certain object. Pathologists deal with issues related to tumor heterogeneity since analyzing a single sample does not necessarily capture a representative depiction of cancer, and a tissue biopsy usually only presents a small fraction of the tumor. Many multiplex tissue imaging platforms (MTIs) make the assumption that tissue microarrays (TMAs) containing small core samples of 2-dimensional (2D) tissue sections are a good approximation of bulk tumors although tumors are not 2D. However, emerging whole slide imaging (WSI) or 3D tumor atlases that use MTIs like cyclic immunofluorescence (CyCIF) strongly challenge this assumption. In spite of the additional insight gathered by measuring the tumor microenvironment in WSI or 3D, it can be prohibitively expensive and time-consuming to process tens or hundreds of tissue sections with CyCIF. Even when resources are not limited, the criteria for region of interest (ROI) selection in tissues for downstream analysis remain largely qualitative and subjective as stratified sampling requires the knowledge of objects and evaluates their features. Despite the fact TMAs fail to adequately approximate whole tissue features, a theoretical subsampling of tissue exists that can best represent the tumor in the whole slide image. Methods: To address these challenges, we propose deep learning approaches to learn multi-modal image translation tasks from two aspects: 1) generative modeling approach to reconstruct 3D CyCIF representation and 2) co-embedding CyCIF image and Hematoxylin and Eosin (H&E) section to learn multi-modal mappings by a cross-domain translation for minimum representative ROI selection. Results and discussion: We demonstrate that generative modeling enables a 3D virtual CyCIF reconstruction of a colorectal cancer specimen given a small subset of the imaging data at training time. By co-embedding histology and MTI features, we propose a simple convex optimization for objective ROI selection. We demonstrate the potential application of ROI selection and the efficiency of its performance with respect to cellular heterogeneity.

Multi-view Multi-modal Feature Embedding for Endomicroscopy Mosaic Classification

Context aware decision support in neurosurgical oncology based on an efficient classification of endomicroscopic data

Deep Graph-Based Multimodal Feature Embedding for Endomicroscopy Image Retrieval

Unsupervised Feature Learning for Endomicroscopy Image Retrieval

Weakly Supervised Representation Learning for Endomicroscopy Image Analysis.

Transfer Recurrent Feature Learning for Endomicroscopy Image Recognition.

Robust Mosaicing of Endomicroscopic Videos Via Context-Weighted Correlation Ratio.

Toward Robust Histology-Prior Embedding for Endomicroscopy Image Classification

Multi-scale discriminative network for prostate cancer lesion segmentation in multiparametric MR images

Clustering-Guided Twin Contrastive Learning for Endomicroscopy Image Classification

MTECC: A Multi-Task Learning Framework for Esophageal Cancer Analysis

Robust Endoscopic Image Mosaicking Via Fusion of Multimodal Estimation

Endoscopic Video Manifolds for Targeted Optical Biopsy

High-Resolution Image Processing of Probe-Based Confocal Laser Endomicroscopy Based on Multistage Neural Networks and Cross-Channel Attention Module

A Multimodal Approach For Endoscopic VCE Image Classification Using BiomedCLIP-PubMedBERT

3D multiplexed tissue imaging reconstruction and optimized region of interest (ROI) selection through deep learning model of channels embedding

A Global Approach for Automatic Fibroscopic Video Mosaicing in Minimally Invasive Diagnosis

XRelevanceCAM: towards explainable tissue characterization with improved localisation of pathological structures in probe-based confocal laser endomicroscopy

Electromagnetic Tracking of Handheld High-Resolution Endomicroscopy Probes to Assist with Real-Time Video Mosaicking

A Multi-Stage Fusion Strategy for Multi-Scale GLCM-CNN Model in Differentiating Malignant from Benign Polyps.

A Robust Mosaicing Method with Super-Resolution for Optical Medical Images.