Abstract:Gastric cancer (GC) is a prevalent malignancy worldwide, ranking as the fifth most common cancer with over 1 million new cases and 700 thousand deaths in 2020. Locally advanced gastric cancer (LAGC) accounts for approximately two-thirds of GC diagnoses, and neoadjuvant chemotherapy (NACT) has emerged as the standard treatment for LAGC. However, the effectiveness of NACT varies significantly among patients, with a considerable subset displaying treatment resistance. Ineffective NACT not only leads to adverse effects but also misses the optimal therapeutic window, resulting in lower survival rate. However, existing multimodal learning methods assume the availability of all modalities for each patient, which does not align with the reality of clinical practice. The limited availability of modalities for each patient would cause information loss, adversely affecting predictive accuracy. In this study, we propose an incomplete multimodal data integration framework for GC (iMD4GC) to address the challenges posed by incomplete multimodal data, enabling precise response prediction and survival analysis. Specifically, iMD4GC incorporates unimodal attention layers for each modality to capture intra-modal information. Subsequently, the cross-modal interaction layers explore potential inter-modal interactions and capture complementary information across modalities, thereby enabling information compensation for missing modalities. To evaluate iMD4GC, we collected three multimodal datasets for GC study: GastricRes (698 cases) for response prediction, GastricSur (801 cases) for survival analysis, and TCGA-STAD (400 cases) for survival analysis. The scale of our datasets is significantly larger than previous studies. The iMD4GC achieved impressive performance with an 80.2% AUC on GastricRes, 71.4% C-index on GastricSur, and 66.1% C-index on TCGA-STAD, significantly surpassing other compared methods.

Leveraging Multimodal Semantic Fusion for Gastric Cancer Screening via Hierarchical Attention Mechanism

A homogeneous ensemble method for predicting gastric cancer based on gastroscopy reports

SCNET: A Novel UGI Cancer Screening Framework Based on Semantic-Level Multimodal Data Fusion

Early gastric cancer detection and lesion segmentation based on deep learning and gastroscopic images

Improving diagnosis and outcome prediction of gastric cancer via multimodal learning using whole slide pathological images and gene expression

GCLDNet: Gastric Cancer Lesion Detection Network Combining Level Feature Aggregation and Attention Feature Fusion

Semantics-Aware Attention Guidance for Diagnosing Whole Slide Images

A Gastric Cancer Recognition Algorithm on Gastric Pathological Sections Based on Multistage Attention‐densenet

iMD4GC: Incomplete Multimodal Data Integration to Advance Precise Treatment Response Prediction and Survival Analysis for Gastric Cancer

Automatic Classification of Gastric Lesions in Gastroscopic Images Using a Lightweight Deep Learning Model with Attention Mechanism and Cost-Sensitive Learning

Gastric Polyp Detection Module Based on Improved Attentional Feature Fusion.

Broad learning system stacking with multi-scale attention for the diagnosis of gastric intestinal metaplasia

An Improved SSD-Based Gastric Cancer Detection Method

Identification of Upper Gastrointestinal Diseases during Screening Gastroscopy through Deep Convolutional Neural Network Algorithm

Multi-Scale Hybrid Vision Transformer for Learning Gastric Histology: AI-Based Decision Support System for Gastric Cancer Treatment

Fusion of medical imaging and electronic health records with attention and multi-head machanisms

Early Gastric Cancer Detection Based on the Combination of Convolutional Neural Network and Attention Mechanism

MLMSA: Multi-Level and Multi-Scale Attention for Lesion Detection in Endoscopy

[A novel attention fusion network-based multiple instance learning framework to automate diagnosis of chronic gastritis with multiple indicators].

Deep Learning-Based Multimodal Image Analysis for Cervical Cancer Detection.

Early diagnosis of gastric cancer based on deep learning combined with the spectral-spatial classification method.