Abstract:Background: Accurate segmentation of pancreatic cancer tumors using positron emission tomography/computed tomography (PET/CT) multimodal images is crucial for clinical diagnosis and prognosis evaluation. However, deep learning methods for automated medical image segmentation require a substantial amount of manually labeled data, making it time-consuming and labor-intensive. Moreover, addition or simple stitching of multimodal images leads to redundant information, failing to fully exploit the complementary information of multimodal images. Therefore, we developed a semisupervised multimodal network that leverages limited labeled samples and introduces a cross-fusion and mutual information minimization (MIM) strategy for PET/CT 3D segmentation of pancreatic tumors. Methods: Our approach combined a cross multimodal fusion (CMF) module with a cross-attention mechanism. The complementary multimodal features were fused to form a multifeature set to enhance the effectiveness of feature extraction while preserving specific features of each modal image. In addition, we designed an MIM module to mitigate redundant high-level modal information and compute the latent loss of PET and CT. Finally, our method employed the uncertainty-aware mean teacher semi-supervised framework to segment regions of interest from PET/CT images using a small amount of labeled data and a large amount of unlabeled data. Results: We evaluated our combined MIM and CMF semisupervised segmentation network (MIM-CMFNet) on a private dataset of pancreatic cancer, yielding an average Dice coefficient of 73.14%, an average Jaccard index score of 60.56%, and an average 95% Hausdorff distance (95HD) of 6.30 mm. In addition, to verify the broad applicability of our method, we used a public dataset of head and neck cancer, yielding an average Dice coefficient of 68.71%, an average Jaccard index score of 57.72%, and an average 95HD of 7.88 mm. Conclusions: The experimental results demonstrate the superiority of our MIM-CMFNet over existing semisupervised techniques. Our approach can achieve a performance similar to that of fully supervised segmentation methods while significantly reducing the data annotation cost by 80%, suggesting it is highly practicable for clinical application.

Bimodal Segmentation and Classification of Endoscopic Ultrasonography Images for Solid Pancreatic Tumor.

Semisupervised 3D segmentation of pancreatic tumors in positron emission tomography/computed tomography images using a mutual information minimization and cross-fusion strategy

Automatic multi-tissue segmentation in pancreatic pathological images with selected multi-scale attention network

Attention-enhanced multiscale feature fusion network for pancreas and tumor segmentation

Ssm-Net: Semi-Supervised Multi-Task Network for Joint Lesion Segmentation and Classification from Pancreatic Eus Images

Application of a pyramid pooling Unet model with integrated attention mechanism and Inception module in pancreatic tumor segmentation

Semantic segmentation of pancreatic medical images by using convolutional neural network

Multi-scale Coarse-to-Fine Segmentation for Screening Pancreatic Ductal Adenocarcinoma

Robust Pancreatic Ductal Adenocarcinoma Segmentation with Multi-institutional Multi-phase Partially-Annotated CT Scans

Cross-Organ Domain Adaptive Neural Network for Pancreatic Endoscopic Ultrasound Image Segmentation

Deep learning‐based multimodal fusion network for segmentation and classification of breast cancers using B‐mode and elastography ultrasound images

Segmentation of pancreatic tumors based on multi‐scale convolution and channel attention mechanism in the encoder‐decoder scheme

Fully end-to-end deep-learning-based diagnosis of pancreatic tumors

Multimodal brain tumor image segmentation based on DenseNet

Accurate and Feasible Deep Learning Based Semi-Automatic Segmentation in CT for Radiomics Analysis in Pancreatic Neuroendocrine Neoplasms

Multimodal Brain Tumor Segmentation Based on an Intelligent UNET-LSTM Algorithm in Smart Hospitals

Automatic Pancreatic Ductal Adenocarcinoma Detection in Whole Slide Images Using Deep Convolutional Neural Networks

MFCNet: A multi-modal fusion and calibration networks for 3D pancreas tumor segmentation on PET-CT images

Multimodal ultrasound fusion network for differentiating between benign and malignant solid renal tumors

DenseNet model incorporating hybrid attention mechanisms and clinical features for pancreatic cystic tumor classification

3D Reconstruction-Oriented Fully Automatic Multi-Modal Tumor Segmentation by Dual Attention-Guided VNet