Abstract:BACKGROUND:Multi-modal learning is widely adopted to learn the latent complementary information between different modalities in multi-modal medical image segmentation tasks. Nevertheless, the traditional multi-modal learning methods require spatially well-aligned and paired multi-modal images for supervised training, which cannot leverage unpaired multi-modal images with spatial misalignment and modality discrepancy. For training accurate multi-modal segmentation networks using easily accessible and low-cost unpaired multi-modal images in clinical practice, unpaired multi-modal learning has received comprehensive attention recently.PURPOSE:Existing unpaired multi-modal learning methods usually focus on the intensity distribution gap but ignore the scale variation problem between different modalities. Besides, within existing methods, shared convolutional kernels are frequently employed to capture common patterns in all modalities, but they are typically inefficient at learning global contextual information. On the other hand, existing methods highly rely on a large number of labeled unpaired multi-modal scans for training, which ignores the practical scenario when labeled data is limited. To solve the above problems, we propose a modality-collaborative convolution and transformer hybrid network (MCTHNet) using semi-supervised learning for unpaired multi-modal segmentation with limited annotations, which not only collaboratively learns modality-specific and modality-invariant representations, but also could automatically leverage extensive unlabeled scans for improving performance.METHODS:We make three main contributions to the proposed method. First, to alleviate the intensity distribution gap and scale variation problems across modalities, we develop a modality-specific scale-aware convolution (MSSC) module that can adaptively adjust the receptive field sizes and feature normalization parameters according to the input. Secondly, we propose a modality-invariant vision transformer (MIViT) module as the shared bottleneck layer for all modalities, which implicitly incorporates convolution-like local operations with the global processing of transformers for learning generalizable modality-invariant representations. Third, we design a multi-modal cross pseudo supervision (MCPS) method for semi-supervised learning, which enforces the consistency between the pseudo segmentation maps generated by two perturbed networks to acquire abundant annotation information from unlabeled unpaired multi-modal scans.RESULTS:Extensive experiments are performed on two unpaired CT and MR segmentation datasets, including a cardiac substructure dataset derived from the MMWHS-2017 dataset and an abdominal multi-organ dataset consisting of the BTCV and CHAOS datasets. Experiment results show that our proposed method significantly outperforms other existing state-of-the-art methods under various labeling ratios, and achieves a comparable segmentation performance close to single-modal methods with fully labeled data by only leveraging a small portion of labeled data. Specifically, when the labeling ratio is 25%, our proposed method achieves overall mean DSC values of 78.56% and 76.18% in cardiac and abdominal segmentation, respectively, which significantly improves the average DSC value of two tasks by 12.84% compared to single-modal U-Net models.CONCLUSIONS:Our proposed method is beneficial for reducing the annotation burden of unpaired multi-modal medical images in clinical applications.

Unpaired Multi-Modal Tumor Segmentation with Structure Adaptation

Mutual Information-Based Graph Co-Attention Networks for Multimodal Prior-Guided Magnetic Resonance Imaging Segmentation

Tumor Segmentation in Weakly Paired Anatomical and Functional MRI Images with Multimodal Information Fusion

M3AE: Multimodal Representation Learning for Brain Tumor Segmentation with Missing Modalities

Cross-modality Synthesis Aiding Lung Tumor Segmentation on Multi-Modal MRI Images

Learning multi-modal brain tumor segmentation from privileged semi-paired MRI images with curriculum disentanglement learning

MedMAP: Promoting Incomplete Multi-modal Brain Tumor Segmentation with Alignment

Multimodality-Assisted Semi-Supervised Brain Tumor Segmentation in Nondominant Modality Based on Consistency Learning

Modality-Pairing Learning for Brain Tumor Segmentation

Enhancing Incomplete Multi-modal Brain Tumor Segmentation with Intra-modal Asymmetry and Inter-modal Dependency

Robust Multimodal Brain Tumor Segmentation via Feature Disentanglement and Gated Fusion

Multi-task Learning for Segmentation and Classification of Tumors in 3D Automated Breast Ultrasound Images

A bidirectional multilayer contrastive adaptation network with anatomical structure preservation for unpaired cross-modality medical image segmentation

Unified Generative Adversarial Networks for Multimodal Segmentation from Unpaired 3D Medical Images.

Multi-Modal Pulmonary Mass Segmentation Network Based on Cross-Modal Spatial Alignment

Anatomical Consistency Distillation and Inconsistency Synthesis for Brain Tumor Segmentation with Missing Modalities

Pancreatic cancer segmentation in unregistered multi-parametric MRI with adversarial learning and multi-scale supervision

Robust Semi-supervised Multimodal Medical Image Segmentation via Cross Modality Collaboration

Multi-Modal Tumor Segmentation with Deformable Aggregation and Uncertain Region Inpainting

A Comprehensive Multi-modal Domain Adaptative Aid Framework for Brain Tumor Diagnosis.

A Modality-Collaborative Convolution and Transformer Hybrid Network for Unpaired Multi-Modal Medical Image Segmentation with Limited Annotations