Abstract:Multi-modality (MM) semi-supervised learning (SSL) based medical image segmentation has recently gained increasing attention for its ability to utilize MM data and reduce reliance on labeled images. However, current methods face several challenges: (1) Complex network designs hinder scalability to scenarios with more than two modalities. (2) Focusing solely on modality-invariant representation while neglecting modality-specific features, leads to incomplete MM learning. (3) Leveraging unlabeled data with generative methods can be unreliable for SSL. To address these problems, we propose Double Bank Dual Consistency (DBDC), a novel MM-SSL approach for medical image segmentation. To address challenge (1), we propose a modality all-in-one segmentation network that accommodates data from any number of modalities, removing the limitation on modality count. To address challenge (2), we design two learnable plug-in banks, Modality-Level Modulation bank (MLMB) and Modality-Level Prototype (MLPB) bank, to capture both modality-invariant and modality-specific knowledge. These banks are updated using our proposed Modality Prototype Contrastive Learning (MPCL). Additionally, we design Modality Adaptive Weighting (MAW) to dynamically adjust learning weights for each modality, ensuring balanced MM learning as different modalities learn at different rates. Finally, to address challenge (3), we introduce a Dual Consistency (DC) strategy that enforces consistency at both the image and feature levels without relying on generative methods. We evaluate our method on a 2-to-4 modality segmentation task using three open-source datasets, and extensive experiments show that our method outperforms state-of-the-art approaches.

What problem does this paper attempt to address?

This paper attempts to solve several key problems in multi - modality (MM) semi - supervised learning (SSL) in medical image segmentation: 1. **Complexity of Expansion to More Modalities**: Existing methods are usually only applicable to two modalities and rely on multiple independent or complex network designs, making it difficult to expand to cases with more than two modalities. 2. **Incomplete Learning of Modality - Invariant and Modality - Specific Features**: Existing methods often only focus on modality - invariant representations and ignore modality - specific features, resulting in incomplete multi - modality learning. 3. **Unreliability of Generation Methods in SSL**: Using generation methods to utilize unlabeled data may introduce noise and reduce the reliability of SSL. To solve these problems, the authors propose a new method named Double Bank Dual Consistency (DBDC). Specifically: - **Modality All - in - One Network**: A unified network structure that can handle an arbitrary number of modalities is proposed, which is based on the standard U - Net architecture and eliminates the need for modality registration or additional network structures. - **Modality - Level Modulation Bank (MLMB) and Modality - Level Prototype Bank (MLPB)**: Through these two plug - in modules, modality - invariant features and modality - specific features are learned respectively to ensure comprehensive multi - modality feature extraction. - **Modality Adaptive Weighting (MAW)**: An adaptive weight adjustment mechanism is designed to dynamically balance the learning speeds of different modalities and ensure a stable learning process. - **Dual Consistency (DC) Strategy**: Consistency constraints are imposed at both the image and feature levels without relying on generation methods, improving the reliability of SSL training. Through these innovations, the DBDC method aims to improve the effectiveness of multi - modality semi - supervised medical image segmentation, especially when dealing with multi - modality data.

Double Banking on Knowledge: Customized Modulation and Prototypes for Multi-Modality Semi-supervised Medical Image Segmentation

Robust Semi-supervised Multimodal Medical Image Segmentation via Cross Modality Collaboration

Dual-Decoder Consistency via Pseudo-Labels Guided Data Augmentation for Semi-Supervised Medical Image Segmentation

Multi-Consistency Training for Semi-Supervised Medical Image Segmentation

Dual-Task Mutual Learning for Semi-Supervised Medical Image Segmentation

Towards Cross-modality Medical Image Segmentation with Online Mutual Knowledge Distillation

Semisupervised Medical Image Segmentation through Prototype-Based Mutual Consistency Learning

Semi-supervised Medical Image Segmentation through Dual-task Consistency

Robust Semi-supervised 3D Medical Image Segmentation with Diverse Joint-task Learning and Decoupled Inter-student Learning

MVPCL: multi-view prototype consistency learning for semi-supervised medical image segmentation

Multi-modal contrastive mutual learning and pseudo-label re-learning for semi-supervised medical image segmentation

Dual-consistency guidance semi-supervised medical image segmentation with low-level detail feature augmentation

Semi-Supervised Medical Image Segmentation Via Multi-Group Network with Consistency among Heterogeneous Loss Supervision

MulModSeg: Enhancing Unpaired Multi-Modal Medical Image Segmentation with Modality-Conditioned Text Embedding and Alternating Training

Dual consistency regularization with subjective logic for semi-supervised medical image segmentation

Semi-Supervised 3D Medical Image Segmentation Based on Dual-Task Consistent Joint Learning and Task-Level Regularization.

MASS: Modality-collaborative semi-supervised segmentation by exploiting cross-modal consistency from unpaired CT and MRI images

S&D Messenger: Exchanging Semantic and Domain Knowledge for Generic Semi-Supervised Medical Image Segmentation

Mutually aided uncertainty incorporated Dual Consistency Regularization with Pseudo Label for Semi-Supervised Medical Image Segmentation

Unpaired Multi-modal Segmentation via Knowledge Distillation

Modality-Agnostic Learning for Medical Image Segmentation Using Multi-modality Self-distillation