Abstract:Radiologists must utilize multiple modal images for tumor segmentation and diagnosis due to the limitations of medical imaging and the diversity of tumor signals. This leads to the development of multimodal learning in segmentation. However, the redundancy among modalities creates challenges for existing subtraction-based joint learning methods, such as misjudging the importance of modalities, ignoring specific modal information, and increasing cognitive load. These thorny issues ultimately decrease segmentation accuracy and increase the risk of overfitting. This paper presents the complementary information mutual learning (CIML) framework, which can mathematically model and address the negative impact of inter-modal redundant information. CIML adopts the idea of addition and removes inter-modal redundant information through inductive bias-driven task decomposition and message passing-based redundancy filtering. CIML first decomposes the multimodal segmentation task into multiple subtasks based on expert prior knowledge, minimizing the information dependence between modalities. Furthermore, CIML introduces a scheme in which each modality can extract information from other modalities additively through message passing. To achieve non-redundancy of extracted information, the redundant filtering is transformed into complementary information learning inspired by the variational information bottleneck. The complementary information learning procedure can be efficiently solved by variational inference and cross-modal spatial attention. Numerical results from the verification task and standard benchmarks indicate that CIML efficiently removes redundant information between modalities, outperforming SOTA methods regarding validation accuracy and segmentation effect.

What problem does this paper attempt to address?

The problem that this paper attempts to solve is how to effectively eliminate redundant information between modalities, improve segmentation accuracy, and reduce the risk of over - fitting in multi - modal medical image segmentation. Specifically, the paper points out that in multi - modal medical image analysis, due to the limitations of different imaging techniques and the diversity of tumor signals, radiologists need to use multi - modal medical images for tumor segmentation and diagnosis. However, the redundant information between modalities poses challenges to existing joint learning methods based on subtraction, such as misjudging the importance of modalities, ignoring information of specific modalities, and increasing cognitive load, etc. These problems ultimately reduce the accuracy of segmentation and increase the risk of over - fitting. To solve the above problems, the paper proposes the Complementary Information Mutual Learning (CIML) framework. CIML deals with the negative impact of redundant information between modalities through mathematical modeling, adopts the idea of addition, and removes redundant information between modalities through inductive - bias - driven task decomposition and message - passing - based redundancy filtering. CIML first decomposes the multi - modal segmentation task into multiple sub - tasks according to expert prior knowledge to minimize the information dependence between modalities. In addition, CIML introduces a mechanism that enables each modality to extract information from other modalities through message passing, and at the same time, through the complementary information learning process inspired by the Variational Information Bottleneck, ensures that the extracted information is non - redundant. This process can be efficiently solved through variational inference and cross - modal spatial attention. In summary, CIML aims to minimize the redundant information between modalities on which the algorithm depends during the segmentation process through two mechanisms, task decomposition and redundancy filtering, thereby improving the efficiency and accuracy of multi - modal medical image segmentation.

Complementary Information Mutual Learning for Multimodality Medical Image Segmentation

Mutual Information-Based Graph Co-Attention Networks for Multimodal Prior-Guided Magnetic Resonance Imaging Segmentation

Weakly-Interactive-Mixed Learning: Less Labelling Cost for Better Medical Image Segmentation.

Mmformer: Multimodal Medical Transformer for Incomplete Multimodal Learning of Brain Tumor Segmentation

Multimodal Priors Guided Segmentation of Liver Lesions in MRI Using Mutual Information Based Graph Co-Attention Networks.

Robust Semi-supervised Multimodal Medical Image Segmentation via Cross Modality Collaboration

Modality-aware Mutual Learning for Multi-modal Medical Image Segmentation

Multi-modal contrastive mutual learning and pseudo-label re-learning for semi-supervised medical image segmentation

M3AE: Multimodal Representation Learning for Brain Tumor Segmentation with Missing Modalities

Cross-View Mutual Learning for Semi-Supervised Medical Image Segmentation

Cross-Modal Conditioned Reconstruction for Language-guided Medical Image Segmentation

Unpaired Dual-Modal Image Complementation Learning for Single-Modal Medical Image Segmentation

Robust Divergence Learning for Missing-Modality Segmentation

MulModSeg: Enhancing Unpaired Multi-Modal Medical Image Segmentation with Modality-Conditioned Text Embedding and Alternating Training

Multimodality-Assisted Semi-Supervised Brain Tumor Segmentation in Nondominant Modality Based on Consistency Learning

Correlation-Aware Mutual Learning for Semi-supervised Medical Image Segmentation

Cross-Modal Information Maximization for Medical Imaging: CMIM

MASS: Modality-collaborative semi-supervised segmentation by exploiting cross-modal consistency from unpaired CT and MRI images

Robust Multimodal Brain Tumor Segmentation via Feature Disentanglement and Gated Fusion

Deep Class-Specific Affinity-Guided Convolutional Network for Multimodal Unpaired Image Segmentation

Mutually enhanced multi-view information learning for segmentation of lung tumor in CT images