Deep evidential fusion with uncertainty quantification and contextual discounting for multimodal medical image segmentation

Ling Huang,Su Ruan,Pierre Decazes,Thierry Denoeux

2024-08-19

Abstract:Single-modality medical images generally do not contain enough information to reach an accurate and reliable diagnosis. For this reason, physicians generally diagnose diseases based on multimodal medical images such as, e.g., PET/CT. The effective fusion of multimodal information is essential to reach a reliable decision and explain how the decision is made as well. In this paper, we propose a fusion framework for multimodal medical image segmentation based on deep learning and the Dempster-Shafer theory of evidence. In this framework, the reliability of each single modality image when segmenting different objects is taken into account by a contextual discounting operation. The discounted pieces of evidence from each modality are then combined by Dempster's rule to reach a final decision. Experimental results with a PET-CT dataset with lymphomas and a multi-MRI dataset with brain tumors show that our method outperforms the state-of-the-art methods in accuracy and reliability.

Image and Video Processing,Computer Vision and Pattern Recognition

What problem does this paper attempt to address?

The paper attempts to address the problem of how to effectively fuse information from different imaging modalities in multimodal medical image segmentation, quantify segmentation uncertainty during the fusion process, and improve segmentation accuracy. Specifically, the authors propose a fusion framework based on deep neural networks and evidence theory (particularly Dempster-Shafer theory), aiming to improve the segmentation performance of multimodal medical images by considering the relative reliability of each modality's information. Additionally, this method provides insights into the contribution of each imaging modality to the segmentation process.

The main contributions of the paper include:
1. Proposing a new multimodal medical image fusion architecture that includes feature extraction, evidence mapping, and combination modules.
2. Integrating mechanisms within this architecture for (i) quantifying segmentation uncertainty using Dempster-Shafer mass functions, (ii) adjusting these mass functions through contextual discounting to account for the relative reliability of each imaging modality, and (iii) combining the adjusted mass functions from different sources to make the final segmentation decision.
3. Introducing an improved two-part loss function that allows for optimizing the segmentation performance of each individual source modality as well as the overall performance of the combined decision.
4. Demonstrating through extensive experiments on two real medical image datasets that the proposed decision-level fusion scheme enhances segmentation reliability and quality compared to other methods utilizing different image modalities.
5. Showing that the learned reliability coefficients provide insights into the contribution of each imaging modality in the segmentation process.

Overall, this study aims to address the challenges in multimodal medical image segmentation by combining the strengths of deep learning and evidence theory, particularly in handling uncertainty and improving segmentation accuracy.