Abstract:Medical imaging is indispensable for accurate diagnosis and effective treatment, with modalities like MRI and CT providing diverse yet complementary information. Traditional image fusion methods, while essential in consolidating information from multiple modalities, often suffer from poor image quality and loss of crucial details due to inadequate handling of semantic information and limited feature extraction capabilities. This paper introduces a novel medical image fusion technique leveraging unsupervised image segmentation to enhance the semantic understanding of the fusion process. The proposed method, named DUSMIF, employs a multi-branch, multi-scale deep learning architecture that integrates advanced attention mechanisms to refine the feature extraction and fusion processes. An innovative approach that utilizes unsupervised image segmentation to extract semantic information is introduced, which is then integrated into the fusion process. This not only enhances the semantic relevance of the fused images but also improves the overall fusion quality. The paper proposes a sophisticated network structure that extracts and fuses features at multiple scales and across multiple branches. This structure is designed to capture a comprehensive range of image details and contextual information, significantly improving the fusion outcomes. Multiple attention mechanisms are incorporated to selectively emphasize important features and integrate them effectively across different modalities and scales. This approach ensures that the fused images maintain high quality and detail fidelity. A joint loss function combining content loss, structural similarity loss, and semantic loss is formulated. This function not only guides the network in preserving image brightness and texture but also ensures that the fused image closely resembles the source images in both content and structure. The proposed method demonstrates superior performance over existing fusion techniques in objective assessments and subjective evaluations, confirming its effectiveness in enhancing the diagnostic utility of fused medical images.

SIMFusion: A Semantic Information-Guided Modality-Specific Fusion Network for MR Images

Image Fusion of CT and MR with Sparse Representation in NSST Domain

MRSCFusion: Joint Residual Swin Transformer and Multiscale CNN for Unsupervised Multimodal Medical Image Fusion.

MSAIF-Net: A Multistage Spatial Attention-Based Invertible Fusion Network for MR Images.

A multibranch and multiscale neural network based on semantic perception for multimodal medical image fusion

Multimodal MRI Volumetric Data Fusion With Convolutional Neural Networks

A Semantic-Aware and Multi-Guided Network for Infrared-Visible Image Fusion

IGNFusion: An Unsupervised Information Gate Network for Multimodal Medical Image Fusion

RTFusion: A Multimodal Fusion Network with Significant Information Enhancement

Medical Image Segmentation Based on Multi-Modal Convolutional Neural Network: Study on Image Fusion Schemes

A novel approach for multimodal medical image fusion

A Semantic-based Medical Image Fusion Approach

M4FNet: Multimodal Medical Image Fusion Network Via Multi-Receptive-field and Multi-Scale Feature Integration.

Learning Cross-Modal Deep Representations for Multi-Modal MR Image Segmentation

A layer-wise fusion network incorporating self-supervised learning for multimodal MR image synthesis

MDC-RHT: Multi-Modal Medical Image Fusion via Multi-Dimensional Dynamic Convolution and Residual Hybrid Transformer

Multi-Modality Image Fusion and Object Detection Based on Semantic Information

MRASFusion: A Multi-Scale Residual Attention Infrared and Visible Image Fusion Network Based on Semantic Segmentation Guidance

Three-Dimensional Medical Image Fusion with Deformable Cross-Attention

SuperFusion: A Versatile Image Registration and Fusion Network with Semantic Awareness

High-Quality Fusion and Visualization for MR-PET Brain Tumor Images via Multi-Dimensional Features