Abstract:Medical imaging is indispensable for accurate diagnosis and effective treatment, with modalities like MRI and CT providing diverse yet complementary information. Traditional image fusion methods, while essential in consolidating information from multiple modalities, often suffer from poor image quality and loss of crucial details due to inadequate handling of semantic information and limited feature extraction capabilities. This paper introduces a novel medical image fusion technique leveraging unsupervised image segmentation to enhance the semantic understanding of the fusion process. The proposed method, named DUSMIF, employs a multi-branch, multi-scale deep learning architecture that integrates advanced attention mechanisms to refine the feature extraction and fusion processes. An innovative approach that utilizes unsupervised image segmentation to extract semantic information is introduced, which is then integrated into the fusion process. This not only enhances the semantic relevance of the fused images but also improves the overall fusion quality. The paper proposes a sophisticated network structure that extracts and fuses features at multiple scales and across multiple branches. This structure is designed to capture a comprehensive range of image details and contextual information, significantly improving the fusion outcomes. Multiple attention mechanisms are incorporated to selectively emphasize important features and integrate them effectively across different modalities and scales. This approach ensures that the fused images maintain high quality and detail fidelity. A joint loss function combining content loss, structural similarity loss, and semantic loss is formulated. This function not only guides the network in preserving image brightness and texture but also ensures that the fused image closely resembles the source images in both content and structure. The proposed method demonstrates superior performance over existing fusion techniques in objective assessments and subjective evaluations, confirming its effectiveness in enhancing the diagnostic utility of fused medical images.

WAE-TLDN: self-supervised fusion for multimodal medical images via a weighted autoencoder and a tensor low-rank decomposition network

Multi-modal medical image fusion based on densely-connected high-resolution CNN and hybrid transformer

MDC-RHT: Multi-Modal Medical Image Fusion via Multi-Dimensional Dynamic Convolution and Residual Hybrid Transformer

Multimodal medical image fusion using convolutional neural network and extreme learning machine

Multi-modal Medical Neurological Image Fusion using Wavelet Pooled Edge Preserving Autoencoder

Dual Low-Rank Multimodal Fusion

LatLRR-FCNs: Latent Low-Rank Representation With Fully Convolutional Networks for Medical Image Fusion

A New Multimodal Medical Image Fusion based on Laplacian Autoencoder with Channel Attention

LRFNet: A real-time medical image fusion method guided by detail information

Multimodal Medical Image Fusion: The Perspective of Deep Learning

EMOST: A dual-branch hybrid network for medical image fusion via efficient model module and sparse transformer

A multibranch and multiscale neural network based on semantic perception for multimodal medical image fusion

An Improved Hybrid Network With a Transformer Module for Medical Image Fusion

Multimodal Image Fusion Via Self-Supervised Transformer

AMMNet: A multimodal medical image fusion method based on an attention mechanism and MobileNetV3

DFENet: A dual-branch feature enhanced network integrating transformers and convolutional feature learning for multimodal medical image fusion

Pulse Coupled Neural Network-Based Multimodal Medical Image Fusion via Guided Filtering and WSEML in NSCT Domain

Edge-Enhanced Dilated Residual Attention Network for Multimodal Medical Image Fusion

DDIFN: A Dual-discriminator Multi-modal Medical Image Fusion Network

Feature extraction of multimodal medical image fusion using novel deep learning and contrast enhancement method

Weakly paired multimodal fusion using multilayer extreme learning machine