Abstract:Abstract Background In medical diagnosis of brain, the role of multi-modal medical image fusion is becoming more prominent. Among them, there is no lack of filtering layered fusion and newly emerging deep learning algorithms. The former has a fast fusion speed but the fusion image texture is blurred; the latter has a better fusion effect but requires higher machine computing capabilities. Therefore, how to find a balanced algorithm in terms of image quality, speed and computing power is still the focus of all scholars. Methods We built an end-to-end Hahn-PCNN-CNN. The network is composed of feature extraction module, feature fusion module and image reconstruction module. We selected 8000 multi-modal brain medical images downloaded from the Harvard Medical School website to train the feature extraction layer and image reconstruction layer to enhance the network’s ability to reconstruct brain medical images. In the feature fusion module, we use the moments of the feature map combined with the pulse-coupled neural network to reduce the information loss caused by convolution in the previous fusion module and save time. Results We choose eight sets of registered multi-modal brain medical images in four diease to verify our model. The anatomical structure images are from MRI and the functional metabolism images are SPECT and 18F-FDG. At the same time, we also selected eight representative fusion models as comparative experiments. In terms of objective quality evaluation, we select six evaluation metrics in five categories to evaluate our model. Conclusions The fusion image obtained by our model can retain the effective information in source images to the greatest extent. In terms of image fusion evaluation metrics, our model is superior to other comparison algorithms. In terms of time computational efficiency, our model also performs well. In terms of robustness, our model is very stable and can be generalized to multi-modal image fusion of other organs.

Multi-modal Medical Image Fusion Via Multi-Dictionary and Truncated Huber Filtering

MDC-RHT: Multi-Modal Medical Image Fusion via Multi-Dimensional Dynamic Convolution and Residual Hybrid Transformer

Joint medical image fusion, denoising and enhancement via discriminative low-rank sparse dictionaries learning

Multi-Modal Medical Image Fusion Based on FusionNet in YIQ Color Space

Multi-modal medical image fusion based on densely-connected high-resolution CNN and hybrid transformer

An Integrated Dictionary-Learning Entropy-Based Medical Image Fusion Framework

Joint sparse model with coupled dictionary for medical image fusion

Multi-Scale Fusion Global Feature Extraction Network for Multi-Modal Medical Image Fusion

Simultaneous Tri-Modal Medical Image Fusion and Super-Resolution using Conditional Diffusion Model

Medical image fusion based on sparse representation and neighbor energy activity

Hahn-PCNN-CNN: an end-to-end multi-modal brain medical image fusion framework useful for clinical diagnosis

Multimodal medical image fusion based on interval gradients and convolutional neural networks

Multi-Sensor Medical-Image Fusion Technique Based on Embedding Bilateral Filter in Least Squares and Salient Detection

Multi-scale Feature Fusion Convolutional Neural Network for Multi-Modal Medical Image Fusion.

MRI Image Fusion Based on Optimized Dictionary Learning and Binary Map Refining in Gradient Domain

Medical image fusion based on improved multi-scale morphology gradient-weighted local energy and visual saliency map

An Image Fusion Method Based on Sparse Representation and Sum Modified-Laplacian in NSCT Domain

Multimodal MRI Volumetric Data Fusion With Convolutional Neural Networks

Adaptive Decomposition Method for Multi-Modal Medical Image Fusion.

Three-Dimensional Medical Image Fusion with Deformable Cross-Attention

A Medical Image Fusion Method Based on Convolutional Neural Networks