Abstract:Multimodal image fusion aims to combine relevant information from images acquired with different sensors. In medical imaging, fused images play an essential role in both standard and automated diagnosis. In this paper, we propose a novel multimodal image fusion method based on coupled dictionary learning. The proposed method is general and can be employed for different medical imaging modalities. Unlike many current medical fusion methods, the proposed approach does not suffer from intensity attenuation nor loss of critical information. Specifically, the images to be fused are decomposed into coupled and independent components estimated using sparse representations with identical supports and a Pearson correlation constraint, respectively. An alternating minimization algorithm is designed to solve the resulting optimization problem. The final fusion step uses the max-absolute-value rule. Experiments are conducted using various pairs of multimodal inputs, including real MR-CT and MR-PET images. The resulting performance and execution times show the competitiveness of the proposed method in comparison with state-of-the-art medical image fusion methods.

What problem does this paper attempt to address?

The problem that this paper attempts to solve is how to effectively combine complementary information from different imaging techniques in multi - modal medical image fusion while avoiding intensity attenuation and loss of key information. Specifically, the paper proposes a new method based on Coupled Dictionary Learning (CDL) for multi - modal medical image fusion. This method can decompose the relevant and independent components in the image and solve the corresponding optimization problems through an optimization algorithm, and finally use the maximum absolute value rule for fusion. The paper aims to provide a general method applicable to different medical imaging modalities, such as MRI - CT, MRI - PET, etc., and shows its competitive advantages over existing methods in experiments. ### Main contributions of the paper: 1. **General decomposition model**: A general decomposition model applicable to various medical imaging modalities is proposed, which can retain important features in the input modalities and reduce the loss of key information. 2. **Coupled Dictionary Learning (CDL)**: A CDL method is proposed based on simultaneous sparse approximation to estimate relevant features. By assuming a common support set rather than the same sparse representation, relevant elements are allowed to be represented with different salience in each dictionary. 3. **Independent component estimation**: Pearson correlation constraints are used to estimate independent components to ensure low correlation. 4. **Alternating optimization method**: An alternating optimization method is designed to perform dictionary learning and image decomposition simultaneously. 5. **Fusion step**: The final fusion step combines direct summation and the maximum absolute value rule. ### Mathematical formulas: - **Minimization problem of CDL problem**: \[ \min_{D_1, D_2, A_1, A_2} \|D_1 A_1 - X_1\|_F^2+\|D_2 A_2 - X_2\|_F^2 \] where \(\|A_i\|_0\leq T\), \(\| [D_1]_t \|_2 = 1\), \(\| [D_2]_t \|_2 = 1\), \(\forall t, i\). - **Minimization problem of independent components**: \[ \min_{D_1, D_2, A_1, A_2, E_1, E_2} \sum_{i = 1,...,m} \sum_{j = 1,...,p} \phi([E_1]_{i,j}, [E_2]_{i,j})+\frac{\rho}{2} \sum_{k = 1,2} \|D_k A_k+E_k - X_k\|_F^2 \] where \(\phi([E_1]_{i,j}, [E_2]_{i,j})=\left( \frac{([E_1]_{i,j}-\mu_{1,j})([E_2]_{i,j}-\mu_{2,j})}{\sigma_{1,j} \sigma_{2,j}} \right)^2\), \(\mu_{1,j}\) and \(\sigma_{1,j}\) are local mean and standard deviation. ### Experimental results: The paper conducted experiments on multiple actual multi - modal image pairs, including MR - CT, MR - PET and MR - SPECT, etc. The experimental results show that the proposed method is superior to other existing techniques in the fusion of local intensity and texture information, especially in retaining modality - specific information. ### Conclusion: The paper proposes a multi - modal medical image fusion method based on coupled dictionary learning, which can effectively combine information from different imaging techniques, reduce information loss and improve the quality of the fused image. The experimental results verify the effectiveness and superiority of this method.

Coupled Feature Learning for Multimodal Medical Image Fusion

Multimodal image fusion via coupled feature learning

A novel approach for multimodal medical image fusion

Feature extraction of multimodal medical image fusion using novel deep learning and contrast enhancement method

Advancing multimodal medical image fusion: an adaptive image decomposition approach based on multilevel Guided filtering

Multimodal Medical Image Fusion: The Perspective of Deep Learning

Multimodal medical image fusion and classification using deep learning techniques

A novel dictionary learning approach for multi-modality medical image fusion.

Medical Image Fusion for Multiple Diseases Features Enhancement

Adaptive Multimodal Image Fusion with a Deep Pyramidal Residual Learning Network

Multimodal medical image fusion based on interval gradients and convolutional neural networks

Multimodal medical image fusion review: Theoretical background and recent advances

Multimodal Medical Image Fusion Based on Joint Bilateral Filter and Local Gradient Energy.

Multi-modality Medical Image Fusion Based on Separable Dictionary Learning and Gabor Filtering

Multimodality Medical Image Fusion Using Clustered Dictionary Learning in Non-Subsampled Shearlet Transform

Intelligent multimodal medical image fusion with deep guided filtering

Multimodal Image Fusion in Clinical Research

A Novel Method of Multimodal Medical Image Fusion Based on Hybrid Approach of NSCT and DTCWT

Multimodal Medical Image Fusion Techniques – A Review

A review of deep learning-based information fusion techniques for multimodal medical image classification

Multi-modality medical image fusion based on guided filter and image statistics in multidirectional shearlet transform domain