Medical image fusion with deep neural networks

Nannan Liang

DOI: https://doi.org/10.1038/s41598-024-58665-9

IF: 4.6

2024-04-05

Scientific Reports

Abstract:Medical image fusion aims to fuse multiple images from a single or multiple imaging modes to enhance their corresponding clinical applications in diagnosing and evaluating medical problems, a trend that has attracted increasing attention. However, most recent medical image fusion methods require prior knowledge, making it difficult to select image features. In this paper, we propose a novel deep medical image fusion method based on a deep convolutional neural network (DCNN) for directly learning image features from original images. Specifically, source images are first decomposed by low rank representation to obtain the principal and salient components, respectively. Following that, the deep features are extracted from the decomposed principal components via DCNN and fused by a weighted-average rule. Then, considering the complementary between the salient components obtained by the low rank representation, a simple yet effective sum rule is designed to fuse the salient components. Finally, the fused result is obtained by reconstructing the principal and salient components. The experimental results demonstrate that the proposed method outperforms several state-of-the-art medical image fusion approaches in terms of both objective indices and visual quality.

multidisciplinary sciences

What problem does this paper attempt to address?

### Problems Addressed by the Paper

This paper aims to propose a novel medical image fusion method based on Deep Convolutional Neural Networks (DCNN) to directly learn image features from raw images. Specifically, the method first decomposes the source images into principal components and salient components through low-rank representation; then, it uses a pre-trained DCNN to extract deep features of the principal components and employs a weighted average rule for fusion; next, considering the complementarity between the salient components obtained by low-rank representation, a simple and effective summation rule is designed to fuse the salient components; finally, the fusion result is obtained by reconstructing the principal and salient components.

Experimental results show that the proposed method outperforms several state-of-the-art medical image fusion methods in terms of both objective metrics and visual quality. The main contributions of the paper include:
1. Proposing a novel CNN-based medical image fusion method, opening new opportunities for medical image fusion.
2. For the first time, using low-rank representation to decompose medical images into principal components and salient layers, effectively separating structural information and detail information.
3. Experimental validation on 44 pairs of medical images demonstrated that the proposed method's fusion performance is superior to other methods.

In summary, this paper mainly addresses the problem of how to effectively utilize CNN for medical image fusion and demonstrates its superior performance in the fusion of various medical image modalities.