Using Sparse Parts in Fused Information to Enhance Performance in Latent Low-Rank Representation-Based Fusion of Visible and Infrared Images

Chen-Yu Hao,Yao-Chung Chen,Fang-Shii Ning,Tien-Yin Chou,Mei-Hsin Chen

DOI: https://doi.org/10.3390/s24051514

IF: 3.9

2024-02-27

Sensors

Abstract:Latent Low-Rank Representation (LatLRR) has emerged as a prominent approach for fusing visible and infrared images. In this approach, images are decomposed into three fundamental components: the base part, salient part, and sparse part. The aim is to blend the base and salient features to reconstruct images accurately. However, existing methods often focus more on combining the base and salient parts, neglecting the importance of the sparse component, whereas we advocate for the comprehensive inclusion of all three parts generated from LatLRR image decomposition into the image fusion process, a novel proposition introduced in this study. Moreover, the effective integration of Convolutional Neural Network (CNN) technology with LatLRR remains challenging, particularly after the inclusion of sparse parts. This study utilizes fusion strategies involving weighted average, summation, VGG19, and ResNet50 in various combinations to analyze the fusion performance following the introduction of sparse parts. The research findings show a significant enhancement in fusion performance achieved through the inclusion of sparse parts in the fusion process. The suggested fusion strategy involves employing deep learning techniques for fusing both base parts and sparse parts while utilizing a summation strategy for the fusion of salient parts. The findings improve the performance of LatLRR-based methods and offer valuable insights for enhancement, leading to advancements in the field of image fusion.

engineering, electrical & electronic,chemistry, analytical,instruments & instrumentation

What problem does this paper attempt to address?

### What problems does this paper attempt to solve?

This paper aims to solve two main problems existing in the current visible light and infrared image fusion methods:

1. **Ignoring the importance of the sparse part**: When fusing visible light and infrared images, the existing LatLRR (Latent Low - Rank Representation) methods usually only focus on the base part and the salient part, while ignoring the sparse part. However, the sparse part may contain important image information, and ignoring this part may lead to a decline in the quality of the fused image.

2. **Challenges in integrating deep - learning techniques with LatLRR**: Integrating deep - learning techniques such as Convolutional Neural Networks (CNN) with LatLRR to improve the fusion effect still faces challenges, especially how to effectively integrate these techniques after introducing the sparse part.

To solve these problems, the author proposes a new image fusion strategy, specifically including:

- **Fully utilizing the three parts of LatLRR decomposition**: The base part, the salient part, and the sparse part. By introducing the sparse part, it is expected to retain more original image features, thereby improving the quality of the fused image.

- **Combining deep - learning models**: Using pre - trained CNN models such as VGG19 and ResNet50 to extract weight maps and applying them to the fusion process of different parts. This helps to better fuse image features and improve the fusion effect.

### Formula summary

The main formulas involved in the paper are as follows:

- **LatLRR optimization problem**:
\[
\min_{Z,L,E}\|Z\|_*+\|L\|_*+\lambda\|E\|_1\quad\text{s.t.}\quad X = XZ+LX + E
\]
where $\lambda$ is the balance coefficient, $\|\cdot\|_*$ represents the nuclear norm (i.e., the sum of the singular values of the matrix), $\|\cdot\|_1$ represents the L1 norm, $X$ is the observed data matrix, $Z$ is the low - rank coefficient, $L$ is the salient coefficient, and $E$ is the sparse part.

- **Weighted average fusion formula**:
\[
IM_{\text{weighted\_average}}=\text{weight}_{IR}\times IR_{\text{part}}+\text{weight}_{VIS}\times VIS_{\text{part}}
\]

- **Summation fusion formula**:
\[
IM_{\text{summation}}=IR_{\text{part}}+VIS_{\text{part}}
\]

- **Final fused image formula**:
\[
Image_{VIF}=VIF_{\text{Base\_part}}+VIF_{\text{salient\_part}}+VIF_{\text{sparse\_part}}
\]

Through these methods and formulas, the paper shows how to effectively use the sparse part and deep - learning models to improve the effect of visible light and infrared image fusion.