DAF-Net: A Dual-Branch Feature Decomposition Fusion Network with Domain Adaptive for Infrared and Visible Image Fusion

Jian Xu,Xin He

2024-09-18

Abstract:Infrared and visible image fusion aims to combine complementary information from both modalities to provide a more comprehensive scene understanding. However, due to the significant differences between the two modalities, preserving key features during the fusion process remains a challenge. To address this issue, we propose a dual-branch feature decomposition fusion network (DAF-Net) with domain adaptive, which introduces Multi-Kernel Maximum Mean Discrepancy (MK-MMD) into the base encoder and designs a hybrid kernel function suitable for infrared and visible image fusion. The base encoder built on the Restormer network captures global structural information while the detail encoder based on Invertible Neural Networks (INN) focuses on extracting detail texture information. By incorporating MK-MMD, the DAF-Net effectively aligns the latent feature spaces of visible and infrared images, thereby improving the quality of the fused images. Experimental results demonstrate that the proposed method outperforms existing techniques across multiple datasets, significantly enhancing both visual quality and fusion performance. The related Python code is available at <a class="link-external link-https" href="https://github.com/xujian000/DAF-Net" rel="external noopener nofollow">this https URL</a>.

Computer Vision and Pattern Recognition,Machine Learning

What problem does this paper attempt to address?

The problem that this paper attempts to solve is how to maintain the integrity of key features during the fusion of infrared and visible - light images. Since there are significant differences between these two - modal images in terms of imaging principles, resolutions, and spectral responses, it is a challenge to maintain the key features of these differences during the fusion process. Specifically, the paper aims to:

1. **Combine complementary information**: Combine the complementary information in infrared images (which are good at capturing thermal radiation) and visible - light images (which retain rich details and colors) to provide a more comprehensive understanding of the scene.
2. **Improve fusion quality**: By designing a new dual - branch feature decomposition and fusion network (DAF - Net), ensure that both global structural information and local detailed texture information can be retained during the fusion process.
3. **Solve the domain adaptation problem**: Introduce the multi - kernel maximum mean discrepancy (MK - MMD) to align the latent feature spaces of infrared and visible - light images, thereby reducing the distribution differences between different modalities and improving the quality of the fused image.

### Specific methods

To achieve the above goals, the paper proposes the following methods:

- **Dual - branch network structure**: DAF - Net includes a base encoder and a detail encoder. The base encoder is based on the Restormer network and is used to capture global structural information; the detail encoder is based on the invertible neural network (INN) and is used to extract detailed texture information.
- **Domain - adaptive layer**: By introducing MK - MMD in the base encoder, the model can align the feature distributions of infrared and visible - light images in the shared feature space, thereby improving the fusion effect.
- **Mixed kernel function**: Design a mixed kernel function that combines the advantages of the Gaussian kernel and the Laplacian kernel to better capture the differences between global structures and local details.
- **Two - stage training**: Train the encoder - decoder branch in the first stage and the fusion layer in the second stage to gradually optimize the model performance.

### Experimental results

The experimental results show that DAF - Net significantly improves the visual quality and fusion performance of fused images on multiple datasets and is superior to many existing methods.

Through these methods, the paper successfully solves the problem of maintaining key features in the fusion of infrared and visible - light images and provides new ideas and technical means for research in this field.