Abstract:Infrared and visible image fusion aims to reconstruct fused images with comprehensive visual information by merging the complementary features of source images captured by different imaging sensors. This technology has been widely used in civil and military fields, such as urban security monitoring, remote sensing measurement, and battlefield reconnaissance. However, the existing methods still suffer from the preset fusion strategies that cannot be adjustable to different fusion demands and the loss of information during the feature propagation process, thereby leading to the poor generalization ability and limited fusion performance. Therefore, we propose an unsupervised end-to-end network with learnable fusion strategy for infrared and visible image fusion in this paper. The presented network mainly consists of three parts, including the feature extraction module, the fusion strategy module, and the image reconstruction module. First, in order to preserve more information during the process of feature propagation, dense connections and residual connections are applied to the feature extraction module and the image reconstruction module, respectively. Second, a new convolutional neural network is designed to adaptively learn the fusion strategy, which is able to enhance the generalization ability of our algorithm. Third, due to the lack of ground truth in fusion tasks, a loss function that consists of saliency loss and detail loss is exploited to guide the training direction and balance the retention of different types of information. Finally, the experimental results verify that the proposed algorithm delivers competitive performance when compared with several state-of-the-art algorithms in terms of both subjective and objective evaluations. Our codes are available at https://github.com/MinjieWan/Unsupervised-end-to-end-infrared-and-visible-image-fusion-network-using-learnable-fusion-strategy.

Infrared and Visible Image Fusion with Convolutional Neural Networks.

Fusion of Low-Illuminance Visible and Near-Infrared Images Based on Convolutional Neural Networks

Fusion of infrared and visual images through multiscale hybrid unidirectional total variation

Multi-scale Convolutional Neural Networks and Saliency Weight Maps for Infrared and Visible Image Fusion

Fusion of Infrared and Visible Images Via Multi-Layer Convolutional Sparse Representation

Infrared and visible image fusion with entropy-based adaptive fusion module and mask-guided convolutional neural network

Visible and Infrared Image Fusion Based on Attention and Multiscale Residuals

HDCTfusion: Hybrid Dual-Branch Network Based on CNN and Transformer for Infrared and Visible Image Fusion

Infrared-visible Image Fusion Using Accelerated Convergent Convolutional Dictionary Learning

Infrared and Visible Image Fusion Based on a Two-Stage Class Conditioned Auto-Encoder Network.

Fusion of Infrared and Visible Light Images Based on Improved Adaptive Dual-Channel Pulse Coupled Neural Network

Multi-scale attention-based lightweight network with dilated convolutions for infrared and visible image fusion

Infrared–Visible Image Fusion through Feature-Based Decomposition and Domain Normalization

Fusion of Infrared and Visible Images based on Spatial-Channel Attentional Mechanism

An infrared and visible image fusion network based on multi‐scale feature cascades and non‐local attention

Infrared and Visible Image Fusion Based on Filtering Enhancement

Infrared and Visible Images Fusion Using Visual Saliency and Dual-PCNN

MIFFuse: A Multi-Level Feature Fusion Network for Infrared and Visible Images

Unsupervised end-to-end infrared and visible image fusion network using learnable fusion strategy

A review on infrared and visible image fusion algorithms based on neural networks

Novel fusion method for visible light and infrared images based on NSST–SF–PCNN