Abstract:Infrared and visible image fusion is an essential task for multi-sensor image fusion. Generative adversarial networks (GAN) have achieved remarkable performance in the fusion of infrared and visible image. Existing GAN based fusion methods merely using infrared and visible image as input for the fusion, while we found that differential images obtained by subtraction between two image sources could provide contrast information for the fusion. To this end, a novel dual fusion path generative adversarial network (DFPGAN) is proposed in this paper for infrared and visible image fusion. We divided the generator of generative adversarial network into two fusion paths namely infrared-visible path and differential path. The input of infrared-visible path concatenated two image sources to make infrared intensity and texture details keep balance fusion in this path. The input of differential path concatenated differential images obtained by subtraction between two image sources to make contrast information fusion in this path. The features extracted by two fusion paths are concatenated at the end of the generator to generate fused images with contrast effect and balanced information distribution. Meanwhile, we have implemented dual self-attention feature refine module (DSAM) on two fusion paths to refine feature maps in two fusion paths. We adopted switchable normalization layer (SN) substitute for batch normalization layer (BN) in the generator and discriminator to avoid fusion artifact. Furthermore, a mixed content loss is integrated in the generator loss functions to guide the generated image keep balanced information distribution and preserving contrast simultaneously. The adversarial training employed dual adversarial architecture to balance the distribution of infrared intensity and texture details. To verifying the improvement effect of fusion image on target detection, we introduce the Scaled-YOLOv4 target detection framework as evaluation framework, and use the proposed network to fuse RGB images and infrared images for target detection. The results of qualitative and quantitative experiments conducted on public datasets demonstrated the superiority of proposed network over other state-of-the-art methods and could generate fused images with distinctly contrast.

Fusion-UDCGAN: Multifocus Image Fusion via a U-Type Densely Connected Generation Adversarial Network

MFF-GAN: An unsupervised generative adversarial network with adaptive and gradient joint constraints for multi-focus image fusion

StackMFF: End-to-end Multi-Focus Image Stack Fusion Network

MFIF-GAN: A New Generative Adversarial Network for Multi-Focus Image Fusion

FuseGAN: Learning to fuse Multi-focus Image via Conditional Generative Adversarial Network

FFusionCGAN: An end-to-end fusion method for few-focus images using conditional GAN in cytopathological digital slides

UFA-FUSE: A novel deep supervised and hybrid model for multi-focus image fusion

The Fusion of Multi-Focus Images Based on the Complex Shearlet Features-Motivated Generative Adversarial Network

Multi-Focus Image Fusion Using U-Shaped Networks with a Hybrid Objective

Multi-focus image fusion with a deep convolutional neural network

Focus Affinity Perception and Super-Resolution Embedding for Multifocus Image Fusion

Image fusion based on generative adversarial network consistent with perception

Multiscale Feature Interactive Network for Multifocus Image Fusion

Boundary Aware Multi-focus Image Fusion Using Deep Neural Network.

AT-GAN: A generative adversarial network with attention and transition for infrared and visible image fusion

FusionGCN: Multi-focus image fusion using superpixel features generation GCN and pixel-level feature reconstruction CNN

Multi-focus image fusion with deep residual learning and focus property detection

Fully Convolutional Network-Based Multifocus Image Fusion.

Mutual-Guided Dynamic Network for Image Fusion

DFPGAN: Dual Fusion Path Generative Adversarial Network for Infrared and Visible Image Fusion

U2Fusion: A Unified Unsupervised Image Fusion Network