Self-supervised feature adaption for infrared and visible image fusion

Fan Zhao,Wenda Zhao,Libo Yao,Yu Liu

DOI: https://doi.org/10.1016/j.inffus.2021.06.002

IF: 18.6

2021-12-01

Information Fusion

Abstract:<p>Benefitting from the strong feature extraction capability of deep learning, infrared and visible image fusion has made a great progress. Since infrared and visible images are obtained by different sensors with different imaging mechanisms, there exists domain discrepancy, which becomes stumbling block for effective fusion. In this paper, we propose a novel self-supervised feature adaption framework for infrared and visible image fusion. We implement a self-supervised strategy that facilitates the backbone network to extract features with adaption while retaining the vital information by reconstructing the source images. Specifically, we preliminary adopt an encoder network to extract features with adaption. Then, two decoders with attention mechanism blocks are utilized to reconstruct the source images in a self-supervised way, forcing the adapted features to contain vital information of the source images. Further, considering the case that source images contain low-quality information, we design a novel infrared and visible image fusion and enhancement model, improving the fusion method's robustness. Experiments are constructed to evaluate the proposed method qualitatively and quantitatively, which show that the proposed method achieves the state-of-art performance comparing with existing infrared and visible image fusion methods.</p>

computer science, artificial intelligence, theory & methods

What problem does this paper attempt to address?

This paper mainly discusses a core issue in the fusion of infrared and visible light images, that is, due to different imaging mechanisms and domain discrepancies caused by different sensors used for capturing the two types of images, effective fusion is hindered. To solve this problem, the paper proposes a novel self-supervised feature adaptation framework for the fusion of infrared and visible light images.

The author first introduces traditional image fusion methods, such as multi-scale transformation, sparse representation, and subspace methods, and points out that these methods have limited performance in dealing with domain differences when fusing infrared and visible light images. In recent years, with the powerful feature extraction capability of deep learning, many studies have been applied to the fusion of infrared and visible light images. However, existing methods often use the same convolutional operations to adapt to domain differences, which can easily lead to the loss of important details.

In the paper, the author proposes a self-supervised strategy, which uses an encoding network to extract adaptive features and then uses two decoders with attention mechanisms to reconstruct the source images in a self-supervised manner, forcing the adaptive features to contain important information from the source images. In addition, considering that the source images may contain low-quality information, they also design a new fusion and enhancement model for infrared and visible light images to improve the robustness of the method.

Experimental results show that the proposed method outperforms existing infrared and visible light image fusion methods in both qualitative and quantitative evaluations. Through this self-supervised feature adaptation method, domain differences can be better handled, key information can be preserved, and the quality and visual perception of the fusion images can be improved.