SPFusion: A multi-task semantic perception infrared and visible light fusion method with quality assessment
Zhenyang Liang,Mingxin Yu,Yichen Sun,Mingli Dong
DOI: https://doi.org/10.1016/j.displa.2024.102903
IF: 3.074
2024-12-02
Displays
Abstract:The fusion of infrared and visible light aims to combine the advantages of infrared highlighting thermal targets and the rich textures of visible light, to produce a fused image that is both texture-rich and has prominent targets. However, under extreme conditions such as heavy fog and overexposure, directly fusing low-quality visible light images with infrared images may retain excessive redundant information in these areas. Therefore, this paper proposes a method for achieving adaptive fusion under extreme conditions, called SPFusion. Specifically, the network is guided by a quality assessment module and advanced visual tasks such as segmentation, classification, and reconstruction, to learn an adaptive balance point for rationally distributing infrared and visible light information. Additionally, a progressive fusion strategy and a modality complementarity module are employed to fully integrate information from each modality. Consequently, the network can autonomously allocate visible and infrared information based on image quality and semantics. Extensive experiments have demonstrated that SPFusion achieves state-of-the-art (SOTA) performance on the TNO, MSRS, LLVIP, and RoadScene datasets, and enhances the performance of advanced visual tasks such as object detection. The code is available as open source at linshenj/SPFusion (github.com).
engineering, electrical & electronic,instruments & instrumentation,optics,computer science, hardware & architecture