Abstract:Infrared and visible (IR-VIS) image fusion has gained significant attention for its broad application value. However, existing methods often neglect the complementary role of infrared image in restoring visible image features under hazy conditions. To address this, we propose a joint learning framework that utilizes infrared image for the restoration and fusion of hazy IR-VIS images. To mitigate the adverse effects of feature diversity between IR-VIS images, we introduce a prompt generation mechanism that regulates modality-specific feature incompatibility. This creates a prompt selection matrix from non-shared image information, followed by prompt embeddings generated from a prompt pool. These embeddings help generate candidate features for dehazing. We further design an infrared-assisted feature restoration mechanism that selects candidate features based on haze density, enabling simultaneous restoration and fusion within a single-stage framework. To enhance fusion quality, we construct a multi-stage prompt embedding fusion module that leverages feature supplementation from the prompt generation module. Our method effectively fuses IR-VIS images while removing haze, yielding clear, haze-free fusion results. In contrast to two-stage methods that dehaze and then fuse, our approach enables collaborative training in a single-stage framework, making the model relatively lightweight and suitable for practical deployment. Experimental results validate its effectiveness and demonstrate advantages over existing methods.

What problem does this paper attempt to address?

This paper attempts to solve the problems existing in the fusion of infrared and visible (IR - VIS) images under hazy conditions. Specifically, existing methods usually ignore the complementary role of infrared images in restoring the features of visible images, especially under hazy conditions. In addition, the traditional two - stage processing method (first defogging and then fusion) is difficult to balance the relationship between defogging and fusion tasks, and the model is relatively complex, which is not conducive to practical deployment. To solve these problems, the author proposes a new single - stage framework that uses infrared images to assist in joint restoration and fusion. The following are the main contributions of this method: 1. **Innovative joint framework**: An infrared - assisted joint learning framework is proposed, which can cooperatively train defogging and fusion tasks within a single stage. Compared with the traditional two - stage method, this framework makes more efficient use of the correlation between defogging and fusion tasks, enabling them to complement each other and achieve balanced results. 2. **Prompt generation module and multi - stage fusion**: A prompt generation module is designed. Through non - shared information, a prompt selection matrix is created to intelligently select and embed prompts to help restore visible images under hazy conditions. This not only enhances the restoration of visible image features but also reduces the adverse effects of modal differences between infrared and visible features. In addition, a multi - stage feature fusion mechanism based on prompt embedding is proposed, which further improves the fusion results. 3. **Efficient and compact model structure**: A single - stage processing framework is adopted, making the model structure more compact and easier to achieve a balance between tasks than the traditional two - stage defogging and fusion methods. This method effectively coordinates defogging and fusion, enabling the restored image features to better support the generation of clear, fog - free, high - quality fusion results. This design not only improves the fusion effect but also enhances the applicability of the model. In conclusion, this paper aims to solve the problems existing in the IR - VIS image fusion of existing methods under hazy conditions by introducing an infrared - assisted joint learning framework, thereby providing a more efficient and more compact solution.

Infrared-Assisted Single-Stage Framework for Joint Restoration and Fusion of Visible and Infrared Images under Hazy Conditions

Fusion of Visible and Infrared Images Using Saliency Analysis and Detail Preserving Based Image Decomposition

Detail preserved fusion of visible and infrared images using regional saliency extraction and multi-scale image decomposition

Infrared and visible image fusion method based on saliency detection and target-enhancement

Fusion of Low-Quality Visible and Infrared Images Based on Multi-Level Latent Low-Rank Representation Joint with Retinex Enhancement and Multi-Visual Weight Information

Fusion of Infrared and Visible Images Via Multi-Layer Convolutional Sparse Representation

Color-preserving visible and near-infrared image fusion for removing fog

A Deep Learning Framework for Infrared and Visible Image Fusion Without Strict Registration

Infrared–Visible Image Fusion through Feature-Based Decomposition and Domain Normalization

Infrared and Visible Image Fusion Method Based on Hierarchical Attention Mechanism

Infrared and Visual Image Fusion Through Infrared Feature Extraction and Visual Information Preservation

Infrared and Visible Image Fusion with Hybrid Image Filtering

Multi-Scale Fusion of Stretched Infrared and Visible Images

Infrared and visible image fusion with entropy-based adaptive fusion module and mask-guided convolutional neural network

Air infrared small target local dehazing based on multiple-factor fusion cascade network

Adaptive low light visual enhancement and high-significant target detection for infrared and visible image fusion

Infrared and visible image fusion based on infrared background suppression

Infrared and Visible Image Fusion Based on Filtering Enhancement

A Multi-scale Information Integration Framework for Infrared and Visible Image Fusion

SimpleFusion: A Simple Fusion Framework for Infrared and Visible Images

Infrared and visible image fusion based on saliency detection and deep multi-scale orientational features