Abstract:The multi-modality image fusion goal is to create a single image that provides a comprehensive scene description and conforms to visual perception by integrating complementary information about the merits of the different modalities, e.g., salient intensities of infrared images and detail textures of visible images. Although some works explore decoupled representations of multi-modality images, they struggle with complex nonlinear relationships, fine modal decoupling, and noise handling. To cope with this issue, we propose an adaptive frequency decoupling module to perceive the associative invariant and inherent specific among cross-modality by dynamically adjusting the learnable low frequency weight of the kernel. Specifically, we utilize a contrastive learning loss for restricting the solution space of feature decoupling to learn representations of both the invariant and specific in the multi-modality images. The underlying idea is that: in decoupling, low frequency features, which are similar in the representation space, should be pulled closer to each other, signifying the associative invariant, while high frequencies are pushed farther away, also indicating the intrinsic specific. Additionally, a multi-stage training manner is introduced into our framework to achieve decoupling and fusion. Stage I, MixEncoder and MixDecoder with the same architecture but different parameters are trained to perform decoupling and reconstruction supervised by the contrastive self-supervised mechanism. Stage II, two feature fusion modules are added to integrate the invariant and specific features and output the fused image. Extensive experiments demonstrated the proposed method superiority over the state-of-the-art methods in both qualitative and quantitative evaluation on two multi-modal image fusion tasks.

Image Decomposition Fusion Method Based on Sparse Representation and Neural Network

Image Denoising Via Multi-Scale Gated Fusion Network

Image Fusion with Nonsubsampled Contourlet Transform and Sparse Representation.

An Improved Combination of Image Denoisers Using Spatial Local Fusion Strategy

CCSR-Net: Unfolding Coupled Convolutional Sparse Representation for Multi-focus Image Fusion.

Adaptive Fusion of Remote Sensing Images with Sparse Representation

Image Fusion Method Based on Simultaneous Sparse Representation with Non-Subsampled Contourlet Transform

Fusion of Infrared and Visible Images Via Multi-Layer Convolutional Sparse Representation

Joint Image Fusion and Denoising Via Three-Layer Decomposition and Sparse Representation

An Infrared and Visible Image Fusion Method Based on Non-Subsampled Contourlet Transform and Joint Sparse Representation.

A Novel Multi-Modality Image Fusion Method Based on Image Decomposition and Sparse Representation

The Multi-Focus-Image-Fusion Method Based on Convolutional Neural Network and Sparse Representation.

DCFusion: Dual-Headed Fusion Strategy and Contextual Information Awareness for Infrared and Visible Remote Sensing Image

Deep Convolutional Sparse Coding Networks for Interpretable Image Fusion.

Image Fusion with Convolutional Sparse Representation

An Image Fusion Method Based on Sparse Representation and Sum Modified-Laplacian in NSCT Domain

An Image Fusion Method Based on NSCT and Dual-channel PCNN Model

Deep Convolutional Neural Network for Multi-modal Image Restoration and Fusion

Image Fusion Method Based on Spatially Masked Convolutional Sparse Representation

AFDFusion: an Adaptive Frequency Decoupling Fusion Network for Multi-Modality Image

SDNet: A Versatile Squeeze-and-Decomposition Network for Real-Time Image Fusion