Abstract:The fusion of far infrared (FIR) and visible images aims to generate a high-quality composite image that contains salient structures and abundant texture details for human visual perception. However, the existing fusion methods typically fall short of utilizing complementary source image characteristics to boost the features extracted from degraded visible or FIR images, thus they cannot generate satisfactory fusion results in adverse lighting or weather conditions. In this paper, we propose a novel Cross-Modal multispectral image Enhancement and Fusion framework (CMEFusion), which adaptively enhances both FIR and visible inputs by leveraging complementary cross-modal features to further facilitate multispectral feature aggregation. Specifically, we first present a new cross-modal image enhancement sub-network (CMIENet), which is built on a CNN-Transformer hybrid architecture to perform the complementary exchange of local-salient and global-contextual features extracted from FIR and visible modalities, respectively. Then, we design a gradient-content differential fusion sub-network (GCDFNet) to progressively integrate decoupled gradient and content information via modified central difference convolution. Finally, we present a comprehensive joint enhancement-fusion multi-term loss function to drive the model to narrow the optimization gap between the above-mentioned two sub-networks based on the self-supervised aspects of exposure, color, structure, and intensity. In this manner, the proposed CMEFusion model facilitates better-performing visible and FIR image fusion in an end-to-end way, achieving enhanced visual quality with more natural and realistic appearances. Extensive experiments validate that CMEFusion surpasses state-of-the-art image fusion algorithms, as evidenced by superior performance in both visual quality and quantitative evaluations.

LeGFusion: Locally Enhanced Global Learning for Multimodal Image Fusion

LeGFusion: Locally-enhanced Global Learning for Multi-Modal Image Fusion

GALFusion: Multi-Exposure Image Fusion via a Global–Local Aggregation Learning Network

Multimodal Image Fusion Via Self-Supervised Transformer

Multi-Modal Image Fusion via Self-Supervised Transformer

MMF: A Multi-scale MobileNet Based Fusion Method for Infrared and Visible Image

LiMFusion: Infrared and Visible Image Fusion Via Local Information Measurement

DGFusion: an Effective Dynamic Generalizable Network for Infrared and Visible Image Fusion

CMEFusion: Cross-Modal Enhancement and Fusion of FIR and Visible Images

Rethinking Normalization Strategies and Convolutional Kernels for Multimodal Image Fusion

Equivariant Multi-Modality Image Fusion

CMFuse: Cross-Modal Features Mixing Via Convolution and MLP for Infrared and Visible Image Fusion

Bridging the Gap between Multi-focus and Multi-modal: A Focused Integration Framework for Multi-modal Image Fusion

Multi-modal Gated Mixture of Local-to-Global Experts for Dynamic Image Fusion

Global Context Enhanced Multi-modal Fusion for Referring Image Segmentation

Multi-modal medical image fusion based on densely-connected high-resolution CNN and hybrid transformer

SADFusion: A multi-scale infrared and visible image fusion method based on salient-aware and domain-specific

MM-Net: A MixFormer-Based Multi-Scale Network for Anatomical and Functional Image Fusion

Efficient Perceiving Local Details Via Adaptive Spatial-Frequency Information Integration for Multi-focus Image Fusion

SGFusion: A Saliency Guided Deep-Learning Framework for Pixel-Level Image Fusion

M4FNet: Multimodal Medical Image Fusion Network Via Multi-Receptive-field and Multi-Scale Feature Integration.