Abstract:The fusion of far infrared (FIR) and visible images aims to generate a high-quality composite image that contains salient structures and abundant texture details for human visual perception. However, the existing fusion methods typically fall short of utilizing complementary source image characteristics to boost the features extracted from degraded visible or FIR images, thus they cannot generate satisfactory fusion results in adverse lighting or weather conditions. In this paper, we propose a novel Cross-Modal multispectral image Enhancement and Fusion framework (CMEFusion), which adaptively enhances both FIR and visible inputs by leveraging complementary cross-modal features to further facilitate multispectral feature aggregation. Specifically, we first present a new cross-modal image enhancement sub-network (CMIENet), which is built on a CNN-Transformer hybrid architecture to perform the complementary exchange of local-salient and global-contextual features extracted from FIR and visible modalities, respectively. Then, we design a gradient-content differential fusion sub-network (GCDFNet) to progressively integrate decoupled gradient and content information via modified central difference convolution. Finally, we present a comprehensive joint enhancement-fusion multi-term loss function to drive the model to narrow the optimization gap between the above-mentioned two sub-networks based on the self-supervised aspects of exposure, color, structure, and intensity. In this manner, the proposed CMEFusion model facilitates better-performing visible and FIR image fusion in an end-to-end way, achieving enhanced visual quality with more natural and realistic appearances. Extensive experiments validate that CMEFusion surpasses state-of-the-art image fusion algorithms, as evidenced by superior performance in both visual quality and quantitative evaluations.

CurveMEF: Multi-eposure fusion via curve embedding network

Deep Guided Learning for Fast Multi-Exposure Image Fusion

Curvelet Based Hyperspectral Image Fusion

Ref-MEF: Reference-Guided Flexible Gated Image Reconstruction Network for Multi-Exposure Image Fusion

Multi-Scale Exposure Fusion Via Content Adaptive Edge-Preserving Smoothing Pyramids.

Rethinking Multi-Exposure Image Fusion with Extreme and Diverse Exposure Levels: A Robust Framework Based on Fourier Transform and Contrastive Learning.

EMEF: Ensemble Multi-Exposure Image Fusion

MEFNet: Multi-scale Event Fusion Network for Motion Deblurring

Efficient Multi-exposure Image Fusion Via Filter-dominated Fusion and Gradient-driven Unsupervised Learning.

Multi Exposure Fusion for High Dynamic Range Imaging Via Multi-Channel Gradient Tensor

Multi-exposure image fusion via deep perceptual enhancement

Multiexposure Fusion Based on Multiscale Edge Information Measurement and Adaptive Pyramid Optimization

MEFLUT: Unsupervised 1D Lookup Tables for Multi-exposure Image Fusion

Multi-Exposure Image Fusion via Deformable Self-Attention

TransMEF: A Transformer-Based Multi-Exposure Image Fusion Framework Using Self-Supervised Multi-Task Learning

Perceptual Multi-Exposure Fusion

CMEFusion: Cross-Modal Enhancement and Fusion of FIR and Visible Images

ACE-MEF: Adaptive Clarity Evaluation-Guided Network with Illumination Correction for Multi-exposure Image Fusion

MEF-GAN: Multi-Exposure Image Fusion via Generative Adversarial Networks