Abstract:The fusion of far infrared (FIR) and visible images aims to generate a high-quality composite image that contains salient structures and abundant texture details for human visual perception. However, the existing fusion methods typically fall short of utilizing complementary source image characteristics to boost the features extracted from degraded visible or FIR images, thus they cannot generate satisfactory fusion results in adverse lighting or weather conditions. In this paper, we propose a novel Cross-Modal multispectral image Enhancement and Fusion framework (CMEFusion), which adaptively enhances both FIR and visible inputs by leveraging complementary cross-modal features to further facilitate multispectral feature aggregation. Specifically, we first present a new cross-modal image enhancement sub-network (CMIENet), which is built on a CNN-Transformer hybrid architecture to perform the complementary exchange of local-salient and global-contextual features extracted from FIR and visible modalities, respectively. Then, we design a gradient-content differential fusion sub-network (GCDFNet) to progressively integrate decoupled gradient and content information via modified central difference convolution. Finally, we present a comprehensive joint enhancement-fusion multi-term loss function to drive the model to narrow the optimization gap between the above-mentioned two sub-networks based on the self-supervised aspects of exposure, color, structure, and intensity. In this manner, the proposed CMEFusion model facilitates better-performing visible and FIR image fusion in an end-to-end way, achieving enhanced visual quality with more natural and realistic appearances. Extensive experiments validate that CMEFusion surpasses state-of-the-art image fusion algorithms, as evidenced by superior performance in both visual quality and quantitative evaluations.

Context Enhancement Through Image Fusion: A Multiresolution Approach Based on Convolution of Cauchy Distributions

Research of improving semantic image segmentation based on a feature fusion model

Contextual fusion enhances robustness to image blurring

Detail enhanced multi-source fusion using visual weight map extraction based on multi scale edge preserving decomposition

The Realistic Fusion of Multi-spectral Images

Dual-Head Fusion Network for Image Enhancement

DCFusion: Dual-Headed Fusion Strategy and Contextual Information Awareness for Infrared and Visible Remote Sensing Image

MDDCMA: A Distributed Image Fusion Framework Based on Multiscale Dense Dilated Convolution and Coordinate Mean Attention

Image Resolution Enhancement Based on Wavelet Domain HMT and Fusion

Enhanced low-light image fusion through multi-stage processing with Bayesian analysis and quadratic contrast function

Fractal Dimension-Based Multi-Focus Image Fusion via Coupled Neural P Systems in NSCT Domain

Multi-scale fusion algorithm of intensity and polarization-difference images based on edge information enhancement

Sparse Codes Fusion for Context Enhancement of Night Video Surveillance

DCAFuse: Dual-Branch Diffusion-CNN Complementary Feature Aggregation Network for Multi-Modality Image Fusion

CMEFusion: Cross-Modal Enhancement and Fusion of FIR and Visible Images

A Multiscale Image Fusion Algorithm Based on Joint Distribution of Source Coefficients

A fast multi-resolution image fusion method based on visual features

CHFusion: A Cross-modality High-resolution Representation Framework for Infrared and Visible Image Fusion

Multiscale Depth Fusion With Contextual Hybrid Enhancement Network for Image Dehazing

DCFusion: A Dual-Frequency Cross-Enhanced Fusion Network for Infrared and Visible Image Fusion.

MEFusion: Unsupervised Mutual Enhancement for Multimodal Image Fusion