Abstract:The fusion of far infrared (FIR) and visible images aims to generate a high-quality composite image that contains salient structures and abundant texture details for human visual perception. However, the existing fusion methods typically fall short of utilizing complementary source image characteristics to boost the features extracted from degraded visible or FIR images, thus they cannot generate satisfactory fusion results in adverse lighting or weather conditions. In this paper, we propose a novel Cross-Modal multispectral image Enhancement and Fusion framework (CMEFusion), which adaptively enhances both FIR and visible inputs by leveraging complementary cross-modal features to further facilitate multispectral feature aggregation. Specifically, we first present a new cross-modal image enhancement sub-network (CMIENet), which is built on a CNN-Transformer hybrid architecture to perform the complementary exchange of local-salient and global-contextual features extracted from FIR and visible modalities, respectively. Then, we design a gradient-content differential fusion sub-network (GCDFNet) to progressively integrate decoupled gradient and content information via modified central difference convolution. Finally, we present a comprehensive joint enhancement-fusion multi-term loss function to drive the model to narrow the optimization gap between the above-mentioned two sub-networks based on the self-supervised aspects of exposure, color, structure, and intensity. In this manner, the proposed CMEFusion model facilitates better-performing visible and FIR image fusion in an end-to-end way, achieving enhanced visual quality with more natural and realistic appearances. Extensive experiments validate that CMEFusion surpasses state-of-the-art image fusion algorithms, as evidenced by superior performance in both visual quality and quantitative evaluations.

Combine Early and Late Fusion Together: A Hybrid Fusion Framework for Image-Text Matching

Comprehensive Framework of Early and Late Fusion for Image-Sentence Retrieval

Cross-Modal Hybrid Feature Fusion for Image-Sentence Matching

Fusion Layer Attention for Image-Text Matching.

DCMFNet: Deep Cross-Modal Fusion Network for Different Modalities with Iterative Gated Fusion

Matching Images and Text with Multi-modal Tensor Fusion and Re-ranking

A Unified Image Fusion Framework with Flexible Bilevel Paradigm Integration

Enhancing Separate Encoding with Multi-layer Feature Alignment for Image-Text Matching

Where Elegance Meets Precision: Towards a Compact, Automatic, and Flexible Framework for Multi-modality Image Fusion and Applications

TextFusion: Unveiling the Power of Textual Semantics for Controllable Image Fusion

From Text to Pixels: A Context-Aware Semantic Synergy Solution for Infrared and Visible Image Fusion

Searching a Hierarchically Aggregated Fusion Architecture for Fast Multi-Modality Image Fusion

Advanced Multimodal Deep Learning Architecture for Image-Text Matching

Heterogeneous Graph Fusion Network for Cross-Modal Image-Text Retrieval

CMEFusion: Cross-Modal Enhancement and Fusion of FIR and Visible Images

Bridging the gap: dual perception attention and local-global similarity fusion for cross-modal image-text matching

Shape-Former: Bridging CNN and Transformer via ShapeConv for multimodal image matching

Feature Fusion Based on Transformer for Cross-modal Retrieval

An Intermediate Fusion ViT Enables Efficient Text-Image Alignment in Diffusion Models

AFDFusion: an Adaptive Frequency Decoupling Fusion Network for Multi-Modality Image

Cross on Cross Attention: Deep Fusion Transformer for Image Captioning