Abstract:Infrared and visible image fusion (IVF) plays an important role in intelligent transportation system (ITS). The early works predominantly focus on boosting the visual appeal of the fused result, and only several recent approaches have tried to combine the high-level vision task with IVF. However, they prioritize the design of cascaded structure to seek unified suitable features and fit different tasks. Thus, they tend to typically bias toward to reconstructing raw pixels without considering the significance of semantic features. Therefore, we propose a novel prior semantic guided image fusion method based on the dual-modality strategy, improving the performance of IVF in ITS. Specifically, to explore the independent significant semantic of each modality, we first design two parallel semantic segmentation branches with a refined feature adaptive-modulation (RFaM) mechanism. RFaM can perceive the features that are semantically distinct enough in each semantic segmentation branch. Then, two pilot experiments based on the two branches are conducted to capture the significant prior semantic of two images, which then is applied to guide the fusion task in the integration of semantic segmentation branches and fusion branches. In addition, to aggregate both high-level semantics and impressive visual effects, we further investigate the frequency response of the prior semantics, and propose a multi-level representation-adaptive fusion (MRaF) module to explicitly integrate the low-frequent prior semantic with the high-frequent details. Extensive experiments on two public datasets demonstrate the superiority of our method over the state-of-the-art image fusion approaches, in terms of either the visual appeal or the high-level semantics.

Visual & textual fusion for region retrieval: from both fuzzy matching and bayesian reasoning aspects.

Visual and textual fusion for semantically supervised region-based retrieval

Region-aware RGB and Near-Infrared Image Fusion

Semantic Sensitive Region Retrieval Using Keyword-Integrated Bayesian Reasoning

Multi-view and region reasoning semantic enhancement for image-text retrieval

Information Fusion in Visual Question Answering: A Survey

Using Visual Dictionary to Associate Semantic Objects in Region-Based Image Retrieval

Multiple Level Visual Semantic Fusion Method for Image Re-Ranking.

TeRF: Text-driven and Region-aware Flexible Visible and Infrared Image Fusion

Improving Fusion of Region Features and Grid Features Via Two-Step Interaction for Image-Text Retrieval

Infrared and visible image fusion method based on visual saliency objects and fuzzy region attributes

Fidelity Based Visual Compensation and Salient Information Rectification for Infrared and Visible Image Fusion

DRM: Dynamic Region Matching for Image Retrieval Using Probabilistic Fuzzy Matching and Boosting Feature Selection

Comprehensive Framework of Early and Late Fusion for Image-Sentence Retrieval

From Text to Pixels: A Context-Aware Semantic Synergy Solution for Infrared and Visible Image Fusion

A Unified Image Fusion Framework with Flexible Bilevel Paradigm Integration

Injecting Linguistic into Visual Backbone: Query-Aware Multimodal Fusion Network for Remote Sensing Visual Grounding

Combining Regional Energy and Intuitionistic Fuzzy Sets for Infrared and Visible Image Fusion

Dual-modal Prior Semantic Guided Infrared and Visible Image Fusion for Intelligent Transportation System

Query Adaptive Fusion for Graph-Based Visual Reranking.

Improving Retrieval Performance by Region Constraints and Relevance Feedback