Abstract:Infrared and visible image fusion (IVF) plays an important role in intelligent transportation system (ITS). The early works predominantly focus on boosting the visual appeal of the fused result, and only several recent approaches have tried to combine the high-level vision task with IVF. However, they prioritize the design of cascaded structure to seek unified suitable features and fit different tasks. Thus, they tend to typically bias toward to reconstructing raw pixels without considering the significance of semantic features. Therefore, we propose a novel prior semantic guided image fusion method based on the dual-modality strategy, improving the performance of IVF in ITS. Specifically, to explore the independent significant semantic of each modality, we first design two parallel semantic segmentation branches with a refined feature adaptive-modulation (RFaM) mechanism. RFaM can perceive the features that are semantically distinct enough in each semantic segmentation branch. Then, two pilot experiments based on the two branches are conducted to capture the significant prior semantic of two images, which then is applied to guide the fusion task in the integration of semantic segmentation branches and fusion branches. In addition, to aggregate both high-level semantics and impressive visual effects, we further investigate the frequency response of the prior semantics, and propose a multi-level representation-adaptive fusion (MRaF) module to explicitly integrate the low-frequent prior semantic with the high-frequent details. Extensive experiments on two public datasets demonstrate the superiority of our method over the state-of-the-art image fusion approaches, in terms of either the visual appeal or the high-level semantics.

Information Fusion on Oversegmented Images: An Application for Urban Scene Understanding.

Multimodal information fusion for urban scene understanding

Information Fusion and Evidential Grammars for Object Class Segmentation

Image Fusion Via Mutual Information Maximization for Semantic Segmentation in Autonomous Vehicles

Multi-Sensor Image Fusion: A Survey of the State of the Art

Robust 3D Semantic Segmentation Based on Multi-Phase Multi-Modal Fusion for Intelligent Vehicles

Sensor Fusion by Spatial Encoding for Autonomous Driving

UIF-BEV: an Underlying Information Fusion Framework for Bird's-Eye-View Semantic Segmentation

FuseSeg: Semantic Segmentation of Urban Scenes Based on RGB and Thermal Data Fusion

Detection-level Fusion for Multi-Object Perception in Dense Traffic Environment

Vehicle Detection by Fusing Part Model Learning and Semantic Scene Information for Complex Urban Surveillance

Color Vision-Based Multi-Level Analysis and Fusion for Road Area Detection

Dual-modal Prior Semantic Guided Infrared and Visible Image Fusion for Intelligent Transportation System

IS-Fusion: Instance-Scene Collaborative Fusion for Multimodal 3D Object Detection

A Quantitative Validation of Multi-Modal Image Fusion and Segmentation for Object Detection and Tracking

3D Instance Segmentation and Object Detection Framework Based on the Fusion of Lidar Remote Sensing and Optical Image Sensing

A Deep-Learning-Based Multimodal Data Fusion Framework for Urban Region Function Recognition

Elevation Information-Guided Multimodal Fusion Robust Framework for Remote Sensing Image Segmentation

ViT-FuseNet: MultiModal Fusion of Vision Transformer for Vehicle-Infrastructure Cooperative Perception

Vehicle Detection and Attribution from a Multi-Sensor Dataset Using a Rule-Based Approach Combined with Data Fusion