Abstract:Infrared and visible image fusion has been developed from vision perception oriented fusion methods to strategies which both consider the vision perception and high-level vision task. However, the existing task-driven methods fail to address the domain gap between semantic and geometric representation. To overcome these issues, we propose a high-level vision task-driven infrared and visible image fusion network via semantic and geometric domain transformation, terms as HSFusion. Specifically, to minimize the gap between semantic and geometric representation, we design two separate domain transformation branches by CycleGAN framework, and each includes two processes: the forward segmentation process and the reverse reconstruction process. CycleGAN is capable of learning domain transformation patterns, and the reconstruction process of CycleGAN is conducted under the constraint of these patterns. Thus, our method can significantly facilitate the integration of semantic and geometric information and further reduces the domain gap. In fusion stage, we integrate the infrared and visible features that extracted from the reconstruction process of two seperate CycleGANs to obtain the fused result. These features, containing varying proportions of semantic and geometric information, can significantly enhance the high level vision tasks. Additionally, we generate masks based on segmentation results to guide the fusion task. These masks can provide semantic priors, and we design adaptive weights for two distinct areas in the masks to facilitate image fusion. Finally, we conducted comparative experiments between our method and eleven other state-of-the-art methods, demonstrating that our approach surpasses others in both visual appeal and semantic segmentation task.

Fusing Geometry and Appearance for Road Segmentation

A Robust Road Segmentation Method

Improved Mean Shift Algorithm and Its Application

Road Scene Segmentation Via Fusing Camera and Lidar Data.

Lane segmentation based on fusion of color and edge information

Homography Guided Temporal Fusion for Road Line and Marking Segmentation

Road Segmentation with Image-LiDAR Data Fusion

An RGB-D Fusion Based Semantic Segmentation Algorithm Based on Neighborhood Metric Relations

Color Vision-Based Multi-Level Analysis and Fusion for Road Area Detection

ResNet-based Surface Normal Estimator with Multilevel Fusion Approach with Adaptive Median Filter Region Growth Algorithm for Road Scene Segmentation

Smooth fusion of multi-spectral images via total variation minimization for traffic scene semantic segmentation

MAFNet: Segmentation of Road Potholes With Multimodal Attention Fusion Network for Autonomous Vehicles

Enhancing Feature Fusion with Spatial Aggregation and Channel Fusion for Semantic Segmentation

Sequential Monocular Road Detection by Fusing Appearance and Geometric Information

Improving Road Segmentation in Challenging Domains Using Similar Place Priors

A duplex transform heterogeneous feature fusion network for road segmentation

GAF-Net: Geometric Contextual Feature Aggregation and Adaptive Fusion for Large-Scale Point Cloud Semantic Segmentation

Road Weather Condition Recognition via Fusing Images and Upsampled Point Cloud Reflection Intensities

HSFusion: A high-level vision task-driven infrared and visible image fusion network via semantic and geometric domain transformation

Multi-Scale Feature Fusion: Learning Better Semantic Segmentation for Road Pothole Detection

Unsupervised segmentation via semantic-apparent feature fusion