Abstract:Multispectral pedestrian detection has shown many advantages in a variety of environments, particularly poor illumination conditions, by leveraging visible-thermal modalities. However, in-depth insight into distinguishing the complementary content of multimodal data and exploring the extent of multimodal feature fusion is still lacking. In this paper, we propose a novel multispectral pedestrian detector with multiscale cross-modal homogeneity enhancement and confidence-aware feature fusion. RGB and thermal streams are constructed to extract features and generate candidate proposals. During feature extraction, multiscale cross-modal homogeneity enhancement is proposed to enhance single-modal features using the separated homogeneous features via modal interactions. Homogeneity features encode the semantic information of the scene and are extracted from the RGB-thermal pairs by employing a channel attention mechanism. Proposals from two modalities are united to obtain multimodal proposals. Then, confidence measurement fusion is proposed to achieve multispectral feature fusion in each proposal by measuring the internal confidence of each modality and the interaction confidence between modalities. In addition, a confidence transfer loss function is designed to focus more on hard-to-detect samples during training. Experimental results on two challenging datasets demonstrate that the proposed method achieves better performance compared to existing methods.

Pedestrian Detection Aided by Fusion of Binocular Information

Online Multipedestrian Tracking Based on Fused Detections of Millimeter Wave Radar and Vision

A robust pedestrian detector based on heterogeneous feature fusion

Pedestrian Facial Attention Detection Using Deep Fusion and Multi-modal Fusion Classifier

Pedestrian Detection in Binocular Stereo Sequence Based on Appearance Consistency

Optimal Fusion-based Asymmetric Two-stream Networks for Multispectral Image Pedestrian Detection

Locality guided cross-modal feature aggregation and pixel-level fusion for multispectral pedestrian detection

Pedestrian detection based on channel feature fusion and enhanced semantic segmentation

Pedestrian Detection with Multi-View Convolution Fusion Algorithm

Attention Fusion for One-Stage Multispectral Pedestrian Detection

FDENet: Fusion Depth Semantics and Edge-Attention Information for Multispectral Pedestrian Detection

A Pedestrian Detection Algorithm Based on Score Fusion for Multi-LiDAR Systems

Multi-view and Multi-Plane Data Fusion for Effective Pedestrian Detection in Intelligent Visual Surveillance

Exploiting fusion architectures for multispectral pedestrian detection and segmentation

A Dual-Modality Pedestrian Detection Method Based on Multi-Scale Feature Fusion

Real-Time Pedestrian Detection Based on Dual-Modal Relevant Image Fusion

Pedestrian Detection Using Multi-Channel Visual Feature Fusion by Learning Deep Quality Model.

Pedestrian Tracking Based on Laser and Image Data Fusion.

A Fast RetinaNet Fusion Framework for Multi-Spectral Pedestrian Detection

An Improved Real-time Detection and Localization Scheme for Pedestrian Based on Information Fusion

Multiscale Cross-modal Homogeneity Enhancement and Confidence-aware Fusion for Multispectral Pedestrian Detection