Abstract:Multispectral pedestrian detection is a technology designed to detect and locate pedestrians in Color and Thermal images, which has been widely used in automatic driving, video surveillance, etc. So far most available multispectral pedestrian detection algorithms only achieved limited success in pedestrian detection because of the lacking take into account the confusion of pedestrian information and background noise in Color and Thermal images. Here we propose a multispectral pedestrian detection algorithm, which mainly consists of a cascaded information enhancement module and a cross-modal attention feature fusion module. On the one hand, the cascaded information enhancement module adopts the channel and spatial attention mechanism to perform attention weighting on the features fused by the cascaded feature fusion block. Moreover, it multiplies the single-modal features with the attention weight element by element to enhance the pedestrian features in the single-modal and thus suppress the interference from the background. On the other hand, the cross-modal attention feature fusion module mines the features of both Color and Thermal modalities to complement each other, then the global features are constructed by adding the cross-modal complemented features element by element, which are attentionally weighted to achieve the effective fusion of the two modal features. Finally, the fused features are input into the detection head to detect and locate pedestrians. Extensive experiments have been performed on two improved versions of annotations (sanitized annotations and paired annotations) of the public dataset KAIST. The experimental results show that our method demonstrates a lower pedestrian miss rate and more accurate pedestrian detection boxes compared to the comparison method. Additionally, the ablation experiment also proved the effectiveness of each module designed in this paper.

MCANet: Multiscale Cross-Modality Attention Network for Multispectral Pedestrian Detection

Multiscale 3-D-2-D Mixed CNN and Lightweight Attention-Free Transformer for Hyperspectral and LiDAR Classification

Attentive Alignment Network for Multispectral Pedestrian Detection

Transformer fusion and histogram layer multispectral pedestrian detection network

GCANet: A Cross-Modal Pedestrian Detection Method Based on Gaussian Cross Attention Network

Locality guided cross-modal feature aggregation and pixel-level fusion for multispectral pedestrian detection

Attention-based Cross-modality Interaction for Multispectral Pedestrian Detection

Cascaded information enhancement and cross-modal attention feature fusion for multispectral pedestrian detection

CSANet: Channel and Spatial mixed Attention CNN for Pedestrian Detection

HAFNet: Hierarchical Attentive Fusion Network for Multispectral Pedestrian Detection

Learning a Dynamic Cross-Modal Network for Multispectral Pedestrian Detection

Cross-modality interactive attention network for multispectral pedestrian detection.

Lightweight Cross-Modal Multispectral Pedestrian Detection Based on Spatial Reweighted Attention Mechanism

Improving Multispectral Pedestrian Detection by Addressing Modality Imbalance Problems

MFMANet: a multispectral pedestrian detection network using multi-resolution RGB feature reuse with multi-scale FIR attentions

MMCAN: Multi-Modal Cross-Attention Network for Free-Space Detection with Uncalibrated Hyperspectral Sensors

Improving Multispectral Pedestrian Detection with Scale‐aware Permutation Attention and Adjacent Feature Aggregation

INSANet: INtra-INter Spectral Attention Network for Effective Feature Fusion of Multispectral Pedestrian Detection

Multispectral Pedestrian Detection via Simultaneous Detection and Segmentation

MS-DETR: Multispectral Pedestrian Detection Transformer with Loosely Coupled Fusion and Modality-Balanced Optimization

Multi-Task Collaborative Attention Network for Pedestrian Attribute Recognition