Cross-modality Complementary Information Fusion for Multispectral Pedestrian Detection.

Chaoqi Yan,Hong Zhang,Xuliang Li,Yifan Yang,Ding Yuan

DOI: https://doi.org/10.1007/s00521-023-08239-z

2023-01-01

Neural Computing and Applications

Abstract:Multispectral pedestrian detection has received increasing attention in recent years as color and thermal modalities can provide complementary visual information, especially under insufficient illumination conditions. However, there is still a persistent crucial problem that how to design the cross-modality fusion mechanism to fully exploit the complementary characteristics between different modalities. In this paper, we propose a novel cross-modality complementary information fusion network (denoted as CCIFNet) to comprehensively capture the long-range interactions with precise positional information and meanwhile preserve the inter-spatial relationship between different modalities in the feature extraction stage. Further, we design an adaptive illumination-aware weight generation module to adaptively weight the final detection confidence of color and thermal modalities by taking various illumination conditions into consideration. Specifically, we comprehensively compare three different fusion strategies about this module to synthetically explore the best way for generating the final illumination-aware fusion weights. Finally, we present a simple but effective feature alignment module to alleviate the position shift problem caused by the weakly aligned color-thermal image pairs. Extensive experiments and ablation studies on KAIST, CVC-14, FLIR and LLVIP multispectral object detection datasets show that the proposed CCIFNet can achieve state-of-the-art performance under different illumination evaluation settings, while keeping a competitive speed-accuracy trade-off for real-time applications.