Illumination-Guided RGBT Object Detection With Inter- and Intra-Modality Fusion

Yan Zhang,Huai Yu,Yujie He,Xinya Wang,Wen Yang

DOI: https://doi.org/10.1109/TIM.2023.3251414

2023-01-01

Abstract:Robust object detection is hindered by various illumination conditions in real-world applications. Common practice introduces thermal modality to augment the detection capability of RGB images in poor illumination conditions. However, a major challenge for such work is how to leverage the complementary information of RGB and thermal images effectively. In this article, we tackle this by combining the strengths of illumination-guided inter- and intra-modality information in both modalities. Specifically, we propose an effective object detection network dubbed Triple-I Net (TINet) to adaptively fuse the complementary features extracted from RGB and thermal images. First, we design an illumination-guided feature weighting module to guide the network in learning toward a reliable modality. Then, an inter-modality attention module is developed to amplify and complement differential features between these two modalities. We further design an intra-modality attention module using object heatmap prediction to enhance the foreground features of each modality. Finally, we fuse the inter- and intra-modality features for object detection with illumination-guided feature weights. Experimental results demonstrate the effectiveness of the proposed method on the FLIR-aligned dataset and the KAIST multispectral pedestrian detection dataset.