Abstract:Object detection plays an important role in autonomous driving, disaster rescue, robot navigation, intelligent video surveillance, and many other fields. Nonetheless, visible images are poor under weak illumination conditions, and thermal infrared images are noisy and have low resolution. Consequently, neither of these two data sources yields satisfactory results when used alone. While some scholars have combined visible and thermal images for object detection, most did not consider the illumination conditions and the different contributions of diverse data sources to the results. In addition, few studies have made use of the temperature characteristics of thermal images. Therefore, in the present study, visible and thermal images are utilized as the dataset, and RetinaNet is used as the baseline to fuse features from different data sources for object detection. Moreover, a dynamic weight fusion method, which is based on channel attention according to different illumination conditions, is used in the fusion component, and the channel attention and a priori temperature mask (CAPTM) module is proposed; the CAPTM can be applied to a deep learning network as a priori knowledge and maximizes the advantage of temperature information from thermal images. The main innovations of the present research include the following: (1) the consideration of different illumination conditions and the use of different fusion parameters for different conditions in the feature fusion of visible and thermal images; (2) the dynamic fusion of different data sources in the feature fusion of visible and thermal images; (3) the use of temperature information as a priori knowledge (CAPTM) in feature extraction. To a certain extent, the proposed methods improve the accuracy of object detection at night or under other weak illumination conditions and with a single data source. Compared with the state-of-the-art (SOTA) method, the proposed method is found to achieve superior detection accuracy with an overall mean average precision (mAP) improvement of 0.69%, including an AP improvement of 2.55% for the detection of the Person category. The results demonstrate the effectiveness of the research methods for object detection, especially temperature information-rich object detection.

Does Thermal data make the detection systems more reliable?

A Fourier-Transform-Based Framework with Asymptotic Attention for Mobile Thermal InfraRed Object Detection

Enhancing Object Detection in Adverse Conditions using Thermal Imaging

Object Detection in Thermal Spectrum for Advanced Driver-Assistance Systems (ADAS)

Vehicle Detection and Tracking Using Thermal Cameras in Adverse Visibility Conditions

Pedestrian Detection Based on Center, Temperature, Scale and Ratio Prediction in Thermal Imagery

Exploring Thermal Images for Object Detection in Underexposure Regions for Autonomous Driving

GEM: Glare or Gloom, I Can Still See You -- End-to-End Multimodal Object Detection

Robust Environment Perception for Automated Driving: A Unified Learning Pipeline for Visual-Infrared Object Detection

Visible-Thermal Image Object Detection via the Combination of Illumination Conditions and Temperature Information

TIRDet: Mono-Modality Thermal InfraRed Object Detection Based on Prior Thermal-To-Visible Translation

Object Detection and Tracking for Autonomous Vehicles in Adverse Weather Conditions

Borrow from Anywhere: Pseudo Multi-modal Object Detection in Thermal Imagery

Multispectral Deep Neural Network Fusion Method for Low-Light Object Detection

Evaluation of Thermal Imaging on Embedded GPU Platforms for Application in Vehicular Assistance Systems

Spatio-contextual deep network-based multimodal pedestrian detection for autonomous driving

Analysis of Thermal Imaging Performance under Extreme Foggy Conditions: Applications to Autonomous Driving

Deep Multimodal Detection in Reduced Visibility Using Thermal Depth Estimation for Autonomous Driving

Deep Visible and Thermal Image Fusion for Enhanced Pedestrian Visibility

Deep learning based object detection from multi-modal sensors: an overview

Sensor-Fused Nighttime System for Enhanced Pedestrian Detection in ADAS and Autonomous Vehicles