Abstract:In this work, we present an uncertainty-based method for sensor fusion with camera and radar data. The outputs of two neural networks, one processing camera and the other one radar data, are combined in an uncertainty aware manner. To this end, we gather the outputs and corresponding meta information for both networks. For each predicted object, the gathered information is post-processed by a gradient boosting method to produce a joint prediction of both networks. In our experiments we combine the YOLOv3 object detection network with a customized $1D$ radar segmentation network and evaluate our method on the nuScenes dataset. In particular we focus on night scenes, where the capability of object detection networks based on camera data is potentially handicapped. Our experiments show, that this approach of uncertainty aware fusion, which is also of very modular nature, significantly gains performance compared to single sensor baselines and is in range of specifically tailored deep learning based fusion approaches.

What problem does this paper attempt to address?

### Problems the Paper Attempts to Solve This paper aims to address the issue of object detection for autonomous vehicles in complex environments, particularly the limitations of camera data for object detection in nighttime scenarios. Specifically, the paper proposes an uncertainty-based sensor fusion method that combines camera and radar data to improve the accuracy of object detection. ### Main Research Content 1. **Problem Background**: - Autonomous driving systems need to correctly recognize the surrounding environment in various situations. - Data from different sensors need to be correctly interpreted within limited computational resources. - Weather conditions (such as sunlight, heavy rain, fog, snow) pose challenges to existing computer vision systems. - Using multiple sensors (sensor fusion) can improve the accuracy of object detection. 2. **Method Introduction**: - **Uncertainty-Aware Fusion**: The paper proposes an uncertainty-based sensor fusion method that combines the outputs of two neural networks (one processing camera data and the other processing radar data) in an uncertainty-aware manner. - **Experimental Setup**: YOLOv3 is used as the object detection network for camera data, combined with a custom 1D radar segmentation network, and evaluated on the nuScenes dataset. - **Key Scenarios**: Special attention is given to nighttime scenarios, as the object detection capability of camera data may be limited in these scenarios. 3. **Technical Details**: - **Radar Data Preprocessing**: Radar data is projected onto the front-view camera image to generate the input matrix. - **Network Architecture**: The radar network architecture is based on FCN-8, including convolutional layers, deconvolutional layers, and fully connected layers. - **Loss Function**: Binary cross-entropy loss function is used for training. - **Fusion Algorithm**: A gradient boosting method is used to combine the outputs of the two networks and their uncertainty information to generate a joint prediction. 4. **Experimental Results**: - **Performance Improvement**: The uncertainty-aware fusion method significantly improves the accuracy of object detection compared to single-sensor baselines. - **Nighttime Scenario Performance**: The method performs exceptionally well in nighttime scenarios, detecting more vehicles. - **False Positive Reduction**: By lowering the object detection threshold, false positive predictions can be reduced while maintaining a high true positive rate. ### Conclusion The proposed method significantly improves the accuracy of object detection in nighttime scenarios by combining the uncertainty information from camera and radar data, achieving more reliable object detection. The method is modular and can provide redundant backup in case of sensor failure, further enhancing system safety.

YOdar: Uncertainty-based Sensor Fusion for Vehicle Detection with Camera and Radar Sensors

Radar-Image Fusion Transformer for Object Detection with Uncertainty Estimation

Radar and Camera Fusion for Multi-Task Sensing in Autonomous Driving

Multi-Camera Sensor Fusion for Visual Odometry using Deep Uncertainty Estimation

Radar Voxel Fusion for 3D Object Detection

Radar-Camera Sensor Fusion for Joint Object Detection and Distance Estimation in Autonomous Vehicles

Cross-Domain Spatial Matching for Camera and Radar Sensor Data Fusion in Autonomous Vehicle Perception System

ROFusion: Efficient Object Detection using Hybrid Point-wise Radar-Optical Fusion

A Resource Efficient Fusion Network for Object Detection in Bird's-Eye View using Camera and Raw Radar Data

RCM-Fusion: Radar-Camera Multi-Level Fusion for 3D Object Detection

ClusterFusion: Leveraging Radar Spatial Features for Radar-Camera 3D Object Detection in Autonomous Vehicles

BEV-Radar: Bidirectional Radar-Camera Fusion for 3D Object Detection

Object Detection Using Multi-Sensor Fusion Based on Deep Learning

A Multi-scale Fusion Obstacle Detection Algorithm for Autonomous Driving Based on Camera and Radar

A Deep Learning-based Radar and Camera Sensor Fusion Architecture for Object Detection

Radar-Vision Fusion for Vehicle Detection and Tracking

MVFusion: Multi-View 3D Object Detection with Semantic-aligned Radar and Camera Fusion

HVDetFusion: A Simple and Robust Camera-Radar Fusion Framework

RVNet: Deep Sensor Fusion of Monocular Camera and Radar for Image-Based Obstacle Detection in Challenging Environments

Camera-Radar Fusion with Radar Channel Extension and Dual-CBAM-FPN for Object Detection

MS-YOLO: Object Detection Based on YOLOv5 Optimized Fusion Millimeter-wave Radar and Machine Vision