Abstract:The past decade has witnessed the rapid development of autonomous driving systems. However, it remains a daunting task to achieve full autonomy, especially when it comes to understanding the ever-changing, complex driving scenes. To alleviate the difficulty of perception, self-driving vehicles are usually equipped with a suite of sensors (e.g., cameras, LiDARs), hoping to capture the scenes with overlapping perspectives to minimize blind spots. Fusing these data streams and exploiting their complementary properties is thus rapidly becoming the current trend. Nonetheless, combining data that are captured by different sensors with drastically different ranging/ima-ging mechanisms is not a trivial task; instead, many factors need to be considered and optimized. If not careful, data from one sensor may act as noises to data from another sensor, with even poorer results by fusing them. Thus far, there has been no in-depth guidelines to designing the multi-modal fusion based 3D perception algorithms. To fill in the void and motivate further investigation, this survey conducts a thorough study of tens of recent deep learning based multi-modal 3D detection networks (with a special emphasis on LiDAR-camera fusion), focusing on their fusion stage (i.e., when to fuse), fusion inputs (i.e., what to fuse), and fusion granularity (i.e., how to fuse). These important design choices play a critical role in determining the performance of the fusion algorithm. In this survey, we first introduce the background of popular sensors used for self-driving, their data properties, and the corresponding object detection algorithms. Next, we discuss existing datasets that can be used for evaluating multi-modal 3D object detection algorithms. Then we present a review of multi-modal fusion based 3D detection networks, taking a close look at their fusion stage, fusion input and fusion granularity, and how these design choices evolve with time and technology. After the review, we discuss open challenges as well as possible solutions. We hope that this survey can help researchers to get familiar with the field and embark on investigations in the area of multi-modal 3D object detection.

Robust Cognitive Capability in Autonomous Driving Using Sensor Fusion Techniques: A Survey

Multi-Sensor Fusion in Automated Driving: A Survey

Sensor and Sensor Fusion Technology in Autonomous Vehicles: A Review

Autonomous Multi-Sensor Fusion Techniques for Environmental Perception in Self-Driving Vehicles

Multi-modal Sensor Fusion for Auto Driving Perception: A Survey

Sensor Fusion in Autonomous Vehicle with Traffic Surveillance Camera System: Detection, Localization, and AI Networking

Exploring the challenges and opportunities of image processing and sensor fusion in autonomous vehicles: A comprehensive review

Multi-Sensor Fusion and Cooperative Perception for Autonomous Driving: A Review

On the Integration of Enabling Wireless Technologies and Sensor Fusion for Next-Generation Connected and Autonomous Vehicles

Safe Autonomy: Sensor Fusion in Complex Traffic Environments

Autonomous Driving System Driven by Artificial Intelligence Perception Fusion

Data Fusion in Infrastructure-Augmented Autonomous Driving System: Why? Where? and How?

Advancing Autonomous Driving Perception: Analysis of Sensor Fusion and Computer Vision Techniques

The Road Ahead: A Survey of Autonomous Vehicle Challenges and Opportunities

Cooperative Perception Technology of Autonomous Driving in the Internet of Vehicles Environment: A Review

Transformer-Based Sensor Fusion for Autonomous Driving: A Survey

A survey on perception, localization, planning, and control of autonomous vehicles: challenges and solutions

Multi-sensor fusion algorithm in cooperative vehicle-infrastructure system for blind spot warning

Multi-modal 3D Object Detection in Autonomous Driving: A Survey and Taxonomy

Multi-Modal 3D Object Detection in Autonomous Driving: A Survey