Abstract:The past decade has witnessed the rapid development of autonomous driving systems. However, it remains a daunting task to achieve full autonomy, especially when it comes to understanding the ever-changing, complex driving scenes. To alleviate the difficulty of perception, self-driving vehicles are usually equipped with a suite of sensors (e.g., cameras, LiDARs), hoping to capture the scenes with overlapping perspectives to minimize blind spots. Fusing these data streams and exploiting their complementary properties is thus rapidly becoming the current trend. Nonetheless, combining data that are captured by different sensors with drastically different ranging/ima-ging mechanisms is not a trivial task; instead, many factors need to be considered and optimized. If not careful, data from one sensor may act as noises to data from another sensor, with even poorer results by fusing them. Thus far, there has been no in-depth guidelines to designing the multi-modal fusion based 3D perception algorithms. To fill in the void and motivate further investigation, this survey conducts a thorough study of tens of recent deep learning based multi-modal 3D detection networks (with a special emphasis on LiDAR-camera fusion), focusing on their fusion stage (i.e., when to fuse), fusion inputs (i.e., what to fuse), and fusion granularity (i.e., how to fuse). These important design choices play a critical role in determining the performance of the fusion algorithm. In this survey, we first introduce the background of popular sensors used for self-driving, their data properties, and the corresponding object detection algorithms. Next, we discuss existing datasets that can be used for evaluating multi-modal 3D object detection algorithms. Then we present a review of multi-modal fusion based 3D detection networks, taking a close look at their fusion stage, fusion input and fusion granularity, and how these design choices evolve with time and technology. After the review, we discuss open challenges as well as possible solutions. We hope that this survey can help researchers to get familiar with the field and embark on investigations in the area of multi-modal 3D object detection.

A Survey of LiDAR and Camera Fusion Enhancement

Fusion Algorithm of Laser-Point Clouds and Optical Images

Deep Learning for LiDAR-only and LiDAR-fusion 3D Perception: a Survey

Multi-Sensor Image Fusion: A Survey of the State of the Art

SemanticBEVFusion: Rethink LiDAR-Camera Fusion in Unified Bird's-Eye View Representation for 3D Object Detection

DLFusion: Painting-Depth Augmenting-LiDAR for Multimodal Fusion 3D Object Detection

LIF-Seg: LiDAR and Camera Image Fusion for 3D LiDAR Semantic Segmentation

${\mathsf{EZFusion}}$: A Close Look at the Integration of LiDAR, Millimeter-Wave Radar, and Camera for Accurate 3D Object Detection and Tracking

LiDAR-Camera Cross Fusion Network Towards 3D Object Detection in Self-Driving

CLFusion:3D Semantic Segmentation Based on Camera and Lidar Fusion

FFPA-Net: Efficient Feature Fusion with Projection Awareness for 3D Object Detection

Real time object detection using LiDAR and camera fusion for autonomous driving

BiCo-Fusion: Bidirectional Complementary LiDAR-Camera Fusion for Semantic- and Spatial-Aware 3D Object Detection

Camera, LiDAR, and IMU Based Multi-Sensor Fusion SLAM: A Survey

Benchmarking the Robustness of LiDAR-Camera Fusion for 3D Object Detection

Multi-Modal 3D Object Detection in Autonomous Driving: A Survey

Explore the LiDAR-Camera Dynamic Adjustment Fusion for 3D Object Detection

Dense Sequential Fusion: Point Cloud Enhancement Using Foreground Mask Guidance for Multimodal 3-D Object Detection

Influence of Camera-LiDAR Configuration on 3D Object Detection for Autonomous Driving

Real-time depth completion based on LiDAR-stereo for autonomous driving

LIFT: Learning 4D LiDAR Image Fusion Transformer for 3D Object Detection