Abstract:The perception module of self-driving vehicles relies on a multi-sensor system to understand its environment. Recent advancements in deep learning have led to the rapid development of approaches that integrate multi-sensory measurements to enhance perception capabilities. This paper surveys the latest deep learning integration techniques applied to the perception module in autonomous driving systems, categorizing integration approaches based on "what, how, and when to integrate". A new taxonomy of integration is proposed, based on three dimensions: multi-view, multi-modality, and multi-frame. The integration operations and their pros and cons are summarized, providing new insights into the properties of an "ideal" data integration approach that can alleviate the limitations of existing methods. After reviewing hundreds of relevant papers, this survey concludes with a discussion of the key features of an optimal data integration approach.

What problem does this paper attempt to address?

### What problem does this paper attempt to solve? This paper aims to solve the data fusion problem in the perception module of the Autonomous Driving System (ADS). Specifically, it focuses on how to integrate data from different sensors through deep - learning techniques to improve the environmental understanding ability of autonomous vehicles. The following are the key problems that the paper attempts to solve: 1. **Limitations of single - sensor systems**: - A single sensor has limitations in terms of spatio - temporal coverage, information acquisition ability, etc., and cannot achieve comprehensive, accurate, and real - time environmental perception alone. - Different sensors perform differently under specific conditions. For example, cameras have poor performance in low - visibility conditions, LiDAR point clouds are sparse, and radars cannot obtain texture information. 2. **The need for multi - sensor data fusion**: - In order to overcome the shortcomings of single - sensor systems, it is necessary to effectively fuse data from different sensors to obtain more consistent, information - rich, and more accurate and reliable outputs. - Data fusion involves logically or physically transforming information from different sensors, thereby improving perception performance. 3. **Three key issues in data fusion**: - **"What to integrate"**: Determine the data content to be fused, including multi - view, multi - modality, and multi - frame data. - **"How to integrate"**: Select specific fusion operations, such as projection, concatenation, addition/average/weighted sum, probability methods, rule - based transaction processing, time - fusion methods, and neural network/encoder - decoder structures. - **"When to integrate"**: Determine the abstract level at which fusion occurs, including data - level, feature - level, and decision - level fusion. 4. **Limitations of existing methods**: - The paper summarizes the advantages and disadvantages of existing data fusion methods, points out the limitations of these methods in practical applications, and presents an outlook on the ideal data fusion method in the future. ### Main contributions of the paper - Propose a new classification system for data fusion based on the three dimensions of multi - view, multi - modality, and multi - frame. - Summarize common data fusion operations and their advantages and disadvantages. - Analyze the limitations of existing fusion techniques and present a vision of the ideal fusion method. By solving the above problems, the paper provides new insights and guidance for data fusion in autonomous driving systems, promoting further development in this field.

A survey on deep learning approaches for data integration in autonomous driving system

Deep learning-based perception systems for autonomous driving: A comprehensive survey

Deep Transfer Learning for Intelligent Vehicle Perception: a Survey

A Survey of Deep Learning Techniques for Autonomous Driving

Multi-modal Sensor Fusion for Auto Driving Perception: A Survey

A Survey on Theories and Applications for Self-Driving Cars Based on Deep Learning Methods

A Survey of Deep Learning Applications to Autonomous Vehicle Control

Autonomous Driving with Deep Learning: A Survey of State-of-Art Technologies

Overview of Deep Learning Intelligent Driving Methods

Vision-Based Environmental Perception for Autonomous Driving

Deep Learning for Image and Point Cloud Fusion in Autonomous Driving: A Review

Multi-Modal 3D Object Detection in Autonomous Driving: A Survey

A survey to applications of deep learning in autonomous driving

Deep Learning for Autonomous Driving System

Deep Multi-Modal Object Detection and Semantic Segmentation for Autonomous Driving: Datasets, Methods, and Challenges

A Survey of 3D Point Cloud and Deep Learning-Based Approaches for Scene Understanding in Autonomous Driving

Deep learning in autonomous driving: Advantages, limitations, and innovative solutions

Autonomous Driving System Driven by Artificial Intelligence Perception Fusion

Research on autonomous driving perception based on deep learning algorithm

Deep Learning Inspired Object Consolidation Approaches Using LiDAR Data for Autonomous Driving: A Review