End-to-end Autonomous Driving: Challenges and Frontiers

Li Chen,Penghao Wu,Kashyap Chitta,Bernhard Jaeger,Andreas Geiger,Hongyang Li

2024-08-15

Abstract:The autonomous driving community has witnessed a rapid growth in approaches that embrace an end-to-end algorithm framework, utilizing raw sensor input to generate vehicle motion plans, instead of concentrating on individual tasks such as detection and motion prediction. End-to-end systems, in comparison to modular pipelines, benefit from joint feature optimization for perception and planning. This field has flourished due to the availability of large-scale datasets, closed-loop evaluation, and the increasing need for autonomous driving algorithms to perform effectively in challenging scenarios. In this survey, we provide a comprehensive analysis of more than 270 papers, covering the motivation, roadmap, methodology, challenges, and future trends in end-to-end autonomous driving. We delve into several critical challenges, including multi-modality, interpretability, causal confusion, robustness, and world models, amongst others. Additionally, we discuss current advancements in foundation models and visual pre-training, as well as how to incorporate these techniques within the end-to-end driving framework. we maintain an active repository that contains up-to-date literature and open-source projects at <a class="link-external link-https" href="https://github.com/OpenDriveLab/End-to-end-Autonomous-Driving" rel="external noopener nofollow">this https URL</a>.

Robotics,Artificial Intelligence,Computer Vision and Pattern Recognition,Machine Learning

What problem does this paper attempt to address?

### Problems the Paper Aims to Address

This paper primarily explores the research progress and challenges of end-to-end autonomous driving systems and provides a comprehensive review of related technologies. Specifically:

1. **Definition and Advantages of End-to-End Autonomous Driving Systems**:
- Definition: End-to-end autonomous driving systems refer to fully differentiable programs that use raw sensor data as input to directly generate vehicle driving plans or low-level control actions.
- Advantages: Compared to traditional modular designs, end-to-end systems can integrate tasks such as perception, prediction, and planning into a unified model for joint training, thereby optimizing the overall system performance.

2. **Research Motivation**:
- Although traditional modular designs are easy to debug, the different optimization goals of each module may lead to the overall system failing to achieve a unified objective. The end-to-end approach can overcome this issue by achieving better performance through joint optimization.

3. **Development History**:
- The history of end-to-end autonomous driving can be traced back to the ALVINN project in 1988. In recent years, significant progress has been made in imitation learning (IL) and reinforcement learning (RL) with the development of deep neural networks.
- Around 2021, various multimodal and advanced architectures (such as Transformer) emerged, significantly improving performance in CARLA benchmark tests.

4. **Key Challenges**:
- The paper provides a detailed analysis of several important challenges, including multimodal fusion, interpretability, causal confusion, robustness, and world models.
- It also discusses how to improve the system's generalization ability and safety, such as through data augmentation, auxiliary tasks, and attention visualization.

5. **Future Trends**:
- With the development of large-scale foundational models and data engines, end-to-end autonomous driving is expected to further enhance performance.
- The paper advocates for adopting a modular end-to-end planning philosophy and introducing more high-quality data to drive research progress.

In summary, this paper aims to provide a comprehensive review of the latest advancements in the field of end-to-end autonomous driving, deeply exploring the key technologies and challenges to guide future research directions.