Abstract:Fully autonomous driving is considered one of the grand challenges of modern technology and a variety of approaches have emerged for creating and evaluating autonomous driving agents. The self-driving industry typically adopts a modular software architecture and uses large fleets of autonomous vehicles for data collection and evaluation. Simulation generally occupies a minor role in these workflows as a tool for basic unit tests and debugging. However, recent end-to-end methods in academia have elevated simulation’s role in the training of machine learning models and demonstrate state-of-the-art performance on simulated driving benchmarks as a result. In this thesis, we consider two questions. First, what are the state-ofthe-art approaches for end-to-end driving in simulation and how can we improve them? We use the open-source CARLA simulator as our testbed and work with the recent Leaderboard benchmark for evaluation. In our first project, we give a detailed failure analysis of “Learning by Cheating” (LBC) which has been considered the state-of-the-art in imitation learning. We then propose offline imitative reinforcement learning (OIRL) as an alternative approach and investigate its performance. In our second project, we give a failure analysis of the “World on Rails” (WOR) offline reinforcement learning method and propose “Slightly Online Reinforcement Learning” (SORL) as an improvement. Second, what value does simulation provide for the training and evaluation of driving algorithms? We draw attention to the crucial role that privileged information plays in both LBC and WOR’s algorithms. We emphasize that these methods are impossible to replicate without simulation and advocate for privileged information as an important tool for developing planning algorithms. Finally, we give a critical analysis of CARLA by pointing out issues in its perceptual and behavioral realism. We also critique the Leaderboard benchmark and provide discussion on why its primary metric cannot be reliably used as a proxy for good driving behavior.

Virtual Tools for Testing Autonomous Driving: A Survey and Benchmark of Simulators, Datasets, and Competitions

Perspective, Survey and Trends: Public Driving Datasets and Toolsets for Autonomous Driving Virtual Test

A Comprehensive Review on Traffic Datasets and Simulators for Autonomous Vehicles

Choose Your Simulator Wisely: A Review on Open-source Simulators for Autonomous Driving

How Simulation Helps Autonomous Driving:A Survey of Sim2real, Digital Twins, and Parallel Intelligence

A survey of autonomous driving frameworks and simulators

A Survey on Autonomous Driving System Simulators

Realistic 3D Simulators for Automotive: A Review of Main Applications and Features

NAVSIM: Data-Driven Non-Reactive Autonomous Vehicle Simulation and Benchmarking

Scenario-based Accelerated Testing for SOTIF in Autonomous Driving: A Review

Synthetic Datasets for Autonomous Driving: A Survey

Exiting the Simulation: The Road to Robust and Resilient Autonomous Vehicles at Scale

Data-driven Traffic Simulation: A Comprehensive Review

LGSVL Simulator: A High Fidelity Simulator for Autonomous Driving

From Model-Based to Data-Driven Simulation: Challenges and Trends in Autonomous Driving

A Survey on Scenario-Based Testing for Automated Driving Systems in High-Fidelity Simulation

Drive Right: Shaping Public's Trust, Understanding, and Preference Towards Autonomous Vehicles Using a Virtual Reality Driving Simulator

End-to-end Methods for Autonomous Driving in Simulation

Is it Safe to Drive? An Overview of Factors, Challenges, and Datasets for Driveability Assessment in Autonomous Driving

Bridging the Domain Gap between Synthetic and Real-World Data for Autonomous Driving