Abstract:End-to-end autonomous driving has made impressive progress in recent years. Existing methods usually adopt the decoupled encoder-decoder paradigm, where the encoder extracts hidden features from raw sensor data, and the decoder outputs the ego-vehicle's future trajectories or actions. Under such a paradigm, the encoder does not have access to the intended behavior of the ego agent, leaving the burden of finding out safety-critical regions from the massive receptive field and inferring about future situations to the decoder. Even worse, the decoder is usually composed of several simple multi-layer perceptrons (MLP) or GRUs while the encoder is delicately designed (e.g., a combination of heavy ResNets or Transformer). Such an imbalanced resource-task division hampers the learning process. In this work, we aim to alleviate the aforementioned problem by two principles: (1) fully utilizing the capacity of the encoder; (2) increasing the capacity of the decoder. Concretely, we first predict a coarse-grained future position and action based on the encoder features. Then, conditioned on the position and action, the future scene is imagined to check the ramification if we drive accordingly. We also retrieve the encoder features around the predicted coordinate to obtain fine-grained information about the safety-critical region. Finally, based on the predicted future and the retrieved salient feature, we refine the coarse-grained position and action by predicting its offset from ground-truth. The above refinement module could be stacked in a cascaded fashion, which extends the capacity of the decoder with spatial-temporal prior knowledge about the conditioned future. We conduct experiments on the CARLA simulator and achieve state-of-the-art performance in closed-loop benchmarks. Extensive ablation studies demonstrate the effectiveness of each proposed module.

Exploring Latent Pathways: Enhancing the Interpretability of Autonomous Driving with a Variational Autoencoder

Autonomous Driving with Human Guided Image Feature Extraction

Bi-Level Optimization Augmented with Conditional Variational Autoencoder for Autonomous Driving in Dense Traffic

Human Insights Driven Latent Space for Different Driving Perspectives: A Unified Encoder for Efficient Multi-Task Inference

Variational Autoencoder-Based Vehicle Trajectory Prediction with an Interpretable Latent Space

Variational End-to-End Navigation and Localization

Drive Anywhere: Generalizable End-to-end Autonomous Driving with Multi-modal Foundation Models

Exploring Contextual Representation and Multi-Modality for End-to-End Autonomous Driving

Think Twice Before Driving: Towards Scalable Decoders for End-to-End Autonomous Driving

On the Road to Clarity: Exploring Explainable AI for World Models in a Driver Assistance System

Enhancing scene understanding based on deep learning for end-to-end autonomous driving

Learning an Interpretable Model for Driver Behavior Prediction with Inductive Biases

Explaining Autonomous Driving Actions with Visual Question Answering

Deep Steering: Learning End-to-End Driving Model from Spatial and Temporal Visual Cues

Explaining Autonomous Driving by Learning End-to-End Visual Attention

Learning End-to-End Autonomous Steering Model from Spatial and Temporal Visual Cues

Explainable deep learning improves human mental models of self-driving cars

Exploring the Causality of End-to-End Autonomous Driving

Variational Autoencoders for exteroceptive perception in reinforcement learning-based collision avoidance

Attention-Based Deep Driving Model for Autonomous Vehicles with Surround-View Cameras