Abstract:In recent years, autonomous driving algorithms using low-cost vehicle-mounted cameras have attracted increasing endeavors from both academia and industry. There are multiple fronts to these endeavors, including object detection on roads, 3-D reconstruction etc., but in this work we focus on a vision-based model that directly maps raw input images to steering angles using deep networks. This represents a nascent research topic in computer vision. The technical contributions of this work are three-fold. First, the model is learned and evaluated on real human driving videos that are time-synchronized with other vehicle sensors. This differs from many prior models trained from synthetic data in racing games. Second, state-of-the-art models, such as PilotNet, mostly predict the wheel angles independently on each video frame, which contradicts common understanding of driving as a stateful process. Instead, our proposed model strikes a combination of spatial and temporal cues, jointly investigating instantaneous monocular camera observations and vehicle's historical states. This is in practice accomplished by inserting carefully-designed recurrent units (e.g., LSTM and Conv-LSTM) at proper network layers. Third, to facilitate the interpretability of the learned model, we utilize a visual back-propagation scheme for discovering and visualizing image regions crucially influencing the final steering prediction. Our experimental study is based on about 6 hours of human driving data provided by Udacity. Comprehensive quantitative evaluations demonstrate the effectiveness and robustness of our model, even under scenarios like drastic lighting changes and abrupt turning. The comparison with other state-of-the-art models clearly reveals its superior performance in predicting the due wheel angle for a self-driving car.

Autonomous Driving Control Using End-to-End Deep Learning

End-to-end deep learning-based autonomous driving control for high-speed environment

Autonomous Driving with Human Guided Image Feature Extraction

Deep Learning for Autonomous Driving System

Autonomous Vehicle Control: End-to-end Learning in Simulated Urban Environments

Deep learning and control algorithms of direct perception for autonomous driving

An End-to-end Auto-driving Method Based on 3D Lidar

End-to-End Autonomous Driving Through Dueling Double Deep Q-Network

A Convolutional Neural Network-Based End-to-End Self-Driving Using LiDAR and Camera Fusion: Analysis Perspectives in a Real-World Environment

A Survey of Deep Learning Applications to Autonomous Vehicle Control

Advancing Autonomous Navigation: Deep Learning Techniques for Self-Driving Cars

Recent Advancements in End-to-End Autonomous Driving using Deep Learning: A Survey

Vehicle-Related Scene Understanding Using Deep Learning

End-to-End Driving via Self-Supervised Imitation Learning Using Camera and LiDAR Data

A Survey of End-to-End Driving: Architectures and Training Methods

Involvement of Deep Learning for Vision Sensor-Based Autonomous Driving Control: A Review

A Survey of Deep Learning Techniques for Autonomous Driving

Overview of Deep Learning Intelligent Driving Methods

Survey of the Image Recognition Based on Deep Learning Network for Autonomous Driving Car

Autonomous driving in traffic with end-to-end vision-based deep learning

Deep Steering: Learning End-to-End Driving Model from Spatial and Temporal Visual Cues