Abstract:In recent years, autonomous driving algorithms using low-cost vehicle-mounted cameras have attracted increasing endeavors from both academia and industry. There are multiple fronts to these endeavors, including object detection on roads, 3-D reconstruction etc., but in this work we focus on a vision-based model that directly maps raw input images to steering angles using deep networks. This represents a nascent research topic in computer vision. The technical contributions of this work are three-fold. First, the model is learned and evaluated on real human driving videos that are time-synchronized with other vehicle sensors. This differs from many prior models trained from synthetic data in racing games. Second, state-of-the-art models, such as PilotNet, mostly predict the wheel angles independently on each video frame, which contradicts common understanding of driving as a stateful process. Instead, our proposed model strikes a combination of spatial and temporal cues, jointly investigating instantaneous monocular camera observations and vehicle's historical states. This is in practice accomplished by inserting carefully-designed recurrent units (e.g., LSTM and Conv-LSTM) at proper network layers. Third, to facilitate the interpretability of the learned model, we utilize a visual back-propagation scheme for discovering and visualizing image regions crucially influencing the final steering prediction. Our experimental study is based on about 6 hours of human driving data provided by Udacity. Comprehensive quantitative evaluations demonstrate the effectiveness and robustness of our model, even under scenarios like drastic lighting changes and abrupt turning. The comparison with other state-of-the-art models clearly reveals its superior performance in predicting the due wheel angle for a self-driving car.

FlowDriveNet: an End-to-End Network for Learning Driving Policies from Image Optical Flow and LiDAR Point Flow

Learning an Efficient and Safe Policy for Highway Driving Using Supervised Learning and Reinforcement Learning.

Driving Behaviour Style Study with a Hybrid Deep Learning Framework Based on GPS Data

A LiDAR Based End to End Controller for Robot Navigation Using Deep Neural Network

Autonomous Driving with Human Guided Image Feature Extraction

Optical Flow Matters: an Empirical Comparative Study on Fusing Monocular Extracted Modalities for Better Steering

Learning Graph-Based Representations for Scene Flow Estimation

End-to-End Interactive Prediction and Planning with Optical Flow Distillation for Autonomous Driving

Deep Steering: Learning End-to-End Driving Model from Spatial and Temporal Visual Cues

Multi-Task Learning in Autonomous Driving Scenarios Via Adaptive Feature Refinement Networks

LiDAR-Video Driving Dataset: Learning Driving Policies Effectively

FlowDSnet: an End-to-End Scene Flow Based LiDAR De-Skewing Network

LF-Net: A Learning-Based Frenet Planning Approach for Urban Autonomous Driving

Anticipating Driving Behavior through Deep Learning-Based Policy Prediction

LiDAR-as-Camera for End-to-End Driving

ST-P3: End-to-end Vision-based Autonomous Driving via Spatial-Temporal Feature Learning

Dual-flow network with attention for autonomous driving

DBNet: A Large-Scale Dataset for Driving Behavior Learning

DeepGoal: Learning to drive with driving intention from human control demonstration