Abstract:In recent years, autonomous vehicles have attracted many researchers from academia and industry. Their efforts include object detection and tracking in the fields of computer vision, probabilistic-fusion and decision-making algorithms, etc. But most current methods that directly map from image pixels to steering behavior are not able to generate accurate results, especially under scenarios such as light and weather change drastically; this is largely due to the fact that these methods ignore the temporal relationship between frames. In this paper, we propose a novel end-to-end deep learning framework based on temporal and spatial attention mechanism, which aims to solve the problem of inaccurate vehicle steering angle prediction in complex environments and the difficulty of model interpretation. First, we use video sequence and historical steering angle sequence as inputs to the model, instead of just using a single frame as input. Second, we design a temporal attention mechanism to capture the long- and short-term memory in input visual information, and a spatial attention mechanism to capture key objects in the image and obtain their position information. This is achieved by inserting carefully-designed SE-Net, ConvLSTM and CNN layers into the appropriate layers of the network framework. Finally, we demonstrate the feasibility of the proposed model with public Comma2k19, with comparison to current advanced methods. Experimental results show that compared with state-of-the-art methods, the average absolute error (MAE) values of our model on the training set and testing set are reduced by 10.2% and 6.3%, respectively, and has more accurate steering prediction performance. In addition, we explain the trigger mechanism of steering behavior prediction by visualizing the spatial attention map and temporal attention score on Comma2k19 and Udacity datasets, which further demonstrates that the proposed model can learn human-like driving behavior.

PilotAttnNet: Multi-modal Attention Network for End-to-End Steering Control.

End-to-end Multi-Modal Multi-Task Vehicle Control for Self-Driving Cars with Visual Perception

Incorporating Orientations into End-to-end Driving Model for Steering Control

End-to-end Driving Model Based on Deep Learning and Attention Mechanism

Deep Steering: Learning End-to-End Driving Model from Spatial and Temporal Visual Cues

Learning End-to-End Autonomous Steering Model from Spatial and Temporal Visual Cues

A Novel End-to-End Model for Steering Behavior Prediction of Autonomous Ego-Vehicles Using Spatial and Temporal Attention Mechanism

End-to-end Spatiotemporal Attention Model for Autonomous Driving

ST-P3: End-to-end Vision-based Autonomous Driving via Spatial-Temporal Feature Learning

End-To-End Driving Model For Steering Control Of Autonomous Vehicles With Future Spatiotemporal Features

Steering angle prediction YOLOv5-based end-to-end adaptive neural network control for autonomous vehicles

Spatial-Temporal Multimodal End-to-End Autonomous Driving.

Attention-Based Deep Driving Model for Autonomous Vehicles with Surround-View Cameras

Human Insights Driven Latent Space for Different Driving Perspectives: A Unified Encoder for Efficient Multi-Task Inference

Multimodal End-to-End Learning for Autonomous Steering in Adverse Road and Weather Conditions

Deep learning and control algorithms of direct perception for autonomous driving

CNN-based Adaptive Intelligent Driving and Algorithm Optimization

E-DNet: An End-to-End Dual-Branch Network for Driver Steering Intention Detection

Enhancing Accuracy and Robustness of Steering Angle Prediction with Attention Mechanism

Learning A Steering Decision Policy for End-to-End Control of Autonomous Vehicle