Abstract:Humans race drones faster than neural networks trained for end-to-end autonomous flight. This may be related to the ability of human pilots to select task-relevant visual information effectively. This work investigates whether neural networks capable of imitating human eye gaze behavior and attention can improve neural networks’ performance for the challenging task of vision-based autonomous drone racing. We hypothesize that gaze-based attention prediction can be an efficient mechanism for visual information selection and decision making in a simulator-based drone racing task. We test this hypothesis using eye gaze and flight trajectory data from 18 human drone pilots to train a visual attention prediction model. We then use this visual attention prediction model to train an end-to-end controller for vision-based autonomous drone racing using imitation learning. We compare the drone racing performance of the attention-prediction controller to those using raw image inputs and image-based abstractions (i.e., feature tracks). Comparing success rates for completing a challenging race track by autonomous flight, our results show that the attention-prediction based controller (88% success rate) outperforms the RGB-image (61% success rate) and feature-tracks (55% success rate) controller baselines. Furthermore, visual attention-prediction and feature-track based models showed better generalization performance than image-based models when evaluated on hold-out reference trajectories. Our results demonstrate that human visual attention prediction improves the performance of autonomous vision-based drone racing agents and provides an essential step towards vision-based, fast, and agile autonomous flight that eventually can reach and even exceed human performances.

Human Visual Attention Prediction Boosts Learning & Performance of Autonomous Driving Agents

Visual attention for behavioral cloning in autonomous driving

Recent advancements in driver's attention prediction

Explaining Autonomous Driving by Learning End-to-End Visual Attention

Visual attention prediction improves performance of autonomous drone racing agents

Predicting Driver Attention in Critical Situations

Driver's Visual Fixation Attention Prediction in Dynamic Scenes Using Hybrid Neural Networks

Attention-Based Deep Driving Model for Autonomous Vehicles with Surround-View Cameras

Adaptive Visual Interaction Based Multi-Target Future State Prediction For Autonomous Driving Vehicles

Brain Inspired Cognitive Model with Attention for Self-Driving Cars

Perceive, Attend, and Drive: Learning Spatial Attention for Safe Self-Driving

Utilising Visual Attention Cues for Vehicle Detection and Tracking

A Gaze Model Improves Autonomous Driving

Prediction of the driver's focus of attention based on feature visualization of a deep autonomous driving model

How Do Drivers Allocate Their Potential Attention? Driving Fixation Prediction via Convolutional Neural Networks

Using External Attention in Vision-based Autonomous Driving

Where and What: Driver Attention-based Object Detection

End-to-end Spatiotemporal Attention Model for Autonomous Driving

Parallel Vision and Learning for Intelligent Perception in Smart Driving

Explaining autonomous driving with visual attention and end-to-end trainable region proposals

A Novel Driving Behavior Learning and Visualization Method with Natural Gaze Prediction