Abstract:This paper presents a pioneering exploration into the integration of fine-grained human supervision within the autonomous driving domain to enhance system performance. The current advances in End-to-End autonomous driving normally are data-driven and rely on given expert trials. However, this reliance limits the systems' generalizability and their ability to earn human trust. Addressing this gap, our research introduces a novel approach by synchronously collecting data from human and machine drivers under identical driving scenarios, focusing on eye-tracking and brainwave data to guide machine perception and decision-making processes. This paper utilizes the Carla simulation to evaluate the impact brought by human behavior guidance. Experimental results show that using human attention to guide machine attention could bring a significant improvement in driving performance. However, guidance by human intention still remains a challenge. This paper pioneers a promising direction and potential for utilizing human behavior guidance to enhance autonomous systems.

What problem does this paper attempt to address?

The paper attempts to address the issue of how to enhance the performance of end-to-end autonomous driving systems by synchronously collecting human driver behavior data (such as eye-tracking and EEG data). Current end-to-end autonomous driving systems primarily rely on data-driven methods, which are usually based on expert demonstration data. However, this reliance limits the system's generalization ability and its capacity to gain human trust. To solve this problem, the study proposes a novel approach by synchronously collecting data from both human and machine drivers in the same driving scenarios, with a particular focus on eye-tracking and EEG data, to guide the machine's perception and decision-making process. Specifically, the main contributions of the paper include: 1. **Observing like a human driver**: By analyzing the eye movements of human drivers during the driving process, the cognitive process and attention focus are revealed, thereby correcting important objects that the machine may overlook. 2. **Deciding like a human driver**: Using human cognitive data to train an additional reward evaluator to provide extra feedback for imitation learning. The specific method involves first training a simple EEG classifier to identify whether a human would brake urgently based on the current situation, then inputting the simultaneously collected EEG data into the classifier and using the braking signal as additional feedback for fine-tuning in reinforcement learning. Experimental results show that using human eye-tracking data as an auxiliary feedback mechanism can significantly improve driving performance in imitation learning, particularly increasing the driving score on the Carla Long-Set 6 dataset from 50.63 to 51.29. However, the integration of cognitive data has not yet significantly directly improved the machine's driving score, which may be due to the accuracy and quality of EEG data still needing further improvement. In summary, the paper explores how to enhance the performance of autonomous driving systems through the guidance of human behavior data, providing new directions and potential for future research.

Enhancing End-to-End Autonomous Driving Systems Through Synchronized Human Behavior Data

A Shared Control Approach for Autonomous Vehicles via Driver Behaviors Learning

Toward human-vehicle collaboration: Review and perspectives on human-centered collaborative automated driving

An intelligent human-machine interaction-based longitudinal control strategy for autonomous vehicles

Human–machine cooperative decision-making and planning for automated vehicles using spatial projection of hand gestures

Humanlike Driving: Empirical Decision-Making System for Autonomous Vehicles

Understanding common human driving semantics for autonomous vehicles

A Simulation System for Human-in-the-Loop Driving

Towards Collaborative Autonomous Driving: Simulation Platform and End-to-End System

Survey on Human-Vehicle Interactions and AI Collaboration for Optimal Decision-Making in Automated Driving

Toward human-in-the-loop AI: Enhancing deep reinforcement learning via real-time human guidance for autonomous driving

Human-Like Decision-Making of Autonomous Vehicles in Dynamic Traffic Scenarios

Human Visual Attention Prediction Boosts Learning & Performance of Autonomous Driving Agents

Human-in-the-Loop Deep Reinforcement Learning with Application to Autonomous Driving

Learning driving behavior for autonomous vehicles using deep learning based methods*

End-to-end Autonomous Driving: Challenges and Frontiers

Socially-Compatible Behavior Design of Autonomous Vehicles with Verification on Real Human Data.

To Develop Human-like Automated Driving Strategy Based on Cognitive Construction: Appraisal and Perspective.

Human-Vehicle Cooperation on Prediction-Level: Enhancing Automated Driving with Human Foresight

Long-Term Evaluation of Drivers’ Behavioral Adaptation to an Adaptive Collision Avoidance System