Abstract:In the field of autonomous driving, developing safe and trustworthy autonomous driving policies remains a significant challenge. Recently, Reinforcement Learning with Human Feedback (RLHF) has attracted substantial attention due to its potential to enhance training safety and sampling efficiency. Nevertheless, existing RLHF-enabled methods often falter when faced with imperfect human demonstrations, potentially leading to training oscillations or even worse performance than rule-based approaches. Inspired by the human learning process, we propose Physics-enhanced Reinforcement Learning with Human Feedback (PE-RLHF). This novel framework synergistically integrates human feedback (e.g., human intervention and demonstration) and physics knowledge (e.g., traffic flow model) into the training loop of reinforcement learning. The key advantage of PE-RLHF is its guarantee that the learned policy will perform at least as well as the given physics-based policy, even when human feedback quality deteriorates, thus ensuring trustworthy safety improvements. PE-RLHF introduces a Physics-enhanced Human-AI (PE-HAI) collaborative paradigm for dynamic action selection between human and physics-based actions, employs a reward-free approach with a proxy value function to capture human preferences, and incorporates a minimal intervention mechanism to reduce the cognitive load on human mentors. Extensive experiments across diverse driving scenarios demonstrate that PE-RLHF significantly outperforms traditional methods, achieving state-of-the-art (SOTA) performance in safety, efficiency, and generalizability, even with varying quality of human feedback. The philosophy behind PE-RLHF not only advances autonomous driving technology but can also offer valuable insights for other safety-critical domains. Demo video and code are available at: \<a class="link-external link-https" href="https://zilin-huang.github.io/PE-RLHF-website/" rel="external noopener nofollow">this https URL</a>

What problem does this paper attempt to address?

The paper attempts to address the problem of developing a driving strategy that is both safe and trustworthy in the field of autonomous driving. Specifically, the paper focuses on the following points: 1. **Safety**: Existing reinforcement learning (RL) methods require extensive interaction with the environment during training, which not only reduces sampling efficiency but also poses safety risks. Additionally, traditional rule-based methods tend to fail in complex and highly interactive scenarios. 2. **Trustworthiness**: Existing human feedback-based reinforcement learning methods (RLHF) can improve training safety and sampling efficiency, but when the quality of human demonstrations declines, these methods may lead to training oscillations or performance degradation, sometimes even performing worse than rule-based methods. 3. **Generalization Ability**: Autonomous driving systems need to perform well in various uncertain and complex traffic scenarios, and existing methods fall short in this aspect. To address these issues, the paper proposes a new framework—Physics-enhanced Reinforcement Learning with Human Feedback (PE-RLHF). The main features of this framework include: - **Integration of Physical Knowledge**: By incorporating physical knowledge (such as traffic flow models) into the reinforcement learning training loop, it ensures that even when the quality of human demonstrations declines, the learned strategy can at least perform comparably to the given physical baseline strategy, thereby providing trustworthy safety improvements. - **Dynamic Action Selection Mechanism**: A Physics-enhanced Human-AI (PE-HAI) collaboration paradigm is introduced, dynamically selecting actions generated by either human or physical baseline strategies to reduce the cognitive load on human mentors. - **Reward-free Design**: A reward-free approach is adopted, capturing human preferences through the agent's value function, thus avoiding the challenge of designing complex reward functions. - **Minimal Intervention Mechanism**: The minimal intervention mechanism reduces the frequency of human mentor involvement, further enhancing the system's efficiency and reliability. The paper validates the superiority and robustness of PE-RLHF through extensive experiments in various driving scenarios. Notably, it outperforms existing methods in terms of safety, efficiency, and generalization ability, especially when faced with human feedback of varying quality.

Trustworthy Human-AI Collaboration: Reinforcement Learning with Human Feedback and Physics Knowledge for Safe Autonomous Driving

HAIM-DRL: Enhanced Human-in-the-loop Reinforcement Learning for Safe and Efficient Autonomous Driving

Human-AI Shared Control via Policy Dissection

Optimizing Autonomous Driving for Safety: A Human-Centric Approach with LLM-Enhanced RLHF

Trustworthy safety improvement for autonomous driving using reinforcement learning

Human-in-the-Loop Deep Reinforcement Learning with Application to Autonomous Driving

Efficient Learning of Safe Driving Policy via Human-AI Copilot Optimization

Prioritized Experience-Based Reinforcement Learning With Human Guidance for Autonomous Driving

Autonomous Highway Driving using Deep Reinforcement Learning

Enhancing System-Level Safety in Mixed-Autonomy Platoon via Safe Reinforcement Learning

Towards Robust Decision-Making for Autonomous Highway Driving Based on Safe Reinforcement Learning

HGRL: Human-Driving-Data Guided Reinforcement Learning for Autonomous Driving

Safe-State Enhancement Method for Autonomous Driving Via Direct Hierarchical Reinforcement Learning.

Autonomous Highway Merging in Mixed Traffic Using Reinforcement Learning and Motion Predictive Safety Controller

Toward human-in-the-loop AI: Enhancing deep reinforcement learning via real-time human guidance for autonomous driving

Deep Reinforcement Learning for Human-Like Driving Policies in Collision Avoidance Tasks of Self-Driving Cars

Deep Reinforcement Learning with Enhanced Safety for Autonomous Highway Driving

Comprehensive Training and Evaluation on Deep Reinforcement Learning for Automated Driving in Various Simulated Driving Maneuvers

Efficient Deep Reinforcement Learning with Imitative Expert Priors for Autonomous Driving

From Naturalistic Traffic Data to Learning-Based Driving Policy: A Sim-to-Real Study

Human-Like Autonomous Car-Following Model with Deep Reinforcement Learning