Abstract:In the field of autonomous driving, developing safe and trustworthy autonomous driving policies remains a significant challenge. Recently, Reinforcement Learning with Human Feedback (RLHF) has attracted substantial attention due to its potential to enhance training safety and sampling efficiency. Nevertheless, existing RLHF-enabled methods often falter when faced with imperfect human demonstrations, potentially leading to training oscillations or even worse performance than rule-based approaches. Inspired by the human learning process, we propose Physics-enhanced Reinforcement Learning with Human Feedback (PE-RLHF). This novel framework synergistically integrates human feedback (e.g., human intervention and demonstration) and physics knowledge (e.g., traffic flow model) into the training loop of reinforcement learning. The key advantage of PE-RLHF is its guarantee that the learned policy will perform at least as well as the given physics-based policy, even when human feedback quality deteriorates, thus ensuring trustworthy safety improvements. PE-RLHF introduces a Physics-enhanced Human-AI (PE-HAI) collaborative paradigm for dynamic action selection between human and physics-based actions, employs a reward-free approach with a proxy value function to capture human preferences, and incorporates a minimal intervention mechanism to reduce the cognitive load on human mentors. Extensive experiments across diverse driving scenarios demonstrate that PE-RLHF significantly outperforms traditional methods, achieving state-of-the-art (SOTA) performance in safety, efficiency, and generalizability, even with varying quality of human feedback. The philosophy behind PE-RLHF not only advances autonomous driving technology but can also offer valuable insights for other safety-critical domains. Demo video and code are available at: \<a class="link-external link-https" href="https://zilin-huang.github.io/PE-RLHF-website/" rel="external noopener nofollow">this https URL</a>

Shared Autonomy Based on Human-in-the-loop Reinforcement Learning with Policy Constraints

A Shared Control Approach for Autonomous Vehicles via Driver Behaviors Learning

Human-AI Shared Control via Policy Dissection

Shared Autonomy via Deep Reinforcement Learning

Reinforcement Learning for Human-Robot Shared Control

Human-machine Shared Autonomy Approach for Non-Full-time Effective Human Decisions

Adaptive Arbitration for Minimal Intervention Shared Control Via Deep Reinforcement Learning

Autonomous Boundary of Human-Machine Collaboration System Based on Reinforcement Learning

Human-Vehicle Shared Steering Control for Obstacle Avoidance: A Reference-Free Approach with Reinforcement Learning

Real-World Human-Robot Collaborative Reinforcement Learning

A Semi-Independent Policies Training Method with Shared Representation for Heterogeneous Multi-Agents Reinforcement Learning.

Safe Reinforcement Learning for Autonomous Vehicles through Parallel Constrained Policy Optimization

Policy Learning with Human Reinforcement

Trustworthy Human-AI Collaboration: Reinforcement Learning with Human Feedback and Physics Knowledge for Safe Autonomous Driving

Towards Cooperative Driving among Heterogeneous CAVs: A Safe Multi-Agent Reinforcement Learning Approach

Multi-agent Reinforcement Learning Behavioral Control for Nonlinear Second-Order Systems

Data Efficient Training for Reinforcement Learning with Adaptive Behavior Policy Sharing

Online Learning of Human Constraints from Feedback in Shared Autonomy

Human-in-the-Loop Deep Reinforcement Learning with Application to Autonomous Driving

Safe Autonomous Driving with Latent Dynamics and State-Wise Constraints

Human-AI Collaboration in Real-World Complex Environment with Reinforcement Learning