Abstract:In the field of autonomous driving, developing safe and trustworthy autonomous driving policies remains a significant challenge. Recently, Reinforcement Learning with Human Feedback (RLHF) has attracted substantial attention due to its potential to enhance training safety and sampling efficiency. Nevertheless, existing RLHF-enabled methods often falter when faced with imperfect human demonstrations, potentially leading to training oscillations or even worse performance than rule-based approaches. Inspired by the human learning process, we propose Physics-enhanced Reinforcement Learning with Human Feedback (PE-RLHF). This novel framework synergistically integrates human feedback (e.g., human intervention and demonstration) and physics knowledge (e.g., traffic flow model) into the training loop of reinforcement learning. The key advantage of PE-RLHF is its guarantee that the learned policy will perform at least as well as the given physics-based policy, even when human feedback quality deteriorates, thus ensuring trustworthy safety improvements. PE-RLHF introduces a Physics-enhanced Human-AI (PE-HAI) collaborative paradigm for dynamic action selection between human and physics-based actions, employs a reward-free approach with a proxy value function to capture human preferences, and incorporates a minimal intervention mechanism to reduce the cognitive load on human mentors. Extensive experiments across diverse driving scenarios demonstrate that PE-RLHF significantly outperforms traditional methods, achieving state-of-the-art (SOTA) performance in safety, efficiency, and generalizability, even with varying quality of human feedback. The philosophy behind PE-RLHF not only advances autonomous driving technology but can also offer valuable insights for other safety-critical domains. Demo video and code are available at: \<a class="link-external link-https" href="https://zilin-huang.github.io/PE-RLHF-website/" rel="external noopener nofollow">this https URL</a>

Prioritized Experience-Based Reinforcement Learning With Human Guidance for Autonomous Driving

Human-in-the-Loop Deep Reinforcement Learning with Application to Autonomous Driving

Efficient Deep Reinforcement Learning with Imitative Expert Priors for Autonomous Driving

Toward human-in-the-loop AI: Enhancing deep reinforcement learning via real-time human guidance for autonomous driving

Human-Guided Reinforcement Learning With Sim-to-Real Transfer for Autonomous Navigation

Improved Deep Reinforcement Learning with Expert Demonstrations for Urban Autonomous Driving

Efficient Reinforcement Learning for Autonomous Driving with Parameterized Skills and Priors

Learning to Drive Like Human Beings: A Method Based on Deep Reinforcement Learning

Human-Like Autonomous Car-Following Model with Deep Reinforcement Learning

Learning to drive via Apprenticeship Learning and Deep Reinforcement Learning

Intelligent control of self-driving vehicles based on adaptive sampling supervised actor-critic and human driving experience

Uncertainty-Aware Model-Based Reinforcement Learning with Application to Autonomous Driving

Uncertainty-Aware Model-Based Reinforcement Learning: Methodology and Application in Autonomous Driving

Optimizing Autonomous Driving for Safety: A Human-Centric Approach with LLM-Enhanced RLHF

Autonomous Highway Driving using Deep Reinforcement Learning

Research on a Personalized Decision Control Algorithm for Autonomous Vehicles Based on the Reinforcement Learning from Human Feedback Strategy

HAIM-DRL: Enhanced Human-in-the-loop Reinforcement Learning for Safe and Efficient Autonomous Driving

Trustworthy Human-AI Collaboration: Reinforcement Learning with Human Feedback and Physics Knowledge for Safe Autonomous Driving

HeuRL: A Heuristically Initialized Reinforcement Learning Method for Autonomous Driving Control Task

Conditional Predictive Behavior Planning with Inverse Reinforcement Learning for Human-like Autonomous Driving

Joint Optimization of Sensing, Decision-making and Motion-controlling for Autonomous Vehicles: A Deep Reinforcement Learning Approach