Abstract:Model-based reinforcement learning (RL) is anticipated to exhibit higher sample efficiency compared to model-free RL by utilizing a virtual environment model. However, it is challenging to obtain sufficiently accurate representations of the environmental dynamics due to uncertainties in complex systems and environments. An inaccurate environment model may degrade the sample efficiency and performance of model-based RL. Furthermore, while model-based RL can improve sample efficiency, it often still requires substantial training time to learn from scratch, potentially limiting its advantages over model-free approaches. To address these challenges, this paper introduces a knowledge-informed model-based residual reinforcement learning framework aimed at enhancing learning efficiency by infusing established expert knowledge into the learning process and avoiding the issue of beginning from zero. Our approach integrates traffic expert knowledge into a virtual environment model, employing the Intelligent Driver Model (IDM) for basic dynamics and neural networks for residual dynamics, thus ensuring adaptability to complex scenarios. We propose a novel strategy that combines traditional control methods with residual RL, facilitating efficient learning and policy optimization without the need to learn from scratch. The proposed approach is applied to CAV trajectory control tasks for the dissipation of stop-and-go waves in mixed traffic flow. Experimental results demonstrate that our proposed approach enables the CAV agent to achieve superior performance in trajectory control compared to the baseline agents in terms of sample efficiency, traffic flow smoothness and traffic mobility. The source code and supplementary materials are available at <a class="link-external link-https" href="https://github.com/zihaosheng/traffic-expertise-RL/" rel="external noopener nofollow">this https URL</a>.

Accelerating deep reinforcement learning via knowledge-guided policy network

Learning with Training Wheels: Speeding up Training with a Simple Controller for Deep Reinforcement Learning

Reinforcement Learning with Partial Parametric Model Knowledge

Shaping in Reinforcement Learning Via Knowledge Transferred from Human-Demonstrations

Efficient Deep Reinforcement Learning Via Adaptive Policy Transfer

Learning To Walk With Prior Knowledge

Knowledge-Guided Exploration in Deep Reinforcement Learning

Efficient Deep Reinforcement Learning Through Policy Transfer.

Knowledge Transfer from Simple to Complex: A Safe and Efficient Reinforcement Learning Framework for Autonomous Driving Decision-Making

KnowRU: Knowledge Reusing via Knowledge Distillation in Multi-agent Reinforcement Learning

Efficient Deep Reinforcement Learning with Imitative Expert Priors for Autonomous Driving

Deep Reinforcement Learning on Autonomous Driving Policy With Auxiliary Critic Network

Playing a Strategy Game with Knowledge-Based Reinforcement Learning

Interactive Learning with Corrective Feedback for Policies based on Deep Neural Networks

Prioritized Experience-Based Reinforcement Learning With Human Guidance for Autonomous Driving

Knowledge-guided Deep Reinforcement Learning for Interactive Recommendation

Shared Autonomy Based on Human-in-the-loop Reinforcement Learning with Policy Constraints

Traffic expertise meets residual RL: Knowledge-informed model-based residual reinforcement learning for CAV trajectory control

Enhanced Probabilistic Inference Algorithm Using Probabilistic Neural Networks For Learning Control

Efficient Reinforcement Learning for Autonomous Driving with Parameterized Skills and Priors