Abstract:By properly utilizing the learned environment model, model-based reinforcement learning methods can improve the sample efficiency for decision-making problems. Beyond using the learned environment model to train a policy, the success of MCTS-based methods shows that directly incorporating the learned environment model as a planner to make decisions might be more effective. However, when action space is of high dimension and continuous, directly planning according to the learned model is costly and non-trivial. Because of two challenges: (1) the infinite number of candidate actions and (2) the temporal dependency between actions in different timesteps. To address these challenges, inspired by Differential Dynamic Programming (DDP) in optimal control theory, we design a novel Policy Optimization with Model Planning (POMP) algorithm, which incorporates a carefully designed Deep Differential Dynamic Programming (D3P) planner into the model-based RL framework. In D3P planner, (1) to effectively plan in the continuous action space, we construct a locally quadratic programming problem that uses a gradient-based optimization process to replace search. (2) To take the temporal dependency of actions at different timesteps into account, we leverage the updated and latest actions of previous timesteps (i.e., step $1, \cdots, h-1$) to update the action of the current step (i.e., step $h$), instead of updating all actions simultaneously. We theoretically prove the convergence rate for our D3P planner and analyze the effect of the feedback term. In practice, to effectively apply the neural network based D3P planner in reinforcement learning, we leverage the policy network to initialize the action sequence and keep the action update conservative in the planning process. Experiments demonstrate that POMP consistently improves sample efficiency on widely used continuous control tasks. Our code is released at https://github.com/POMP-D3P/POMP-D3P.

Planning Algorithm Based on PSR Models

Survey of Predictive State Representations

A nonlinear model predictive control based on PSO algorithm

Model-Based Robot Learning Control with Uncertainty Directed Exploration

Spacecraft Attitude Maneuver Planning Based on Deep Reinforcement Learning under Complex Constraints

Survey on predictive representations of state

A new path plan method based on hybrid algorithm of reinforcement learning and particle swarm optimization

Planning-Augmented Hierarchical Reinforcement Learning

Provably Efficient UCB-type Algorithms For Learning Predictive State Representations

ProSpec RL: Plan Ahead, then Execute

A new asynchronous reinforcement learning algorithm based on improved parallel PSO

Physics-Informed Model and Hybrid Planning for Efficient Dyna-Style Reinforcement Learning

Predictive Position Control With Stable Control Parameters of Planar Switched Reluctance Motors

Making Better Decision by Directly Planning in Continuous Control

PRRT: An Intelligent Planning Algorithm with Memory

Model-based Reinforcement Learning for Parameterized Action Spaces

Online Planning in POMDPs with State-Requests

A Path Planning Algorithm Based on Deep Reinforcement Learning for Mobile Robots in Unknown Environment

Hybrid Recurrent Models Support Emergent Descriptions for Hierarchical Planning and Control

Research on a Personalized Decision Control Algorithm for Autonomous Vehicles Based on the Reinforcement Learning from Human Feedback Strategy

An Adaptive Online Parameter Control Algorithm For Particle Swarm Optimization Based On Reinforcement Learning