Abstract:Purpose Current reinforcement learning (RL) algorithms are facing issues such as low learning efficiency and poor generalization performance, which significantly limit their practical application in real robots. This paper aims to adopt a hybrid model-based and model-free policy search method with multi-timescale value function tuning, aiming to allow robots to learn complex motion planning skills in multi-goal and multi-constraint environments with a few interactions. Design/methodology/approach A goal-conditioned model-based and model-free search method with multi-timescale value function tuning is proposed in this paper. First, the authors construct a multi-goal, multi-constrained policy optimization approach that fuses model-based policy optimization with goal-conditioned, model-free learning. Soft constraints on states and controls are applied to ensure fast and stable policy iteration. Second, an uncertainty-aware multi-timescale value function learning method is proposed, which constructs a multi-timescale value function network and adaptively chooses the value function planning timescales according to the value prediction uncertainty. It implicitly reduces the value representation complexity and improves the generalization performance of the policy. Findings The algorithm enables physical robots to learn generalized skills in real-world environments through a handful of trials. The simulation and experimental results show that the algorithm outperforms other relevant model-based and model-free RL algorithms. Originality/value This paper combines goal-conditioned RL and the model predictive path integral method into a unified model-based policy search framework, which improves the learning efficiency and policy optimality of motor skill learning in multi-goal and multi-constrained environments. An uncertainty-aware multi-timescale value function learning and selection method is proposed to overcome long horizon problems, improve optimal policy resolution and therefore enhance the generalization ability of goal-conditioned RL.

Model-based Reinforcement Learning with Multi-step Plan Value Estimation

Model-Assisted Reinforcement Learning with Adaptive Ensemble Value Expansion

Model predictive control-based value estimation for efficient reinforcement learning

Towards a Simple Approach to Multi-step Model-based Reinforcement Learning

Theoretically Guaranteed Policy Improvement Distilled from Model-Based Planning

Dynamic Horizon Value Estimation for Model-based Reinforcement Learning

Sample-Efficient Reinforcement Learning with Stochastic Ensemble Value Expansion

Planning with Expectation Models for Control

Value Gradient weighted Model-Based Reinforcement Learning

Dynamic Value Estimation for Single-Task Multi-Scene Reinforcement Learning

The Value Equivalence Principle for Model-Based Reinforcement Learning

Planning with Exploration: Addressing Dynamics Bottleneck in Model-based Reinforcement Learning

Plan to Predict: Learning an Uncertainty-Foreseeing Model for Model-Based Reinforcement Learning.

Hybrid Value Estimation for Off-policy Evaluation and Offline Reinforcement Learning

Experimental evaluations of model-based reinforcement learning combined with MPC

Sample-Efficient Reinforcement Learning Based on Dynamics Models via Meta-policy Optimization

Minimal Value-Equivalent Partial Models for Scalable and Robust Planning in Lifelong Reinforcement Learning

COPlanner: Plan to Roll Out Conservatively but to Explore Optimistically for Model-Based RL

Model-Based Off-Policy Deep Reinforcement Learning with Model-Embedding

A goal-conditioned policy search method with multi-timescale value function tuning

A Reinforcement Learning Based RRT Algorithm with Value Estimation