Abstract:Reinforcement learning recently has achieved impressive success in allowing robots to learn complex motor skills in simulation environments. However, most of these successes are difficult to transfer to physical robots since current algorithms require lots of practical training and complex sim-to-real transfer skills. To improve the learning efficiency and adaptability of physical robots, this article proposes a guided model-based policy search (GMBPS) algorithm inspired by a hypothetical model-free (MF) and model-based (MB) actor-critic brain implementation. This approach bridges the gap between MF and MB control processes, overcoming the suboptimality of MB methods and speeding up the learning rate of MF methods. Additionally, a one-step predictive control framework is proposed for minimizing the impact of delayed sensorimotor information in real-world tasks. This helps to accurately control the action cycle time and ensures the feasibility of MB planning for physical robots. The simulation and experimental results demonstrate that the proposed approach enables a 6-DOF UR5e robot arm to learn various reaching tasks in a few minutes with better policies and higher learning efficiency. Note to Practitioners —Reinforcement learning is becoming a popular framework that allows robots to learn complex motor skills without building analytical models of controlled plants. However, low learning efficiency severely limits its application in practical robots, where robots have to quickly adapt to dynamically changing environments in micro-data situations. To solve the inefficiency problem of physical robot learning from scratch, this paper proposes a MF and MB fusion control algorithm inspired by a hypothetical MF and MB actor-critic brain implementation. The motion decision process is modeled as an optimization problem with inequality constraints. The global MF value function is incorporated into the MB objective function, extending the short-term optimization into a long-term version to overcome the suboptimality of conventional MB methods. The MB policy is searched based on the quadratic penalty method with the guide of the MF policy, which helps improve the quality of policy at every decision-making step. Moreover, since the model dynamics is fitted by a probabilistic neural network, the proposed method is not only applicable to joint-driven robots but also provides a feasible solution for the control of various robotic systems with complex dynamics, such as soft robots and musculoskeletal robots.

A Central Motor System Inspired Pre-training Reinforcement Learning for Robotic Control

Mastering Robot Control Through Point-based Reinforcement Learning with Pre-training.

Hybrid Reinforcement Learning Based on Human Preference and Advice for Efficient Robot Skill Learning

Guided Model-Based Policy Search Method for Fast Motor Learning of Robots with Learned Dynamics

Decentralized Motor Skill Learning for Complex Robotic Systems

Hierarchical Reinforcement Learning Integrating With Human Knowledge for Practical Robot Skill Learning in Complex Multi-Stage Manipulation

Robot Learning System Based on Adaptive Neural Control and Dynamic Movement Primitives.

Efficient Reinforcement Learning Method for Multi-Phase Robot Manipulation Skill Acquisition Via Human Knowledge, Model-Based, and Model-Free Methods

Learning to combine primitive skills: A step towards versatile robotic manipulation

Enhancing Task Performance of Learned Simplified Models via Reinforcement Learning

Model Predictive Actor-Critic: Accelerating Robot Skill Acquisition with Deep Reinforcement Learning

Integrating Reinforcement Learning and Learning From Demonstrations to Learn Nonprehensile Manipulation

Integrating Controllable Motion Skills from Demonstrations

Robot Intelligent Trajectory Planning Based on PCM Guided Reinforcement Learning

Spatiotemporal Predictive Pre-training for Robotic Motor Control

Real-World Robot Reaching Skill Learning Based On Deep Reinforcement Learning

Task-Oriented Deep Reinforcement Learning for Robotic Skill Acquisition and Control

A data-efficient goal-directed deep reinforcement learning method for robot visuomotor skill

Precise and Dexterous Robotic Manipulation via Human-in-the-Loop Reinforcement Learning

Dynamic Movement Primitives Based Robot Skills Learning

Simulator Predictive Control: Using Learned Task Representations and MPC for Zero-Shot Generalization and Sequencing