Abstract:Reinforcement learning recently has achieved impressive success in allowing robots to learn complex motor skills in simulation environments. However, most of these successes are difficult to transfer to physical robots since current algorithms require lots of practical training and complex sim-to-real transfer skills. To improve the learning efficiency and adaptability of physical robots, this article proposes a guided model-based policy search (GMBPS) algorithm inspired by a hypothetical model-free (MF) and model-based (MB) actor-critic brain implementation. This approach bridges the gap between MF and MB control processes, overcoming the suboptimality of MB methods and speeding up the learning rate of MF methods. Additionally, a one-step predictive control framework is proposed for minimizing the impact of delayed sensorimotor information in real-world tasks. This helps to accurately control the action cycle time and ensures the feasibility of MB planning for physical robots. The simulation and experimental results demonstrate that the proposed approach enables a 6-DOF UR5e robot arm to learn various reaching tasks in a few minutes with better policies and higher learning efficiency. Note to Practitioners —Reinforcement learning is becoming a popular framework that allows robots to learn complex motor skills without building analytical models of controlled plants. However, low learning efficiency severely limits its application in practical robots, where robots have to quickly adapt to dynamically changing environments in micro-data situations. To solve the inefficiency problem of physical robot learning from scratch, this paper proposes a MF and MB fusion control algorithm inspired by a hypothetical MF and MB actor-critic brain implementation. The motion decision process is modeled as an optimization problem with inequality constraints. The global MF value function is incorporated into the MB objective function, extending the short-term optimization into a long-term version to overcome the suboptimality of conventional MB methods. The MB policy is searched based on the quadratic penalty method with the guide of the MF policy, which helps improve the quality of policy at every decision-making step. Moreover, since the model dynamics is fitted by a probabilistic neural network, the proposed method is not only applicable to joint-driven robots but also provides a feasible solution for the control of various robotic systems with complex dynamics, such as soft robots and musculoskeletal robots.

Towards a Reinforcement Learning Environment Toolbox for Intelligent Electric Motor Control

Toward a Reinforcement Learning Environment Toolbox for Intelligent Electric Motor Control

Reinforcement Learning for Motor Control: A Comprehensive Review

Learning with Training Wheels: Speeding up Training with a Simple Controller for Deep Reinforcement Learning

Gym-ANM: Open-source software to leverage reinforcement learning for power system management in research and education

ModelicaGym: Applying Reinforcement Learning to Modelica Models

Controlgym: Large-Scale Control Environments for Benchmarking Reinforcement Learning Algorithms

Learning to Control DC Motor for Micromobility in Real Time with Reinforcement Learning

Reinforcement learning building control approach harnessing imitation learning

Path Following and Stabilisation of a Bicycle Model using a Reinforcement Learning Approach

Guided Model-Based Policy Search Method for Fast Motor Learning of Robots with Learned Dynamics

Adaptive PI Controller Based on a Reinforcement Learning Algorithm for Speed Control of a DC Motor

Experimental Study on Reinforcement Learning-based Control of an Acrobot

CaiRL: A High-Performance Reinforcement Learning Environment Toolkit

An open-source software framework for reinforcement learning-based control of tracked robots in simulated indoor environments

MotorNet: a Python toolbox for controlling differentiable biomechanical effectors with artificial neural networks

Technical Report on Reinforcement Learning Control on the Lucas-Nülle Inverted Pendulum

Neural Internal Model Control: Learning a Robust Control Policy via Predictive Error Feedback

Model-Based Reinforcement Learning Inspired by Augmented PD for Robotic Control

SERL: A Software Suite for Sample-Efficient Robotic Reinforcement Learning

Robot Air Hockey: A Manipulation Testbed for Robot Learning with Reinforcement Learning