Abstract:This paper proposes a novel hierarchical reinforcement learning (HRL) framework of complex manipulation tasks which integrates the human prior knowledge. The framework involves the following steps: $ \textbf{1)} $ The manipulation process is divided into several stages based on human prior knowledge. $ \textbf{2)} $ Transition conditions between stages are determined in the form of “if-then” rules. $ \textbf{3)} $ The key features of each stage are selected, and the corresponding control policies are designed via human knowledge. $ \textbf{4)} $ The policy gradient with parameter-based exploration (PGPE) method is employed to optimize the policy parameters because it does not require the policy to be derivable for the parameters. To increase the convergence speed of this framework, importance sampling and adaptive adjustment of exploration variance are employed to improve it. $ \textbf{5)} $ On this basis, to facilitate the transfer of simulation results to practical experiments, half sim-to-real method is presented, which fully utilizes the simulation results, and the differences between simulation and experimental environments are considered. Simulation and experimental studies show that our framework can deal with the peg-hole-insertion task with a high quality in less than 1600 episodes and can safely adapt the skill into the practical scene with little iterations, which verify the efficiency of the presented method. Note to Practitioners—Intelligent robots will become the right assistants of human beings in the future, especially in various areas of complex manipulation occasions. The important premise is that the robots should have certain ability of complex manipulation skill learning. Complex manipulation tasks can be decomposed into multiple stages, and HRL is a suitable and efficient method for solving this kind of problems. This paper proposes a novel HRL framework which can better integrate the human prior knowledge. In addition, improved PGPE method is proposed to obtain the optimized policy parameters more quickly. More importantly, a novel half sim-to-real transfer method is presented to better integrate the simulation and experiment results. This provides a common paradigm, which leverages the simulation results to reduce the interaction between robot and practical environment and then utilizes the experiment results to further optimize the policy parameters.

PRRM: An Efficient Framework for Learning Multi-step Robotic Manipulation Tasks

Leveraging the Efficiency of Multi-Task Robot Manipulation Via Task-Evoked Planner and Reinforcement Learning

A Reinforcement Learning-Based Framework for Robot Manipulation Skill Acquisition

Efficient Reinforcement Learning Method for Multi-Phase Robot Manipulation Skill Acquisition Via Human Knowledge, Model-Based, and Model-Free Methods

A learning framework for semantic reach-to-grasp tasks integrating machine learning and optimization.

Hierarchical Reinforcement Learning Integrating With Human Knowledge for Practical Robot Skill Learning in Complex Multi-Stage Manipulation

Efficient Stacking and Grasping in Unstructured Environments

A Task-Adaptive Deep Reinforcement Learning Framework for Dual-Arm Robot Manipulation

Learning to combine primitive skills: A step towards versatile robotic manipulation

Multi-Stage Reinforcement Learning for Non-Prehensile Manipulation

Multi-Phase Multi-Objective Dexterous Manipulation with Adaptive Hierarchical Curriculum

Prehensile and Non-Prehensile Robotic Pick-and-Place of Objects in Clutter using Deep Reinforcement Learning

RLAfford: End-to-End Affordance Learning for Robotic Manipulation

Hierarchical Visual Policy Learning for Long-Horizon Robot Manipulation in Densely Cluttered Scenes

Safety Guaranteed Manipulation Based on Reinforcement Learning Planner and Model Predictive Control Actor

Learning Extrinsic Dexterity with Parameterized Manipulation Primitives

RPRG: Toward Real-time Robotic Perception, Reasoning and Grasping with One Multi-task Convolutional Neural Network.

Precise and Dexterous Robotic Manipulation via Human-in-the-Loop Reinforcement Learning

A Knowledge-Based Task Planning Approach for Robot Multi-Task Manipulation

Proactive Action Visual Residual Reinforcement Learning for Contact-Rich Tasks Using a Torque-Controlled Robot

Task-Driven Reinforcement Learning with Action Primitives for Long-Horizon Manipulation Skills.