Abstract:This paper describes a deep reinforcement learning (DRL) approach that won Phase 1 of the Real Robot Challenge (RRC) 2021, and then extends this method to a more difficult manipulation task. The RRC consisted of using a TriFinger robot to manipulate a cube along a specified positional trajectory, but with no requirement for the cube to have any specific orientation. We used a relatively simple reward function, a combination of a goal‐based sparse reward and a distance reward, in conjunction with Hindsight Experience Replay (HER) to guide the learning of the DRL agent (Deep Deterministic Policy Gradient [DDPG]). Our approach allowed our agents to acquire dexterous robotic manipulation strategies in simulation. These strategies were then deployed on the real robot and outperformed all other competition submissions, including those using more traditional robotic control techniques, in the final evaluation stage of the RRC. Here we extend this method, by modifying the task of Phase 1 of the RRC to require the robot to maintain the cube in a particular orientation, while the cube is moved along the required positional trajectory. The requirement to also orient the cube makes the agent less able to learn the task through blind exploration due to increased problem complexity. To circumvent this issue, we make novel use of a Knowledge Transfer (KT) technique that allows the strategies learned by the agent in the original task (which was agnostic to cube orientation) to be transferred to this task (where orientation matters). KT allowed the agent to learn and perform the extended task in the simulator, which improved the average positional deviation from 0.134 to 0.02 m, and average orientation deviation from 142° to 76° during evaluation. This KT concept shows good generalization properties and could be applied to any actor‐critic learning algorithm.

TendencyRL: Multi-stage Discriminative Hints for Efficient Goal-Oriented Reverse Curriculum Learning.

Understanding the Complexity Gains of Contextual Multi-task RL with Curricula

Understanding the Complexity Gains of Single-Task RL with a Curriculum

Trajectory-based Split Hindsight Reverse Curriculum Learning

Density-based Curriculum for Multi-goal Reinforcement Learning with Sparse Rewards

Addressing Reward Engineering For Deep Reinforcement Learning On Multi-Stage Task

Solving Robotic Manipulation With Sparse Reward Reinforcement Learning Via Graph-Based Diversity and Proximity

Quantile Regression Hindsight Experience Replay

Intrinsically Motivated Multi-Goal Reinforcement Learning Using Robotics Environment Integrated with OpenAI Gym

Dexterous robotic manipulation using deep reinforcement learning and knowledge transfer for complex sparse reward‐based tasks

Data-efficient Deep Reinforcement Learning Method Toward Scaling Continuous Robotic Task with Sparse Rewards.

Trying AGAIN instead of Trying Longer: Prior Learning for Automatic Curriculum Learning

Tactical Reward Shaping: Bypassing Reinforcement Learning with Strategy-Based Goals

Reverse Forward Curriculum Learning for Extreme Sample and Demonstration Efficiency in Reinforcement Learning

ACDER: Augmented Curiosity-Driven Experience Replay

Relay Hindsight Experience Replay: Self-guided continual reinforcement learning for sequential object manipulation tasks with sparse rewards

On-Robot Reinforcement Learning with Goal-Contrastive Rewards

Hierarchical reinforcement learning for handling sparse rewards in multi-goal navigation

Sample-Efficient Curriculum Reinforcement Learning for Complex Reward Functions

Learning of Long-Horizon Sparse-Reward Robotic Manipulator Tasks With Base Controllers

Unsupervised Reinforcement Learning for Multi-Task Autonomous Driving: Expanding Skills and Cultivating Curiosity