Abstract:We present a novel solution to the problem of simulation-to-real transfer, which builds on recent advances in robot skill decomposition. Rather than focusing on minimizing the simulation-reality gap, we learn a set of diverse policies that are parameterized in a way that makes them easily reusable. This diversity and parameterization of low-level skills allows us to find a transferable policy that is able to use combinations and variations of different skills to solve more complex, high-level tasks. In particular, we first use simulation to jointly learn a policy for a set of low-level skills, and a "skill embedding" parameterization which can be used to compose them. Later, we learn high-level policies which actuate the low-level policies via this skill embedding parameterization. The high-level policies encode how and when to reuse the low-level skills together to achieve specific high-level tasks. Importantly, our method learns to control a real robot in joint-space to achieve these high-level tasks with little or no on-robot time, despite the fact that the low-level policies may not be perfectly transferable from simulation to real, and that the low-level skills were not trained on any examples of high-level tasks. We illustrate the principles of our method using informative simulation experiments. We then verify its usefulness for real robotics problems by learning, transferring, and composing free-space and contact motion skills on a Sawyer robot using only joint-space control. We experiment with several techniques for composing pre-learned skills, and find that our method allows us to use both learning-based approaches and efficient search-based planning to achieve high-level tasks using only pre-learned skills.

What problem does this paper attempt to address?

The problem that this paper attempts to solve is the **simulation - to - real transfer problem**, especially its application in robot skill learning. Specifically, the author proposes a new method to solve the problem of transferring robot skills from the simulated environment to the real world, instead of focusing on minimizing the gap between simulation and reality. ### Problem Description In the field of robot learning, especially deep reinforcement learning (RL), a major challenge is how to effectively transfer the strategies learned from the simulated environment to real robots. Most deep RL algorithms require a large number of data samples to converge, which makes learning directly on real robots very resource - intensive. Therefore, a promising method is to train deep RL algorithms in a simulated environment that is faster than real - time and then transfer the learned strategies to real robots. However, traditional transfer methods usually rely on reducing the differences between the simulated environment and the real environment (i.e., narrowing the "simulation - reality gap"). Although this method is effective, it often requires complex adjustments and a large amount of experimental verification. In addition, many tasks can be decomposed into multiple low - level subtasks or skills, and these skills can complete more complex high - level tasks in different combinations. If a method can be found to learn these low - level skills and enable them to be flexibly combined in different scenarios, then the transfer process can be greatly simplified and the success rate of transfer can be improved. ### Solutions Proposed in the Paper The paper proposes a novel solution to achieve effective transfer from simulation to reality by learning composable robot skills. Specifically, the author proposes the following points: 1. **Learning Parameterized Representations of Low - Level Skills**: The author first jointly learns a set of low - level skills and their corresponding parameterized representations of "skill embedding" in the simulated environment. This parameterization method makes low - level skills easier to be reused and combined. 2. **Learning High - Level Strategies**: Next, the author learns high - level strategies that activate low - level skills through skill - embedding parameterization. High - level strategies encode how and when to use low - level skills to complete specific high - level tasks. 3. **Transfer and Execution**: Finally, the author transfers the low - level skill strategies, embedding functions, and high - level strategies to real robots and executes these high - level tasks by using only joint - space control. Although low - level skills may not be perfectly transferred from simulation to reality, more complex high - level tasks can still be solved by combining and varying the use of different skills. ### Key Innovation Points - **Skill Decomposition and Combination**: By decomposing complex tasks into multiple low - level skills and learning the combination methods of these skills, the transfer process from simulation to reality is simplified. - **Skill - Embedding Learning**: The concept of skill embedding is introduced, enabling low - level skills to be combined in a parameterized way, which improves the flexibility and reusability of skills. - **Efficient Transfer**: By pre - training low - level skills in the simulated environment and directly applying high - level strategies on real robots, the training time requirement on real robots is reduced. In conclusion, this paper aims to provide a more efficient and flexible simulation - to - real transfer method by learning composable robot skills, thereby promoting the development of the field of robot learning.

Scaling simulation-to-real transfer by learning composable robot skills

GSC: A Graph-Based Skill Composition Framework for Robot Learning

A novel simulation reality closed loop learning framework for autonomous robot skill learning

Modular Deep Q Networks for Sim-to-real Transfer of Visuo-motor Policies

Simulator Predictive Control: Using Learned Task Representations and MPC for Zero-Shot Generalization and Sequencing

Skill Transfer and Discovery for Sim-to-Real Learning: A Representation-Based Viewpoint

Sim-to-Real Transfer with Neural-Augmented Robot Simulation

Overcoming the Sim-to-Real Gap: Leveraging Simulation to Learn to Explore for Real-World RL

Gen2Sim: Scaling up Robot Learning in Simulation with Generative Models

Sim-to-real via latent prediction: Transferring visual non-prehensile manipulation policies

A Data-Efficient Framework for Training and Sim-to-Real Transfer of Navigation Policies

Learning Transferable Motor Skills with Hierarchical Latent Mixture Policies

Diff-Transfer: Model-based Robotic Manipulation Skill Transfer via Differentiable Physics Simulation

Sim-to-Real Transfer Learning using Robustified Controllers in Robotic Tasks involving Complex Dynamics

Soft Robots Learn to Crawl: Jointly Optimizing Design and Control with Sim-to-Real Transfer

GenSim2: Scaling Robot Data Generation with Multi-modal and Reasoning LLMs

TRANSIC: Sim-to-Real Policy Transfer by Learning from Online Correction

Scaling Up and Distilling Down: Language-Guided Robot Skill Acquisition

Scaling Cross-Embodied Learning: One Policy for Manipulation, Navigation, Locomotion and Aviation

Human-Robot Skill Transfer with Enhanced Compliance via Dynamic Movement Primitives

AdaptSim: Task-Driven Simulation Adaptation for Sim-to-Real Transfer