Abstract:Reinforcement Learning has been shown to have a great potential for robotics. It demonstrated the capability to solve complex manipulation and locomotion tasks, even by learning end-to-end policies that operate directly on visual input, removing the need for custom perception systems. However, for practical robotics applications, its scarce sample efficiency, the need for huge amounts of resources, data, and computation time can be an insurmountable obstacle. One potential solution to this sample efficiency issue is the use of simulated environments. However, the discrepancy in visual and physical characteristics between reality and simulation, namely the sim-to-real gap, often significantly reduces the real-world performance of policies trained within a simulator. In this work we propose a sim-to-real technique that trains a Soft-Actor Critic agent together with a decoupled feature extractor and a latent-space dynamics model. The decoupled nature of the method allows to independently perform the sim-to-real transfer of feature extractor and control policy, and the presence of the dynamics model acts as a constraint on the latent representation when finetuning the feature extractor on real-world data. We show how this architecture can allow the transfer of a trained agent from simulation to reality without retraining or finetuning the control policy, but using real-world data only for adapting the feature extractor. By avoiding training the control policy in the real domain we overcome the need to apply Reinforcement Learning on real-world data, instead, we only focus on the unsupervised training of the feature extractor, considerably reducing real-world experience collection requirements. We evaluate the method on sim-to-sim and sim-to-real transfer of a policy for table-top robotic object pushing. We demonstrate how the method is capable of adapting to considerable variations in the task observations, such as changes in point-of-view, colors, and lighting, all while substantially reducing the training time with respect to policies trained directly in the real.

Bridging the simulation-to-real gap of depth images for deep reinforcement learning

Modular Deep Q Networks for Sim-to-real Transfer of Visuo-motor Policies

Sim-to-Real Strategy for Spatially Aware Robot Navigation in Uneven Outdoor Environments

A Data-Efficient Framework for Training and Sim-to-Real Transfer of Navigation Policies

Autonomous Single-Image Drone Exploration With Deep Reinforcement Learning and Mixed Reality

Sim-to-real via latent prediction: Transferring visual non-prehensile manipulation policies

A Platform-Agnostic Deep Reinforcement Learning Framework for Effective Sim2Real Transfer towards Autonomous Driving

A novel simulation reality closed loop learning framework for autonomous robot skill learning

Flying Through a Narrow Gap Using End-to-end Deep Reinforcement Learning Augmented with Curriculum Learning and Sim2Real

Close the Sim2real Gap via Physically-based Structured Light Synthetic Data Simulation

RL-CycleGAN: Reinforcement Learning Aware Simulation-To-Real

RL-GSBridge: 3D Gaussian Splatting Based Real2Sim2Real Method for Robotic Manipulation Learning

Overcoming the Sim-to-Real Gap: Leveraging Simulation to Learn to Explore for Real-World RL

A Sim-to-Real Pipeline for Deep Reinforcement Learning for Autonomous Robot Navigation in Cluttered Rough Terrain

Virtual to Real Reinforcement Learning for Autonomous Driving

Real–Sim–Real Transfer for Real-World Robot Control Policy Learning with Deep Reinforcement Learning

Domain-Transferred Synthetic Data Generation for Improving Monocular Depth Estimation

From Naturalistic Traffic Data to Learning-Based Driving Policy: A Sim-to-Real Study

Sim-Real Joint Reinforcement Transfer for 3D Indoor Navigation

Real2Sim or Sim2Real: Robotics Visual Insertion using Deep Reinforcement Learning and Real2Sim Policy Adaptation