ASID: Active Exploration for System Identification in Robotic Manipulation

Marius Memmel,Andrew Wagenmaker,Chuning Zhu,Patrick Yin,Dieter Fox,Abhishek Gupta

2024-06-27

Abstract:Model-free control strategies such as reinforcement learning have shown the ability to learn control strategies without requiring an accurate model or simulator of the world. While this is appealing due to the lack of modeling requirements, such methods can be sample inefficient, making them impractical in many real-world domains. On the other hand, model-based control techniques leveraging accurate simulators can circumvent these challenges and use a large amount of cheap simulation data to learn controllers that can effectively transfer to the real world. The challenge with such model-based techniques is the requirement for an extremely accurate simulation, requiring both the specification of appropriate simulation assets and physical parameters. This requires considerable human effort to design for every environment being considered. In this work, we propose a learning system that can leverage a small amount of real-world data to autonomously refine a simulation model and then plan an accurate control strategy that can be deployed in the real world. Our approach critically relies on utilizing an initial (possibly inaccurate) simulator to design effective exploration policies that, when deployed in the real world, collect high-quality data. We demonstrate the efficacy of this paradigm in identifying articulation, mass, and other physical parameters in several challenging robotic manipulation tasks, and illustrate that only a small amount of real-world data can allow for effective sim-to-real transfer. Project website at <a class="link-external link-https" href="https://weirdlabuw.github.io/asid" rel="external noopener nofollow">this https URL</a>

Robotics,Machine Learning,Systems and Control

What problem does this paper attempt to address?

The paper aims to address the problem of sim-to-real transfer in robotic operations. Specifically, the paper proposes a method called ASID (Active Exploration for System Identification) to achieve this goal through the following steps:

1. **Active Exploration**: Utilize an initial (possibly inaccurate) simulator to design effective exploration strategies and collect high-quality data in the real environment.
2. **System Identification**: Use the data collected in the real environment to calibrate the simulator parameters, thereby obtaining a more accurate simulation environment.
3. **Policy Training and Deployment**: Train policies on the updated simulator and transfer these policies to the real environment with zero samples.

The core idea of the paper is that although directly deploying policies trained in simulation to the real world often performs poorly, learning accurate simulator parameters through initial real-world exploration can significantly improve the sim-to-real transfer. This method is particularly suitable for complex robotic tasks that require high-precision physical parameters, such as ball manipulation, laptop hinge identification, and pole balancing.

Experimental results show that the ASID method can effectively identify unknown parameters in various challenging robotic tasks and successfully transfer the policies trained in simulation to real-world applications.