RoboTwin: Dual-Arm Robot Benchmark with Generative Digital Twins (early version)

Yao Mu,Tianxing Chen,Shijia Peng,Zanxin Chen,Zeyu Gao,Yude Zou,Lunkai Lin,Zhiqiang Xie,Ping Luo

2024-09-05

Abstract:Effective collaboration of dual-arm robots and their tool use capabilities are increasingly important areas in the advancement of robotics. These skills play a significant role in expanding robots' ability to operate in diverse real-world environments. However, progress is impeded by the scarcity of specialized training data. This paper introduces RoboTwin, a novel benchmark dataset combining real-world teleoperated data with synthetic data from digital twins, designed for dual-arm robotic scenarios. Using the COBOT Magic platform, we have collected diverse data on tool usage and human-robot interaction. We present a innovative approach to creating digital twins using AI-generated content, transforming 2D images into detailed 3D models. Furthermore, we utilize large language models to generate expert-level training data and task-specific pose sequences oriented toward functionality. Our key contributions are: 1) the RoboTwin benchmark dataset, 2) an efficient real-to-simulation pipeline, and 3) the use of language models for automatic expert-level data generation. These advancements are designed to address the shortage of robotic training data, potentially accelerating the development of more capable and versatile robotic systems for a wide range of real-world applications. The project page is available at <a class="link-external link-https" href="https://robotwin-benchmark.github.io/early-version/" rel="external noopener nofollow">this https URL</a>

Robotics,Artificial Intelligence,Computation and Language

What problem does this paper attempt to address?

The paper attempts to address the issue of the lack of specialized high-quality training data in the areas of dual-arm robot operation and tool usage capabilities. Specifically, the paper introduces the RoboTwin benchmark dataset, which combines real-world teleoperation data with synthetic data generated by digital twins, aiming to provide comprehensive data support for dual-arm robot scenarios. By using the COBOT Magic platform, researchers collected diverse data on tool usage and human-robot interaction and proposed an innovative method to convert 2D images into detailed 3D models using AI-generated content. Additionally, the paper leverages large language models to generate expert-level training data and function-oriented task-specific posture sequences. These contributions aim to address the shortage of robot training data, thereby accelerating the development of more powerful and versatile robotic systems to adapt to a wide range of real-world application environments.