Bimanual Dexterity for Complex Tasks

Kenneth Shaw,Yulong Li,Jiahui Yang,Mohan Kumar Srirama,Ray Liu,Haoyu Xiong,Russell Mendonca,Deepak Pathak
2024-11-21
Abstract:To train generalist robot policies, machine learning methods often require a substantial amount of expert human teleoperation data. An ideal robot for humans collecting data is one that closely mimics them: bimanual arms and dexterous hands. However, creating such a bimanual teleoperation system with over 50 DoF is a significant challenge. To address this, we introduce Bidex, an extremely dexterous, low-cost, low-latency and portable bimanual dexterous teleoperation system which relies on motion capture gloves and teacher arms. We compare Bidex to a Vision Pro teleoperation system and a SteamVR system and find Bidex to produce better quality data for more complex tasks at a faster rate. Additionally, we show Bidex operating a mobile bimanual robot for in the wild tasks. The robot hands (5k USD) and teleoperation system (7k USD) is readily reproducible and can be used on many robot arms including two xArms (16k USD). Website at <a class="link-external link-https" href="https://bidex-teleop.github.io/" rel="external noopener nofollow">this https URL</a>
Robotics,Artificial Intelligence,Computer Vision and Pattern Recognition,Machine Learning
What problem does this paper attempt to address?
The problem that this paper attempts to solve is how to efficiently and accurately collect high - quality expert data for training bimanual dexterous robots. Specifically, the researchers aim to develop a low - cost, easy - to - set - up - and - use, low - latency, and high - precision tele - operation system, enabling operators to easily collect high - quality data in complex tasks to train robot strategies. ### Core of the Problem 1. **High - Dimensional Control Challenges**: Bimanual dexterous robots have more than 50 degrees of freedom (DoF), which imposes extremely high requirements on the precision and response speed of the tele - operation system. 2. **Limitations of Existing Methods**: - Although VR headsets are easy to use, wrist and finger tracking is inaccurate. - The SteamVR system relies on external laser lighthouses, is not suitable for mobile robot setups, and has problems of high latency and occasional disconnections. - High - fidelity motion - capture systems (such as Vicon or Optitrack) are very expensive and difficult to set up. ### Solution To this end, the researchers introduced the BiDex system, an extremely dexterous, low - cost, low - latency, and portable bimanual dexterous tele - operation system. The main features of BiDex include: - **Hand Tracking**: Use Manus Meta gloves for high - precision finger tracking, avoiding the occlusion and illumination problems common in visual methods. - **Arm Tracking**: Joint - level tele - operation inspired by GELLO, achieving accurate tracking of robot arms through lightweight teacher arms. - **Cost - Effectiveness**: The cost of the entire system (including two Manus gloves and the arm tele - operation system) is approximately $6,000. Adding the cost of the robot arms and hands, the total cost is less than $30,000. - **Portability and Adaptability**: No external tracking devices are required, and it is suitable for desktop and mobile environments. ### Experimental Verification The researchers verified the effectiveness of BiDex through a series of experiments, including comparisons with the Vision Pro VR and SteamVR systems. The results show that BiDex performs excellently in terms of completion rate and time, especially in complex tasks such as pouring water from a bottle and stacking cups. ### Summary The BiDex system overcomes the deficiencies of existing tele - operation systems in collecting data for high - degree - of - freedom bimanual dexterous robots, providing a more reliable and faster method to collect high - quality expert data, thus providing a better foundation for behavior cloning and reinforcement learning.