Bunny-VisionPro: Real-Time Bimanual Dexterous Teleoperation for Imitation Learning

Runyu Ding,Yuzhe Qin,Jiyue Zhu,Chengzhe Jia,Shiqi Yang,Ruihan Yang,Xiaojuan Qi,Xiaolong Wang

2024-07-03

Abstract:Teleoperation is a crucial tool for collecting human demonstrations, but controlling robots with bimanual dexterous hands remains a challenge. Existing teleoperation systems struggle to handle the complexity of coordinating two hands for intricate manipulations. We introduce Bunny-VisionPro, a real-time bimanual dexterous teleoperation system that leverages a VR headset. Unlike previous vision-based teleoperation systems, we design novel low-cost devices to provide haptic feedback to the operator, enhancing immersion. Our system prioritizes safety by incorporating collision and singularity avoidance while maintaining real-time performance through innovative designs. Bunny-VisionPro outperforms prior systems on a standard task suite, achieving higher success rates and reduced task completion times. Moreover, the high-quality teleoperation demonstrations improve downstream imitation learning performance, leading to better generalizability. Notably, Bunny-VisionPro enables imitation learning with challenging multi-stage, long-horizon dexterous manipulation tasks, which have rarely been addressed in previous work. Our system's ability to handle bimanual manipulations while prioritizing safety and real-time performance makes it a powerful tool for advancing dexterous manipulation and imitation learning.

Robotics,Computer Vision and Pattern Recognition,Machine Learning

What problem does this paper attempt to address?

The problem this paper attempts to address is:

Existing teleoperation systems face challenges in handling complex coordination tasks with dual-arm dexterous hands. Specifically, controlling robots to perform fine movements that require coordinated operation of both hands remains very difficult, and current systems struggle to cope with this complexity. To address this issue, the authors propose Bunny-VisionPro, a real-time teleoperation system for dual-arm dexterous hands.

### Main Issues
1. **Complex Dual-Arm Coordination**: Existing teleoperation systems find it difficult to handle complex tasks that require coordinated operation of both hands.
2. **Real-Time Performance**: Ensuring the system operates efficiently in real-time environments to avoid imprecise operations caused by delays.
3. **Safety**: Preventing collisions and singularities during operation to ensure the system's safety and stability.
4. **Collection of High-Quality Demonstration Data**: Providing high-quality demonstration data for imitation learning to improve the generalization ability of learning algorithms.

### Solution
Bunny-VisionPro addresses the above issues through the following innovative modules:
1. **Arm Motion Control Module**: Real-time handling of robot singularities and collision avoidance to ensure safe and smooth operation of the arms.
2. **Dexterous Hand Relocation Module**: Accurately mapping human finger movements to the robot hand, supporting real-time relocation of robot hands with cyclic joints (such as four-bar linkages).
3. **Haptic Feedback Module**: Using low-cost Eccentric Rotating Mass (ERM) vibration motors to provide haptic feedback, enhancing the operator's sense of immersion and realism.

### Experimental Results
- In standard task suites, Bunny-VisionPro demonstrated a higher success rate and shorter task completion time compared to previous systems.
- The collected high-quality demonstration data significantly improved the performance of downstream imitation learning, particularly in multi-stage, long-duration dexterous manipulation tasks.

### Conclusion
Through innovative design and modular architecture, Bunny-VisionPro successfully addresses the key challenges in dual-arm dexterous hand teleoperation, providing a powerful tool for advancing dexterous manipulation and imitation learning.