RealDex: Towards Human-like Grasping for Robotic Dexterous Hand

Yumeng Liu,Yaxun Yang,Youzhuo Wang,Xiaofei Wu,Jiamin Wang,Yichen Yao,Sören Schwertfeger,Sibei Yang,Wenping Wang,Jingyi Yu,Xuming He,Yuexin Ma

DOI: https://doi.org/10.48550/arXiv.2402.13853

2024-02-21

Abstract:In this paper, we introduce RealDex, a pioneering dataset capturing authentic dexterous hand grasping motions infused with human behavioral patterns, enriched by multi-view and multimodal visual data. Utilizing a teleoperation system, we seamlessly synchronize human-robot hand poses in real time. This collection of human-like motions is crucial for training dexterous hands to mimic human movements more naturally and precisely. RealDex holds immense promise in advancing humanoid robot for automated perception, cognition, and manipulation in real-world scenarios. Moreover, we introduce a cutting-edge dexterous grasping motion generation framework, which aligns with human experience and enhances real-world applicability through effectively utilizing Multimodal Large Language Models. Extensive experiments have demonstrated the superior performance of our method on RealDex and other open datasets. The complete dataset and code will be made available upon the publication of this work.

Robotics,Artificial Intelligence,Computer Vision and Pattern Recognition

What problem does this paper attempt to address?

The problem that this paper attempts to solve is to develop a method that enables the robotic dexterous hand to grasp objects like humans. Specifically, existing methods have deficiencies in simulating human grasping behaviors, mainly reflected in the following aspects:

1. **Scarcity of data and limitations of synthetic data**: Many existing methods rely on synthetic datasets, which are generated by optimization methods and lack real - world data, resulting in poor performance of the model in practical applications.
2. **Limitations of reinforcement learning (RL)**: Although reinforcement learning is widely used to train the grasping behavior of dexterous hands, its reward function is artificially designed and it is difficult to comprehensively capture human behavior habits, resulting in grasping actions that are not natural and reasonable enough.
3. **Lack of dynamic grasping sequences**: Most of the existing datasets only contain static grasping postures and lack dynamic grasping motion sequences, which limits the research on complex grasping actions.

To solve these problems, the paper introduces a new dataset named RealDex, which contains a large number of real dexterous - hand grasping actions that match human behavior patterns. In addition, the paper also proposes a framework based on multi - modal large language models (MLLMs) to generate dexterous - hand actions that are closer to human grasping behaviors. By combining real data and advanced generation models, this research aims to improve the application performance of robotic dexterous hands in the real world.

### Specific problem summary:
- **Data authenticity**: How to obtain and utilize human grasping action data in the real world to improve the grasping ability of robotic dexterous hands.
- **Behavior pattern modeling**: How to better model and imitate human grasping behavior patterns to make robot grasping more natural and in line with human habits.
- **Dynamic grasping sequences**: How to generate and utilize dynamic grasping action sequences to achieve more complex grasping tasks.

By solving these problems, this research hopes to promote the progress of robotic dexterous hands in automatic perception, cognition and operation, especially in applications in service and medical fields.