DEFT: Dexterous Fine-Tuning for Real-World Hand Policies

Aditya Kannan,Kenneth Shaw,Shikhar Bahl,Pragna Mannam,Deepak Pathak

2023-12-12

Abstract:Dexterity is often seen as a cornerstone of complex manipulation. Humans are able to perform a host of skills with their hands, from making food to operating tools. In this paper, we investigate these challenges, especially in the case of soft, deformable objects as well as complex, relatively long-horizon tasks. However, learning such behaviors from scratch can be data inefficient. To circumvent this, we propose a novel approach, DEFT (DExterous Fine-Tuning for Hand Policies), that leverages human-driven priors, which are executed directly in the real world. In order to improve upon these priors, DEFT involves an efficient online optimization procedure. With the integration of human-based learning and online fine-tuning, coupled with a soft robotic hand, DEFT demonstrates success across various tasks, establishing a robust, data-efficient pathway toward general dexterous manipulation. Please see our website at <a class="link-external link-https" href="https://dexterous-finetuning.github.io" rel="external noopener nofollow">this https URL</a> for video results.

Robotics,Artificial Intelligence,Computer Vision and Pattern Recognition,Machine Learning

What problem does this paper attempt to address?

The problem this paper attempts to address is how to efficiently learn complex and dexterous tasks in the real world, especially when dealing with soft, deformable objects and more complex, time-spanning tasks. Traditional methods of learning these behaviors from scratch are data inefficient, so the paper proposes a new method called DEFT (DExterous Fine-Tuning for Hand Policies). DEFT leverages human behavior priors and combines them with online optimization techniques to quickly learn and improve hand manipulation strategies in real environments, enabling the successful execution of various tasks.

Specifically, the main contributions of the DEFT method include:
1. **Utilizing human video data**: By analyzing a large number of human videos on the internet, learning human hand movements and grasping techniques as priors for the robot's behavior.
2. **Online optimization**: Conducting efficient online optimization in real environments, gradually improving the success rate of task completion through sampling and iterative refinement.
3. **Soft robotic hand**: Using a soft humanoid hand that can better handle soft and fragile objects and is more robust in actual operations.

Through these methods, DEFT can succeed in various tasks, including picking up a cup, pouring water, opening a drawer, picking up a spoon, scooping grapes, stirring, flipping a bagel, and squeezing a lemon. Experimental results show that DEFT has a higher success rate and better generalization ability compared to methods that rely solely on real-world learning or only use prior models.