Language-driven Grasp Detection

An Dinh Vuong,Minh Nhat Vu,Baoru Huang,Nghia Nguyen,Hieu Le,Thieu Vo,Anh Nguyen

2024-06-14

Abstract:Grasp detection is a persistent and intricate challenge with various industrial applications. Recently, many methods and datasets have been proposed to tackle the grasp detection problem. However, most of them do not consider using natural language as a condition to detect the grasp poses. In this paper, we introduce Grasp-Anything++, a new language-driven grasp detection dataset featuring 1M samples, over 3M objects, and upwards of 10M grasping instructions. We utilize foundation models to create a large-scale scene corpus with corresponding images and grasp prompts. We approach the language-driven grasp detection task as a conditional generation problem. Drawing on the success of diffusion models in generative tasks and given that language plays a vital role in this task, we propose a new language-driven grasp detection method based on diffusion models. Our key contribution is the contrastive training objective, which explicitly contributes to the denoising process to detect the grasp pose given the language instructions. We illustrate that our approach is theoretically supportive. The intensive experiments show that our method outperforms state-of-the-art approaches and allows real-world robotic grasping. Finally, we demonstrate our large-scale dataset enables zero-short grasp detection and is a challenging benchmark for future work. Project website: <a class="link-external link-https" href="https://airvlab.github.io/grasp-anything/" rel="external noopener nofollow">this https URL</a>

Computer Vision and Pattern Recognition

What problem does this paper attempt to address?

The problem that this paper attempts to solve is language - driven grasp detection. Specifically, although existing grasp detection methods perform well in industrial applications, most of them ignore using natural language as a condition to detect grasping postures. This means that when a robot is performing a grasping task, it usually cannot determine the specific grasping action according to human natural language instructions, which limits the robot's application ability in the real world.

To overcome this challenge, the author proposes a new dataset and method, namely Grasp - Anything++, which contains 1 million samples, more than 3 million objects and more than 10 million grasping instructions. By using the base model to generate a large - scale scene corpus and its corresponding images and grasping hints, the author regards the language - driven grasp detection task as a conditional generation problem and proposes a new language - driven grasp detection method based on the diffusion model. The key contribution of this method lies in the contrastive training objective, which explicitly promotes the denoising process to detect grasping postures according to language instructions.

The main contributions of the paper can be summarized as follows:
1. Propose Grasp - Anything++, which is a large - scale language - driven grasp detection dataset.
2. Propose a method based on the diffusion model, whose training objective explicitly promotes the denoising process to detect grasping postures.
3. Experimental results show that this method is superior to existing methods and can be successfully applied in real - world robot grasping tasks.

Through these contributions, the paper aims to bridge the gap between language, vision and control and promote the development of language - driven robotics.