Abstract:This paper studies the task of any objects grasping from the known categories by free-form language instructions. This task demands the technique in computer vision, natural language processing, and robotics. We bring these disciplines together on this open challenge, which is essential to human-robot interaction. Critically, the key challenge lies in inferring the category of objects from linguistic instructions and accurately estimating the 6-DoF information of unseen objects from the known classes. In contrast, previous works focus on inferring the pose of object candidates at the instance level. This significantly limits its applications in real-world <a class="link-external link-http" href="http://scenarios.In" rel="external noopener nofollow">this http URL</a> this paper, we propose a language-guided 6-DoF category-level object localization model to achieve robotic grasping by comprehending human intention. To this end, we propose a novel two-stage method. Particularly, the first stage grounds the target in the RGB image through language description of names, attributes, and spatial relations of objects. The second stage extracts and segments point clouds from the cropped depth image and estimates the full 6-DoF object pose at category-level. Under such a manner, our approach can locate the specific object by following human instructions, and estimate the full 6-DoF pose of a category-known but unseen instance which is not utilized for training the model. Extensive experimental results show that our method is competitive with the state-of-the-art language-conditioned grasp method. Importantly, we deploy our approach on a physical robot to validate the usability of our framework in real-world applications. Please refer to the supplementary for the demo videos of our robot experiments.

Language-Guided Category Push–Grasp Synergy Learning in Clutter by Efficiently Perceiving Object Manipulation Space

Position-aware pushing and grasping synergy with deep reinforcement learning in clutter

Vision-Based Categorical Object Pose Estimation and Manipulation.

Learning 6-DoF Object Poses to Grasp Category-level Objects by Language Instructions

Infusing Multi-Source Heterogeneous Knowledge for Language-Conditioned Segmentation and Grasping

A Joint Modeling of Vision-Language-Action for Target-oriented Grasping in Clutter

Learning Target-Oriented Push-Grasping Synergy in Clutter with Action Space Decoupling

ThinkGrasp: A Vision-Language System for Strategic Part Grasping in Clutter

Efficient and Robust Robotic Grasping in Cluttered Scenes: A Point Cloud-Based Approach with Heuristic Evaluation.

Language Guided Grasping of Unknown Concepts Based on Knowledge System.

Self-Supervised Learning for Joint Pushing and Grasping Policies in Highly Cluttered Environments

Learning Dual-Arm Push and Grasp Synergy in Dense Clutter

Learning Goal-Directed Object Pushing in Cluttered Scenes with Location-Based Attention

Synergistic Pushing and Grasping for Enhanced Robotic Manipulation Using Deep Reinforcement Learning

Robot Learning of Shifting Objects for Grasping in Cluttered Environments

Gated Self Attention Network for Efficient Grasping of Target Objects in Stacked Scenarios

Grounding Object Relations in Language-Conditioned Robotic Manipulation with Semantic-Spatial Reasoning

Deep Reinforcement Learning-Based Robotic Grasping in Clutter and Occlusion

Learning Tri-mode Grasping for Ambidextrous Robot Picking

Grasping Unstructured Objects with Full Convolutional Network in Clutter

Simultaneous Semantic and Collision Learning for 6-DoF Grasp Pose Estimation