Abstract:This paper studies the task of any objects grasping from the known categories by free-form language instructions. This task demands the technique in computer vision, natural language processing, and robotics. We bring these disciplines together on this open challenge, which is essential to human-robot interaction. Critically, the key challenge lies in inferring the category of objects from linguistic instructions and accurately estimating the 6-DoF information of unseen objects from the known classes. In contrast, previous works focus on inferring the pose of object candidates at the instance level. This significantly limits its applications in real-world <a class="link-external link-http" href="http://scenarios.In" rel="external noopener nofollow">this http URL</a> this paper, we propose a language-guided 6-DoF category-level object localization model to achieve robotic grasping by comprehending human intention. To this end, we propose a novel two-stage method. Particularly, the first stage grounds the target in the RGB image through language description of names, attributes, and spatial relations of objects. The second stage extracts and segments point clouds from the cropped depth image and estimates the full 6-DoF object pose at category-level. Under such a manner, our approach can locate the specific object by following human instructions, and estimate the full 6-DoF pose of a category-known but unseen instance which is not utilized for training the model. Extensive experimental results show that our method is competitive with the state-of-the-art language-conditioned grasp method. Importantly, we deploy our approach on a physical robot to validate the usability of our framework in real-world applications. Please refer to the supplementary for the demo videos of our robot experiments.

Egocentric Object Recognition Leveraging The 3d Shape Of The Grasping Hand

In-Hand 3D Object Reconstruction from a Monocular RGB Video

CenterGrasp: Object-Aware Implicit Representation Learning for Simultaneous Shape Reconstruction and 6-DoF Grasp Estimation

3D Eye-to-Hand Coordination for Uninstructed Robot Grasp Planning

3D Hand Pose Estimation in Everyday Egocentric Images

Simple Kinesthetic Haptics for Object Recognition

Egocentric Hand-object Interaction Detection and Application

In My Perspective, In My Hands: Accurate Egocentric 2D Hand Pose and Action Recognition

6D Pose Estimation with Combined Deep Learning and 3D Vision Techniques for a Fast and Accurate Object Grasping

Using Geometry to Detect Grasps in 3D Point Clouds

Seeing by haptic glance: reinforcement learning-based 3D object Recognition

HandNeRF: Learning to Reconstruct Hand-Object Interaction Scene from a Single RGB Image

Recognition of the Three-Dimensional Shape of Objects Grasped for Pesa Multi-Fingered Robot Hand

Learning 6-DOF Grasping Interaction via Deep Geometry-aware 3D Representations

UPG: 3D Vision-Based Prediction Framework for Robotic Grasping in Multi-Object Scenes.

3D Hand Tracking With Head Mounted Gaze-Directed Camera

MVGrasp: Real-time multi-view 3D object grasping in highly cluttered environments

HandGCAT: Occlusion-Robust 3D Hand Mesh Reconstruction from Monocular Images.

3D Hand and Object Tracking for Inside out Activity Analysis

Visual Robotic Object Grasping Through Combining RGB-D Data and 3D Meshes.

Learning 6-DoF Object Poses to Grasp Category-level Objects by Language Instructions