Abstract:Robotic grasping is one of the most fundamental robotic manipulation tasks and has been the subject of extensive research. However, swiftly teaching a robot to grasp a novel target object in clutter remains challenging. This article attempts to address the challenge by leveraging object attributes that facilitate recognition, grasping, and rapid adaptation to new domains. In this work, we present an end-to-end encoder–decoder network to learn attribute-based robotic grasping with data-efficient adaptation capability. We first pretrain the end-to-end model with a variety of basic objects to learn generic attribute representation for recognition and grasping. Our approach fuses the embeddings of a workspace image and a query text using a gated-attention mechanism and learns to predict instance grasping affordances. To train the joint embedding space of visual and textual attributes, the robot utilizes object persistence before and after grasping. Our model is self-supervised in a simulation that only uses basic objects of various colors and shapes but generalizes to novel objects in new environments. To further facilitate generalization, we propose two adaptation methods, adversarial adaption and one-grasp adaptation. Adversarial adaptation regulates the image encoder using augmented data of unlabeled images, whereas one-grasp adaptation updates the overall end-to-end model using augmented data from one grasp trial. Both adaptation methods are data-efficient and considerably improve instance grasping performance. Experimental results in both simulation and the real world demonstrate that our approach achieves over 81% instance grasping success rate on unknown objects, which outperforms several baselines by large margins.

Generating Future Observations to Estimate Grasp Success in Cluttered Environments

S4G: Amodal Single-view Single-Shot SE(3) Grasp Detection in Cluttered Scenes

Instance-Level Coarse-to-Fine High-Precision Grasping in Cluttered Environments

Adapting Skills to Novel Grasps: A Self-Supervised Approach

Self-Supervised Learning for Joint Pushing and Grasping Policies in Highly Cluttered Environments

A Self-supervised Contrastive Learning Method for Grasp Outcomes Prediction

Model-free Grasping with Multi-Suction Cup Grippers for Robotic Bin Picking

Self-Supervised Learning of Grasping Arbitrary Objects On-the-Move

Self-Supervised Instance Segmentation by Grasping

Attribute-Based Robotic Grasping With Data-Efficient Adaptation

Learning an End-to-end Spatial Grasp Generation and Refinement Algorithm from Simulation

Uncertainty-driven Exploration Strategies for Online Grasp Learning

Hierarchical Policies for Cluttered-Scene Grasping with Latent Plans

Efficient and Robust Robotic Grasping in Cluttered Scenes: A Point Cloud-Based Approach with Heuristic Evaluation.

Suction Grasp Region Prediction Using Self-supervised Learning for Object Picking in Dense Clutter

More Than a Feeling: Learning to Grasp and Regrasp using Vision and Touch

Efficient End-to-End Detection of 6-DoF Grasps for Robotic Bin Picking

Probabilistic Closed-Loop Active Grasping

Synergistic Pushing and Grasping for Enhanced Robotic Manipulation Using Deep Reinforcement Learning

A Self-Supervised Learning Manipulator Grasping Approach Based on Instance Segmentation.

Self-Supervised Antipodal Grasp Learning with Fine-Grained Grasp Quality Feedback in Clutter