Abstract:Language guided grasping in cultter environment has become a hot topic for robotic arms, where the robot is supposed to grasp a specific object according to the language instruction. However, existing studies typically fail on grasping unknown objects that are not recognized by the detection model, which post higher requirements of the learning unknown objects from external knowledge. In this paper, unlike traditional target detection algorithms, our system identifies unknown objects through a knowledge-driven approach. We present a knowledge-learning driven system that guides the robot in learning new concepts to grasp unknown objects from pre-built multimodal knowledge base. Specifically, we first retrieve the image knowledge from Internet and text description knowledge from human experts to construct the multimodal knowledge base. Based on the question context, the model will query the object concept knowledge from the knowledge base. Once the concept is known, the model will ground the object by a multimodal fusion module. In contrast, to address the issue of novel concept, we propose a knowledge fusion module that combines BERT, GCN and ResNet to acquire semantic text knowledge, structured text knowledge, and image knowledge derived from the knowledge base. Subsequently, we utilized the re-calibration multimodal fusion module to ground unknown objects. Finally, conditioned on the grounding context, we employ a well-trained grasp network to derive the accurate grasping pose to perform the grasping task. Experiments conducted on the simulation and real-world scenarios demonstrate that our knowledge-learning driven system is effective in assisting robots to learn unknown concepts and achieve successful grasping tasks. A demo video is available online ( https://youtu.be/Fsc4nZyY9LI ).

Continual Learning for Robotic Grasping Detection with Knowledge Transferring

LiteGrasp: A Light Robotic Grasp Detection Via Semi-Supervised Knowledge Distillation

Vision-Based Robotic Object Grasping—A Deep Reinforcement Learning Approach

Efficient Grasp Detection Network with Gaussian-Based Grasp Representation for Robotic Manipulation

Residual Squeeze-and-Excitation Network with Multi-scale Spatial Pyramid Module for Fast Robotic Grasping Detection

Digital Twin Robotic System with Continuous Learning for Grasp Detection in Variable Scenes

Language Guided Grasping of Unknown Concepts Based on Knowledge System.

Efficient Fully Convolutional Network and Optimization Approach for Robotic Grasping Detection Based on RGB-D Images

Two-stage Grasp Detection Method for Robotics Using Point Clouds and Deep Hierarchical Feature Learning Network

Robotic Objects Detection and Grasping in Clutter Based on Cascaded Deep Convolutional Neural Network

A Novel Large-Kernel Residual Grasp Network for Robot Grasp Detection

A Real-Time Grasping Detection Network Architecture for Various Grasping Scenarios

Detecting Graspable Rectangles of Objects in Robotic Grasping

Deep learning for detecting robotic grasps

Robotic Grasp Detection Network Based on Improved Deformable Convolution and Spatial Feature Center Mechanism

Robotic Grasping Point Recognition Model Based on Reinforcement Learning

Enhancement ofreal-timegrasp detection by cascaded deep convolutional neural networks

Deep Reinforcement Learning Enhanced Convolutional Neural Networks for Robotic Grasping

Robotic Grasping Technology Integrating Large Kernel Convolution and Residual Connections

Object Detection Recognition And Robot Grasping Based On Machine Learning: A Survey

Deep Vision Networks for Real-Time Robotic Grasp Detection