Abstract:Language guided grasping in cultter environment has become a hot topic for robotic arms, where the robot is supposed to grasp a specific object according to the language instruction. However, existing studies typically fail on grasping unknown objects that are not recognized by the detection model, which post higher requirements of the learning unknown objects from external knowledge. In this paper, unlike traditional target detection algorithms, our system identifies unknown objects through a knowledge-driven approach. We present a knowledge-learning driven system that guides the robot in learning new concepts to grasp unknown objects from pre-built multimodal knowledge base. Specifically, we first retrieve the image knowledge from Internet and text description knowledge from human experts to construct the multimodal knowledge base. Based on the question context, the model will query the object concept knowledge from the knowledge base. Once the concept is known, the model will ground the object by a multimodal fusion module. In contrast, to address the issue of novel concept, we propose a knowledge fusion module that combines BERT, GCN and ResNet to acquire semantic text knowledge, structured text knowledge, and image knowledge derived from the knowledge base. Subsequently, we utilized the re-calibration multimodal fusion module to ground unknown objects. Finally, conditioned on the grounding context, we employ a well-trained grasp network to derive the accurate grasping pose to perform the grasping task. Experiments conducted on the simulation and real-world scenarios demonstrate that our knowledge-learning driven system is effective in assisting robots to learn unknown concepts and achieve successful grasping tasks. A demo video is available online ( https://youtu.be/Fsc4nZyY9LI ).

Learning to Grasp Without Seeing

More Than a Feeling: Learning to Grasp and Regrasp using Vision and Touch

Vision-Based Robotic Object Grasping—A Deep Reinforcement Learning Approach

Learn to grasp unknown objects in robotic manipulation

Center-of-Mass-Based Regrasping of Unknown Objects Using Reinforcement Learning and Tactile Sensing.

Learning a visuomotor controller for real world robotic grasping using simulated depth images

Learning Fine Pinch-Grasp Skills using Tactile Sensing from A Few Real-world Demonstrations

Learning to Regrasp by Learning to Place

Learning Gentle Grasping from Human-Free Force Control Demonstration

Language Guided Grasping of Unknown Concepts Based on Knowledge System.

Bayesian Grasp: Robotic visual stable grasp based on prior tactile knowledge

Learning Task-Based Robotic Grasping with Vision, Haptics and Proprioception

Learning to Regrasp Using Visual–Tactile Representation-Based Reinforcement Learning

Autonomous Robotic Manipulation: Real-Time, Deep-Learning Approach for Grasping of Unknown Objects

Integrating High-Resolution Tactile Sensing into Grasp Stability Prediction

Tactile Regrasp: Grasp Adjustments via Simulated Tactile Transformations

Grasp Anything: Combining Teacher-Augmented Policy Gradient Learning with Instance Segmentation to Grasp Arbitrary Objects

Learning-based robotic grasping: A review

Robotic grasping using visual and tactile sensing

Learning Hand-Eye Coordination for Robotic Grasping with Large-Scale Data Collection

Learning Robust Grasping Strategy Through Tactile Sensing and Adaption Skill