Abstract:With the rapid advancement of robotics technology, an increasing number of researchers are exploring the use of natural language as a communication channel between humans and robots. In scenarios where language conditioned manipulation grounding, prevailing methods rely heavily on supervised multimodal deep learning. In this paradigm, robots assimilate knowledge from both language instructions and visual input. However, these approaches lack external knowledge for comprehending natural language instructions and are hindered by the substantial demand for a large amount of paired data, where vision and language are usually linked through manual annotation for the creation of realistic datasets. To address the above problems, we propose the knowledge enhanced bottom-up affordance grounding network (KBAG-Net), which enhances natural language understanding through external knowledge, improving accuracy in object grasping affordance segmentation. In addition, we introduce a semi-automatic data generation method aimed at facilitating the quick establishment of the language following manipulation grounding dataset. The experimental results on two standard dataset demonstrate that our method outperforms existing methods with the external knowledge. Specifically, our method outperforms the two-stage method by 12.98% and 1.22% of mIoU on the two dataset, respectively. For broader community engagement, we will make the semi-automatic data construction method publicly available at https://github.com/wmqu/Automated-Dataset-Construction4LGM .

Multi-mode Natural Language Processing for Extracting Open Knowledge

Multi-mode Natural Language Processing for Human-Robot Interaction.

Toward Open Knowledge Enabling for Human-Robot Interaction

Handling Open Knowledge for Service Robots

KGGPT: Empowering Robots with OpenAI's ChatGPT and Knowledge Graph.

Developing high-level cognitive functions for service robots

Knowledge-Aware Procedural Text Understanding with Multi-Stage Training

Scene-Driven Multimodal Knowledge Graph Construction for Embodied AI

KeJia Project : Towards Integrated Intelligence for Service Robots

Multi-modal Interaction with Transformers: Bridging Robots and Human with Natural Language

OK-Robot: What Really Matters in Integrating Open-Knowledge Models for Robotics

Network for Knowledge Organization (NEKO): an AI knowledge mining workflow for synthetic biology research

OpenAgents: An Open Platform for Language Agents in the Wild

OPERA: Harmonizing Task-Oriented Dialogs and Information Seeking Experience

Exploring Large Language Models as a Source of Common-Sense Knowledge for Robots

Robo-CSK-Organizer: Commonsense Knowledge to Organize Detected Objects for Multipurpose Robots

Integrating Action Knowledge and LLMs for Task Planning and Situation Handling in Open Worlds

Robot Task Planning and Situation Handling in Open Worlds

New advances on multi-lingual multi-function multi-media intelligent system

DeepKE: A Deep Learning Based Knowledge Extraction Toolkit for Knowledge Base Population

Knowledge enhanced bottom-up affordance grounding for robotic interaction