Abstract:We propose a novel framework for learning high-level cognitive capabilities in robot manipulation tasks, such as making a smiley face using building blocks. These tasks often involve complex multi-step reasoning, presenting significant challenges due to the limited paired data connecting human instructions (e.g., making a smiley face) and robot actions (e.g., end-effector movement). Existing approaches relieve this challenge by adopting an open-loop paradigm decomposing high-level instructions into simple sub-task plans, and executing them step-by-step using low-level control models. However, these approaches are short of instant observations in multi-step reasoning, leading to sub-optimal results. To address this issue, we propose to automatically collect a cognitive robot dataset by Large Language Models (LLMs). The resulting dataset AlphaBlock consists of 35 comprehensive high-level tasks of multi-step text plans and paired observation sequences. To enable efficient data acquisition, we employ elaborated multi-round prompt designs that effectively reduce the burden of extensive human involvement. We further propose a closed-loop multi-modal embodied planning model that autoregressively generates plans by taking image observations as input. To facilitate effective learning, we leverage MiniGPT-4 with a frozen visual encoder and LLM, and finetune additional vision adapter and Q-former to enable fine-grained spatial perception for manipulation tasks. We conduct experiments to verify the superiority over existing open and closed-loop methods, and achieve a significant increase in success rate by 21.4% and 14.5% over ChatGPT and GPT-4 based robot tasks. Real-world demos are shown in <a class="link-external link-https" href="https://www.youtube.com/watch?v=ayAzID1_qQk" rel="external noopener nofollow">this https URL</a> .

FireRobBrain: Planning for a Firefighting Robot using Knowledge Graph and Large Language Model

KGGPT: Empowering Robots with OpenAI's ChatGPT and Knowledge Graph.

RoboBrain: Large-Scale Knowledge Engine for Robots

Robot Planning Based on Behavior Tree and Knowledge Graph

Decision-Making in Robotic Grasping with Large Language Models.

CLFR-M: Continual Learning Framework for Robots Via Human Feedback and Dynamic Memory

Robot Task Planning Based on Large Language Model Representing Knowledge with Directed Graph Structures

Developing high-level cognitive functions for service robots

Large Language Models in Fire Engineering: An Examination of Technical Questions Against Domain Knowledge

Safety Control of Service Robots with LLMs and Embodied Knowledge Graphs

AlphaBlock: Embodied Finetuning for Vision-Language Reasoning in Robot Manipulation

Enhancing intention prediction and interpretability in service robots with LLM and KG

Towards Robust Reasoning over Knowledge Graphs

Hierarchical Understanding in Robotic Manipulation: A Knowledge-Based Framework

How Far Can I Reach? A Case Study in Robot Knowledge Acquisition

RoboMP$^2$: A Robotic Multimodal Perception-Planning Framework with Multimodal Large Language Models

Knowledge-Enabled Decision Making for Robotic System Utilizing Ambient Service Components

An Efficient Firefighting Method for Robotics: A Novel Convolution-Based Lightweight Network Model Guided by Contextual Features with Dual Attention

Leveraging Commonsense Knowledge from Large Language Models for Task and Motion Planning

Robo-CSK-Organizer: Commonsense Knowledge to Organize Detected Objects for Multipurpose Robots

KG-Agent: An Efficient Autonomous Agent Framework for Complex Reasoning over Knowledge Graph