Abstract:We propose a novel framework for learning high-level cognitive capabilities in robot manipulation tasks, such as making a smiley face using building blocks. These tasks often involve complex multi-step reasoning, presenting significant challenges due to the limited paired data connecting human instructions (e.g., making a smiley face) and robot actions (e.g., end-effector movement). Existing approaches relieve this challenge by adopting an open-loop paradigm decomposing high-level instructions into simple sub-task plans, and executing them step-by-step using low-level control models. However, these approaches are short of instant observations in multi-step reasoning, leading to sub-optimal results. To address this issue, we propose to automatically collect a cognitive robot dataset by Large Language Models (LLMs). The resulting dataset AlphaBlock consists of 35 comprehensive high-level tasks of multi-step text plans and paired observation sequences. To enable efficient data acquisition, we employ elaborated multi-round prompt designs that effectively reduce the burden of extensive human involvement. We further propose a closed-loop multi-modal embodied planning model that autoregressively generates plans by taking image observations as input. To facilitate effective learning, we leverage MiniGPT-4 with a frozen visual encoder and LLM, and finetune additional vision adapter and Q-former to enable fine-grained spatial perception for manipulation tasks. We conduct experiments to verify the superiority over existing open and closed-loop methods, and achieve a significant increase in success rate by 21.4% and 14.5% over ChatGPT and GPT-4 based robot tasks. Real-world demos are shown in <a class="link-external link-https" href="https://www.youtube.com/watch?v=ayAzID1_qQk" rel="external noopener nofollow">this https URL</a> .

CLIMB: Language-Guided Continual Learning for Task Planning with Iterative Model Building

A Cascaded Deep Learning Framework for Real-time and Robust Grasp Planning

CLFR-M: Continual Learning Framework for Robots Via Human Feedback and Dynamic Memory

Interactive Task Planning with Language Models

SayPlan: Grounding Large Language Models using 3D Scene Graphs for Scalable Robot Task Planning

CLiMB: A Continual Learning Benchmark for Vision-and-Language Tasks

Lifelong Robot Learning with Human Assisted Language Planners

AlphaBlock: Embodied Finetuning for Vision-Language Reasoning in Robot Manipulation

SayPlan: Grounding Large Language Models using 3D Scene Graphs for Scalable Task Planning

PLATO: Planning with LLMs and Affordances for Tool Manipulation

Verifiably Following Complex Robot Instructions with Foundation Models

Instruct2Act: Mapping Multi-modality Instructions to Robotic Actions with Large Language Model

Grounding LLMs For Robot Task Planning Using Closed-loop State Feedback

SPIRE: Synergistic Planning, Imitation, and Reinforcement Learning for Long-Horizon Manipulation

ARNOLD: A Benchmark for Language-Grounded Task Learning With Continuous States in Realistic 3D Scenes

Closed-Loop Long-Horizon Robotic Planning via Equilibrium Sequence Modeling

SAGCI-System: Towards Sample-Efficient, Generalizable, Compositional, and Incremental Robot Learning

Robot Task Planning Based on Large Language Model Representing Knowledge with Directed Graph Structures

Nl2Hltl2Plan: Scaling Up Natural Language Understanding for Multi-Robots Through Hierarchical Temporal Logic Task Representation

Learning adaptive planning representations with natural language guidance

Spatial-Language Attention Policies for Efficient Robot Learning