Abstract:Vision-Language Models (VLM) can generate plausible high-level plans when prompted with a goal, the context, an image of the scene, and any planning constraints. However, there is no guarantee that the predicted actions are geometrically and kinematically feasible for a particular robot embodiment. As a result, many prerequisite steps such as opening drawers to access objects are often omitted in their plans. Robot task and motion planners can generate motion trajectories that respect the geometric feasibility of actions and insert physically necessary actions, but do not scale to everyday problems that require common-sense knowledge and involve large state spaces comprised of many variables. We propose VLM-TAMP, a hierarchical planning algorithm that leverages a VLM to generate goth semantically-meaningful and horizon-reducing intermediate subgoals that guide a task and motion planner. When a subgoal or action cannot be refined, the VLM is queried again for replanning. We evaluate VLM- TAMP on kitchen tasks where a robot must accomplish cooking goals that require performing 30-50 actions in sequence and interacting with up to 21 objects. VLM-TAMP substantially outperforms baselines that rigidly and independently execute VLM-generated action sequences, both in terms of success rates (50 to 100% versus 0%) and average task completion percentage (72 to 100% versus 15 to 45%). See project site <a class="link-external link-https" href="https://zt-yang.github.io/vlm-tamp-robot/" rel="external noopener nofollow">this https URL</a> for more information.

Using AI Planning Techniques to Automatically Generate Image Processing Procedures

Autonomously Generating Operations Sequences for a Mars Rover Using Artificial Intelligence-Based Planning

Multimodal Procedural Planning via Dual Text-Image Prompting

Autonomously generating operations sequences for a Mars rover using AI-based planning

Learning to Imagine Manipulation Goals for Robot Task Planning

Work in Progress -- Automated Generation of Robotic Planning Domains from Observations

Visual Robot Task Planning

Intelligent Execution through Plan Analysis

VisionCoder: Empowering Multi-Agent Auto-Programming for Image Processing with Hybrid LLMs

Generating machine-executable plans from end-user's natural-language instructions

Combined Task and Motion Planning as Classical AI Planning

Simulated Mental Imagery for Robotic Task Planning

Path Planning Based on an Artificial Vision System and Optical Character Recognition (OCR)

Guiding Long-Horizon Task and Motion Planning with Vision Language Models

Select2Plan: Training-Free ICL-Based Planning through VQA and Memory Retrieval

Multi-agent Planning using Visual Language Models

SMUG Planner: A Safe Multi-Goal Planner for Mobile Robots in Challenging Environments

An AI Planning-Based Approach to the Multi-Agent Plan Recognition Problem

Primitive Action Based Combined Task and Motion Planning for the Service Robot

Automated Planning Domain Inference for Task and Motion Planning

Integrating Robot Task Planner with Common-sense Knowledge Base to Improve the Efficiency of Planning