Abstract:Generalizable articulated object manipulation is essential for home-assistant robots. Recent efforts focus on imitation learning from demonstrations or reinforcement learning in simulation, however, due to the prohibitive costs of real-world data collection and precise object simulation, it still remains challenging for these works to achieve broad adaptability across diverse articulated objects. Recently, many works have tried to utilize the strong in-context learning ability of Large Language Models (LLMs) to achieve generalizable robotic manipulation, but most of these researches focus on high-level task planning, sidelining low-level robotic control. In this work, building on the idea that the kinematic structure of the object determines how we can manipulate it, we propose a kinematic-aware prompting framework that prompts LLMs with kinematic knowledge of objects to generate low-level motion trajectory waypoints, supporting various object manipulation. To effectively prompt LLMs with the kinematic structure of different objects, we design a unified kinematic knowledge parser, which represents various articulated objects as a unified textual description containing kinematic joints and contact location. Building upon this unified description, a kinematic-aware planner model is proposed to generate precise 3D manipulation waypoints via a designed kinematic-aware chain-of-thoughts prompting method. Our evaluation spanned 48 instances across 16 distinct categories, revealing that our framework not only outperforms traditional methods on 8 seen categories but also shows a powerful zero-shot capability for 8 unseen articulated object categories. Moreover, the real-world experiments on 7 different object categories prove our framework's adaptability in practical scenarios. Code is released at <a class="link-external link-https" href="https://github.com/GeWu-Lab/LLM_articulated_object_manipulation/tree/main" rel="external noopener nofollow">this https URL</a>.

Keyframer: Empowering Animation Design using Large Language Models

Luminate: Structured Generation and Exploration of Design Space with Large Language Models for Human-AI Co-Creation

Anim-Director: A Large Multimodal Model Powered Agent for Controllable Animation Video Generation

LogoMotion: Visually Grounded Code Generation for Content-Aware Animation

Real-time Animation Generation and Control on Rigged Models via Large Language Models

LLM Blueprint: Enabling Text-to-Image Generation with Complex and Detailed Prompts

Creativity Support in the Age of Large Language Models: An Empirical Study Involving Emerging Writers

Visualizationary: Automating Design Feedback for Visualization Designers using LLMs

Iterative Motion Editing with Natural Language

On the Creativity of Large Language Models

Animate, or Inanimate, That is the Question for Large Language Models

BlenderAlchemy: Editing 3D Graphics with Vision-Language Models

LLM as an Art Director (LaDi): Using LLMs to improve Text-to-Media Generators

Characterising the Creative Process in Humans and Large Language Models

Exploring the Potential of Large Language Models in Artistic Creation: Collaboration and Reflection on Creative Programming

Steering Large Language Models to Evaluate and Amplify Creativity

LAVE: LLM-Powered Agent Assistance and Language Augmentation for Video Editing

A Framework for Collaborating a Large Language Model Tool in Brainstorming for Triggering Creative Thoughts

Kinematic-aware Prompting for Generalizable Articulated Object Manipulation with LLMs

Prompt Sketching for Large Language Models

Large Language Model Agents Enabled Generative Design of Fluidic Computation Interfaces