Abstract:Large language models (LLMs) have demonstrated exciting progress in acquiring diverse new capabilities through in-context learning, ranging from logical reasoning to code-writing. Robotics researchers have also explored using LLMs to advance the capabilities of robotic control. However, since low-level robot actions are hardware-dependent and underrepresented in LLM training corpora, existing efforts in applying LLMs to robotics have largely treated LLMs as semantic planners or relied on human-engineered control primitives to interface with the robot. On the other hand, reward functions are shown to be flexible representations that can be optimized for control policies to achieve diverse tasks, while their semantic richness makes them suitable to be specified by LLMs. In this work, we introduce a new paradigm that harnesses this realization by utilizing LLMs to define reward parameters that can be optimized and accomplish variety of robotic tasks. Using reward as the intermediate interface generated by LLMs, we can effectively bridge the gap between high-level language instructions or corrections to low-level robot actions. Meanwhile, combining this with a real-time optimizer, MuJoCo MPC, empowers an interactive behavior creation experience where users can immediately observe the results and provide feedback to the system. To systematically evaluate the performance of our proposed method, we designed a total of 17 tasks for a simulated quadruped robot and a dexterous manipulator robot. We demonstrate that our proposed method reliably tackles 90% of the designed tasks, while a baseline using primitive skills as the interface with Code-as-policies achieves 50% of the tasks. We further validated our method on a real robot arm where complex manipulation skills such as non-prehensile pushing emerge through our interactive system.

RoCo: Dialectic Multi-Robot Collaboration with Large Language Models

MHRC: Closed-loop Decentralized Multi-Heterogeneous Robot Collaboration with Large Language Models

Conversational Language Models for Human-in-the-Loop Multi-Robot Coordination

COHERENT: Collaboration of Heterogeneous Multi-Robot System with Large Language Models

RoCo: Robust Cooperative Perception by Iterative Object Matching and Pose Adjustment

RoCo:Robust Collaborative Perception By Iterative Object Matching and Pose Adjustment

Nl2Hltl2Plan: Scaling Up Natural Language Understanding for Multi-Robots Through Hierarchical Temporal Logic Task Representation

Co-NavGPT: Multi-Robot Cooperative Visual Semantic Navigation using Large Language Models

Leveraging Large Language Model for Heterogeneous Ad Hoc Teamwork Collaboration

Language to Rewards for Robotic Skill Synthesis

RoboCoder: Robotic Learning from Basic Skills to General Tasks with Large Language Models

When Robots Get Chatty: Grounding Multimodal Human-Robot Conversation and Collaboration

SiSCo: Signal Synthesis for Effective Human-Robot Communication Via Large Language Models

LaMI: Large Language Models for Multi-Modal Human-Robot Interaction

RoboMP$^2$: A Robotic Multimodal Perception-Planning Framework with Multimodal Large Language Models

C3F: Constant Collaboration and Communication Framework for Graph-Representation Dynamic Multi-Robotic Systems

A neural network-based exploratory learning and motor planning system for co-robots

ROS-LLM: A ROS framework for embodied AI with task feedback and structured reasoning

Large Language Models for Orchestrating Bimanual Robots

SMART-LLM: Smart Multi-Agent Robot Task Planning using Large Language Models

RoboLLM: Robotic Vision Tasks Grounded on Multimodal Large Language Models