Abstract:Large language models (LLMs) have demonstrated exciting progress in acquiring diverse new capabilities through in-context learning, ranging from logical reasoning to code-writing. Robotics researchers have also explored using LLMs to advance the capabilities of robotic control. However, since low-level robot actions are hardware-dependent and underrepresented in LLM training corpora, existing efforts in applying LLMs to robotics have largely treated LLMs as semantic planners or relied on human-engineered control primitives to interface with the robot. On the other hand, reward functions are shown to be flexible representations that can be optimized for control policies to achieve diverse tasks, while their semantic richness makes them suitable to be specified by LLMs. In this work, we introduce a new paradigm that harnesses this realization by utilizing LLMs to define reward parameters that can be optimized and accomplish variety of robotic tasks. Using reward as the intermediate interface generated by LLMs, we can effectively bridge the gap between high-level language instructions or corrections to low-level robot actions. Meanwhile, combining this with a real-time optimizer, MuJoCo MPC, empowers an interactive behavior creation experience where users can immediately observe the results and provide feedback to the system. To systematically evaluate the performance of our proposed method, we designed a total of 17 tasks for a simulated quadruped robot and a dexterous manipulator robot. We demonstrate that our proposed method reliably tackles 90% of the designed tasks, while a baseline using primitive skills as the interface with Code-as-policies achieves 50% of the tasks. We further validated our method on a real robot arm where complex manipulation skills such as non-prehensile pushing emerge through our interactive system.

GeneWorker: An end-to-end robotic reinforcement learning approach with collaborative generator and worker networks

GSC: A Graph-Based Skill Composition Framework for Robot Learning

Learning Robot Manipulation Skills from Human Demonstration Videos Using Two-Stream 2-D/3-D Residual Networks with Self-Attention

C^2:Co-design of Robots via Concurrent Networks Coupling Online and Offline Reinforcement Learning

RoboGen: Towards Unleashing Infinite Data for Automated Robot Learning via Generative Simulation

SkillMimicGen: Automated Demonstration Generation for Efficient Skill Learning and Deployment

Leveraging the Efficiency of Multi-Task Robot Manipulation Via Task-Evoked Planner and Reinforcement Learning

RoboMatrix: A Skill-centric Hierarchical Framework for Scalable Robot Task Planning and Execution in Open-World

Agentic Skill Discovery

Robot Skill Learning and Generalization Based on Human-robot Collaboration and Computer Vision

Never-Ending Behavior-Cloning Agent for Robotic Manipulation

Unsupervised Skill Discovery for Robotic Manipulation through Automatic Task Generation

C3F: Constant Collaboration and Communication Framework for Graph-Representation Dynamic Multi-Robotic Systems

A Versatile Agent for Fast Learning from Human Instructors

Generative Skill Chaining: Long-Horizon Skill Planning with Diffusion Models

Emergent Real-World Robotic Skills via Unsupervised Off-Policy Reinforcement Learning

Language to Rewards for Robotic Skill Synthesis

Generative Exploration and Exploitation

Adversarial Skill Networks: Unsupervised Robot Skill Learning from Video

Nonverbal social behavior generation for social robots using end-to-end learning

Designs for Enabling Collaboration in Human-Machine Teaming via Interactive and Explainable Systems