Abstract:Collecting large amounts of real-world interaction data to train general robotic policies is often prohibitively expensive, thus motivating the use of simulation data. However, existing methods for data generation have generally focused on scene-level diversity (e.g., object instances and poses) rather than task-level diversity, due to the human effort required to come up with and verify novel tasks. This has made it challenging for policies trained on simulation data to demonstrate significant task-level generalization. In this paper, we propose to automatically generate rich simulation environments and expert demonstrations by exploiting a large language models' (LLM) grounding and coding ability. Our approach, dubbed GenSim, has two modes: goal-directed generation, wherein a target task is given to the LLM and the LLM proposes a task curriculum to solve the target task, and exploratory generation, wherein the LLM bootstraps from previous tasks and iteratively proposes novel tasks that would be helpful in solving more complex tasks. We use GPT4 to expand the existing benchmark by ten times to over 100 tasks, on which we conduct supervised finetuning and evaluate several LLMs including finetuned GPTs and Code Llama on code generation for robotic simulation tasks. Furthermore, we observe that LLMs-generated simulation programs can enhance task-level generalization significantly when used for multitask policy training. We further find that with minimal sim-to-real adaptation, the multitask policies pretrained on GPT4-generated simulation tasks exhibit stronger transfer to unseen long-horizon tasks in the real world and outperform baselines by 25%. See the project website (<a class="link-external link-https" href="https://liruiw.github.io/gensim" rel="external noopener nofollow">this https URL</a>) for code, demos, and videos.

SimBench: A Rule-Based Multi-Turn Interaction Benchmark for Evaluating an LLM's Ability to Generate Digital Twins

SimulBench: Evaluating Language Models with Creative Simulation Tasks

LLM experiments with simulation: Large Language Model Multi-Agent System for Simulation Model Parametrization in Digital Twins

BeSimulator: A Large Language Model Powered Text-based Behavior Simulator

MathBench: Evaluating the Theory and Application Proficiency of LLMs with a Hierarchical Mathematics Benchmark

CS-Bench: A Comprehensive Benchmark for Large Language Models towards Computer Science Mastery

AgentBench: Evaluating LLMs as Agents

BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructions

BabelBench: An Omni Benchmark for Code-Driven Analysis of Multimodal and Multistructured Data

How Far Are LLMs from Believable AI? A Benchmark for Evaluating the Believability of Human Behavior Simulation

SensorBench: Benchmarking LLMs in Coding-Based Sensor Processing

JudgeBench: A Benchmark for Evaluating LLM-based Judges

GenSim: Generating Robotic Simulation Tasks via Large Language Models

A User-Centric Multi-Intent Benchmark for Evaluating Large Language Models

SciBench: Evaluating College-Level Scientific Problem-Solving Abilities of Large Language Models

Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena

CoderUJB: An Executable and Unified Java Benchmark for Practical Programming Scenarios

A User-Centric Benchmark for Evaluating Large Language Models.

CEBench: A Benchmarking Toolkit for the Cost-Effectiveness of LLM Pipelines

AQA-Bench: An Interactive Benchmark for Evaluating LLMs' Sequential Reasoning Ability