Abstract:Collecting large amounts of real-world interaction data to train general robotic policies is often prohibitively expensive, thus motivating the use of simulation data. However, existing methods for data generation have generally focused on scene-level diversity (e.g., object instances and poses) rather than task-level diversity, due to the human effort required to come up with and verify novel tasks. This has made it challenging for policies trained on simulation data to demonstrate significant task-level generalization. In this paper, we propose to automatically generate rich simulation environments and expert demonstrations by exploiting a large language models' (LLM) grounding and coding ability. Our approach, dubbed GenSim, has two modes: goal-directed generation, wherein a target task is given to the LLM and the LLM proposes a task curriculum to solve the target task, and exploratory generation, wherein the LLM bootstraps from previous tasks and iteratively proposes novel tasks that would be helpful in solving more complex tasks. We use GPT4 to expand the existing benchmark by ten times to over 100 tasks, on which we conduct supervised finetuning and evaluate several LLMs including finetuned GPTs and Code Llama on code generation for robotic simulation tasks. Furthermore, we observe that LLMs-generated simulation programs can enhance task-level generalization significantly when used for multitask policy training. We further find that with minimal sim-to-real adaptation, the multitask policies pretrained on GPT4-generated simulation tasks exhibit stronger transfer to unseen long-horizon tasks in the real world and outperform baselines by 25%. See the project website (<a class="link-external link-https" href="https://liruiw.github.io/gensim" rel="external noopener nofollow">this https URL</a>) for code, demos, and videos.

GenG: An LLM-Based Generic Time Series Data Generation Approach for Edge Intelligence via Cross-Domain Collaboration

An Overview on Generative AI at Scale with Edge-Cloud Computing

Generative AI as a Service in 6G Edge-Cloud: Generation Task Offloading by In-context Learning

Beyond the Cloud: Edge Inference for Generative Large Language Models in Wireless Networks

An Edge-Cloud Collaboration Framework for Generative AI Service Provision with Synergetic Big Cloud Model and Small Edge Models

Towards Integrated Fine-tuning and Inference when Generative AI meets Edge Intelligence

Multi-Agent RL-Based Industrial AIGC Service Offloading over Wireless Edge Networks

Generative AI on the Edge: Architecture and Performance Evaluation

Toward Democratized Generative AI in Next-Generation Mobile Edge Networks

AIGC for Industrial Time Series: From Deep Generative Models to Large Generative Models

NetGPT:An AI-Native Network Architecture for Provisioning Beyond Personalized Generative Services

NetGPT: An AI-Native Network Architecture for Provisioning Beyond Personalized Generative Services

Toward Scalable Generative AI via Mixture of Experts in Mobile Edge Networks

Edge Intelligence Optimization for Large Language Model Inference with Batching and Quantization

Mobile Edge Generation: A New Era to 6G

Large Language Models Empowered Autonomous Edge AI for Connected Intelligence

EDGE: Enhanced Grounded GUI Understanding with Enriched Multi-Granularity Synthetic Data

Towards Self-learning Edge Intelligence in 6G

GenSim: Generating Robotic Simulation Tasks via Large Language Models

Enhancing Vehicular Edge Intelligence Through Distributed Collaborative Generative AI Inference