Learning Interactive Real-World Simulators

Sherry Yang,Yilun Du,Kamyar Ghasemipour,Jonathan Tompson,Leslie Kaelbling,Dale Schuurmans,Pieter Abbeel

2024-09-27

Abstract:Generative models trained on internet data have revolutionized how text, image, and video content can be created. Perhaps the next milestone for generative models is to simulate realistic experience in response to actions taken by humans, robots, and other interactive agents. Applications of a real-world simulator range from controllable content creation in games and movies, to training embodied agents purely in simulation that can be directly deployed in the real world. We explore the possibility of learning a universal simulator (UniSim) of real-world interaction through generative modeling. We first make the important observation that natural datasets available for learning a real-world simulator are often rich along different dimensions (e.g., abundant objects in image data, densely sampled actions in robotics data, and diverse movements in navigation data). With careful orchestration of diverse datasets, each providing a different aspect of the overall experience, we can simulate the visual outcome of both high-level instructions such as "open the drawer" and low-level controls from otherwise static scenes and objects. We use the simulator to train both high-level vision-language policies and low-level reinforcement learning policies, each of which can be deployed in the real world in zero shot after training purely in simulation. We also show that other types of intelligence such as video captioning models can benefit from training with simulated experience, opening up even wider applications. Video demos can be found at <a class="link-external link-https" href="https://universal-simulator.github.io" rel="external noopener nofollow">this https URL</a>.

Artificial Intelligence

What problem does this paper attempt to address?

The problem that this paper attempts to solve is how to construct a universal real - world interaction simulator (UniSim) through a generative model. Specifically, the authors aim to utilize rich multi - dimensional natural datasets (such as the abundant objects in images, the densely sampled actions in robot data, the diverse movements in navigation data, etc.), and by carefully orchestrating these datasets, to simulate the visual effects after humans, robots and other interacting agents perform actions in the real world. The application range of this simulator is wide, from controllable content creation in games and movies to training physical agents that are trained entirely in a simulated environment so that they can be directly deployed in the real world.

The main contributions of the paper include:
1. For the first time, attempt to construct a universal real - world interaction simulator by combining rich datasets in different dimensions (such as objects, scenes, actions, movements, language and motor control, etc.).
2. Formalize the action - input - video - output framework as an observation prediction model based on finite - history conditions and parameterize it using a video diffusion model. Show how to unfold this observation prediction model in an autoregressive manner to obtain consistent and long - term videos.
3. Demonstrate how the simulator enables high - level language policies, low - level control policies and video caption models to be generalized to the real world when trained only through simulation, thus bridging the gap between simulation and reality.

The paper describes in detail how to handle different types of data, unify the action space, and simulate long - term sequential interactions through the video diffusion model. In addition, the paper also shows the application of the simulator in training long - term visual - language policies, reinforcement learning agents and broader visual - language tasks, proving its potential value in multiple fields.