Diffusion Models Are Real-Time Game Engines

Dani Valevski,Yaniv Leviathan,Moab Arar,Shlomi Fruchter

2024-08-27

Abstract:We present GameNGen, the first game engine powered entirely by a neural model that enables real-time interaction with a complex environment over long trajectories at high quality. GameNGen can interactively simulate the classic game DOOM at over 20 frames per second on a single TPU. Next frame prediction achieves a PSNR of 29.4, comparable to lossy JPEG compression. Human raters are only slightly better than random chance at distinguishing short clips of the game from clips of the simulation. GameNGen is trained in two phases: (1) an RL-agent learns to play the game and the training sessions are recorded, and (2) a diffusion model is trained to produce the next frame, conditioned on the sequence of past frames and actions. Conditioning augmentations enable stable auto-regressive generation over long trajectories.

Machine Learning,Artificial Intelligence,Computer Vision and Pattern Recognition

What problem does this paper attempt to address?

The paper aims to address the following issues:

1. **Real-time Interactive Game Simulation**: Achieving real-time complex interactive game environment simulation through neural models, specifically the simulation of the classic game DOOM. Specifically, the research team developed a game engine called GameNGen, capable of simulating the DOOM game at over 20 frames per second on a single TPU.

2. **High-Quality Image Generation**: Achieving high-resolution image generation comparable to the quality of the original game images. The study mentions that the peak signal-to-noise ratio (PSNR) for next-frame prediction reached 29.4, close to the quality of lossy JPEG compression.

3. **Long Trajectory Stability**: Achieving stable autoregressive generation through conditional enhancement techniques, ensuring effective simulation over long time sequences. This method overcomes some hypothetical issues present in existing diffusion model architectures when generating long sequences and avoids the rapid decline in generation quality over time.

4. **Distinguishability Between Interactive Simulation and Real Game**: Experimental results show that human evaluators can hardly distinguish between simulated segments and real game segments in short clips, indicating that the simulation effect is very realistic.

In summary, the main goal of this paper is to demonstrate whether neural models can achieve real-time complex game simulation while maintaining high image quality, and still maintain high stability and realism over long time sequences. The research results show that GameNGen has achieved significant results in these aspects.