WorldSimBench: Towards Video Generation Models as World Simulators

Yiran Qin,Zhelun Shi,Jiwen Yu,Xijun Wang,Enshen Zhou,Lijun Li,Zhenfei Yin,Xihui Liu,Lu Sheng,Jing Shao,Lei Bai,Wanli Ouyang,Ruimao Zhang

2024-10-24

Abstract:Recent advancements in predictive models have demonstrated exceptional capabilities in predicting the future state of objects and scenes. However, the lack of categorization based on inherent characteristics continues to hinder the progress of predictive model development. Additionally, existing benchmarks are unable to effectively evaluate higher-capability, highly embodied predictive models from an embodied perspective. In this work, we classify the functionalities of predictive models into a hierarchy and take the first step in evaluating World Simulators by proposing a dual evaluation framework called WorldSimBench. WorldSimBench includes Explicit Perceptual Evaluation and Implicit Manipulative Evaluation, encompassing human preference assessments from the visual perspective and action-level evaluations in embodied tasks, covering three representative embodied scenarios: Open-Ended Embodied Environment, Autonomous, Driving, and Robot Manipulation. In the Explicit Perceptual Evaluation, we introduce the HF-Embodied Dataset, a video assessment dataset based on fine-grained human feedback, which we use to train a Human Preference Evaluator that aligns with human perception and explicitly assesses the visual fidelity of World Simulators. In the Implicit Manipulative Evaluation, we assess the video-action consistency of World Simulators by evaluating whether the generated situation-aware video can be accurately translated into the correct control signals in dynamic environments. Our comprehensive evaluation offers key insights that can drive further innovation in video generation models, positioning World Simulators as a pivotal advancement toward embodied artificial intelligence.

Computer Vision and Pattern Recognition

What problem does this paper attempt to address?

The problems that this paper attempts to solve mainly focus on two aspects:

1. **Functional Classification of Prediction Models**: The current prediction models have rich application scenarios and diverse designs, but lack classification based on their intrinsic characteristics, which restricts the development of prediction models. Therefore, the first question is whether a reasonable hierarchical system of prediction models can be established to classify them according to the degree of embodiment of the models. Through clear classification, prediction models can be better evaluated from different embodiment perspectives, ensuring that their strengths and weaknesses are fully evaluated.

2. **Detailed Evaluation of Highly Embodied Prediction Models**: Existing evaluation methods usually focus on task - planning ability, which is achieved by evaluating text output or evaluating visual output from an aesthetic perspective. However, these methods are insufficient when evaluating highly embodied prediction models, because the embodied scenarios are more concerned with physical properties (such as view - point consistency, object destructiveness), and existing methods cannot effectively evaluate these characteristics. Therefore, the second question is whether a more detailed evaluation of highly embodied prediction models can be carried out from the embodiment perspective.

To answer these questions, the author proposes the following solutions:

- **Hierarchical Classification of Prediction Models**: The functions of prediction models are divided into four stages from S0 to S3, and each stage defines the capabilities and degrees of embodiment of the models. Models in the S0 stage can only generate text, models in the S1 stage can generate images, models in the S2 stage can generate videos, and models in the S3 stage can generate operable videos, that is, world simulators. These models can generate videos that conform to physical rules and perform actions.

- **Dual - evaluation Framework WorldSimBench**: In order to evaluate world simulators, the author proposes a dual - evaluation framework - WorldSimBench, including explicit perceptual evaluation and implicit manipulative evaluation. Explicit perceptual evaluation focuses on evaluating the visual quality, conditional consistency and degree of embodiment of the generated content from the perspective of human perception; implicit manipulative evaluation evaluates the performance of world simulators in dynamic environments by converting the generated videos into control signals.

- **Specific Application Scenarios**: The paper selects three representative embodied scenarios for evaluation: open - ended embodied environment (OE), autonomous driving (AD) and robot manipulation (RM). Specific evaluation indicators and methods are designed for each scenario to comprehensively evaluate the capabilities of world simulators.

Through the above methods, the paper aims to promote the research and development of prediction models, especially the evaluation of world simulators, and provide valuable insights for future video - generation models.