Abstract:Text-to-video (T2V) models like Sora have made significant strides in visualizing complex prompts, which is increasingly viewed as a promising path towards constructing the universal world simulator. Cognitive psychologists believe that the foundation for achieving this goal is the ability to understand intuitive physics. However, the capacity of these models to accurately represent intuitive physics remains largely unexplored. To bridge this gap, we introduce PhyGenBench, a comprehensive \textbf{Phy}sics \textbf{Gen}eration \textbf{Ben}chmark designed to evaluate physical commonsense correctness in T2V generation. PhyGenBench comprises 160 carefully crafted prompts across 27 distinct physical laws, spanning four fundamental domains, which could comprehensively assesses models' understanding of physical commonsense. Alongside PhyGenBench, we propose a novel evaluation framework called PhyGenEval. This framework employs a hierarchical evaluation structure utilizing appropriate advanced vision-language models and large language models to assess physical commonsense. Through PhyGenBench and PhyGenEval, we can conduct large-scale automated assessments of T2V models' understanding of physical commonsense, which align closely with human feedback. Our evaluation results and in-depth analysis demonstrate that current models struggle to generate videos that comply with physical commonsense. Moreover, simply scaling up models or employing prompt engineering techniques is insufficient to fully address the challenges presented by PhyGenBench (e.g., dynamic scenarios). We hope this study will inspire the community to prioritize the learning of physical commonsense in these models beyond entertainment applications. We will release the data and codes at <a class="link-external link-https" href="https://github.com/OpenGVLab/PhyGenBench" rel="external noopener nofollow">this https URL</a>

An Overview of OpenAI's Sora and Its Potential for Physics Engine Free Games and Virtual Reality

Sora: A Review on Background, Technology, Limitations, and Opportunities of Large Vision Models

Sora Generates Videos with Stunning Geometrical Consistency

How OpenAI's text-to-video tool Sora could change science – and society

Analysing the Public Discourse around OpenAI's Text-To-Video Model 'Sora' using Topic Modeling

Concerns with OpenAI's Sora in Medicine

From Sora What We Can See: A Survey of Text-to-Video Generation

How Far is Video Generation from World Model: A Physical Law Perspective

From text to video with AI: the rise and potential of Sora in education and libraries

"Sora is Incredible and Scary": Emerging Governance Challenges of Text-to-Video Generative AI Models

Sora as an AGI World Model? A Complete Survey on Text-to-Video Generation

Cutting-edge research in physics engines: Exploring parallel technologies, algorithm insights, and upcoming trends

The Dawn of Video Generation: Preliminary Explorations with SORA-like Models

Sora for Foundation Robots with Parallel Intelligence: Three World Models, Three Robotic Systems

Sora OpenAI's Prelude: Social Media Perspectives on Sora OpenAI and the Future of AI Video Generation

AI meets physics: a comprehensive survey

Is Sora a World Simulator? A Comprehensive Survey on General World Models and Beyond

AnySkill: Learning Open-Vocabulary Physical Skill for Interactive Agents

Towards World Simulator: Crafting Physical Commonsense-Based Benchmark for Video Generation

When Does Sora Show: The Beginning of TAO to Imaginative Intelligence and Scenarios Engineering

AI & robotics briefing: How AI images and videos could change science