The Dawn of Video Generation: Preliminary Explorations with SORA-like Models

Ailing Zeng,Yuhang Yang,Weidong Chen,Wei Liu
2024-10-10
Abstract:High-quality video generation, encompassing text-to-video (T2V), image-to-video (I2V), and video-to-video (V2V) generation, holds considerable significance in content creation to benefit anyone express their inherent creativity in new ways and world simulation to modeling and understanding the world. Models like SORA have advanced generating videos with higher resolution, more natural motion, better vision-language alignment, and increased controllability, particularly for long video sequences. These improvements have been driven by the evolution of model architectures, shifting from UNet to more scalable and parameter-rich DiT models, along with large-scale data expansion and refined training strategies. However, despite the emergence of DiT-based closed-source and open-source models, a comprehensive investigation into their capabilities and limitations remains lacking. Furthermore, the rapid development has made it challenging for recent benchmarks to fully cover SORA-like models and recognize their significant advancements. Additionally, evaluation metrics often fail to align with human preferences.
Computer Vision and Pattern Recognition
What problem does this paper attempt to address?
### Problems the Paper Aims to Solve This paper aims to address several key issues in current video generation technology: 1. **Comprehensive Evaluation and Comparison**: Although some benchmarks attempt to fairly compare video generation models, these benchmarks are often outdated and cannot fully cover the latest model performances. Additionally, existing evaluation metrics often fail to fully reflect human preferences. 2. **Gap Between Closed-Source and Open-Source Models**: Closed-source models are typically trained on large-scale computational resources and datasets, outperforming open-source models. This gap leads to a disconnect between academic research and industrial practice. 3. **In-Depth Analysis of Model Capabilities and Limitations**: Existing research lacks a comprehensive investigation into the capabilities and limitations of video generation models. Particularly, there are many unresolved issues regarding the generation of long video sequences, complex physical interactions, and the integration of multimodal information. 4. **Exploration of Practical Applications**: The potential applications of video generation technology in various vertical fields (such as portrait animation, robotics, cartoon animation, etc.) have not been fully explored. A systematic evaluation of these models' performance in practical applications is needed. 5. **Future Research Directions**: Exploring the future development directions of video generation technology, including multimodal generation, real-time interactive video generation, and adaptation to rare scenes. ### Specific Goals - **Design a Comprehensive Evaluation Framework**: By designing over 700 systematic prompts, conduct a comprehensive evaluation of existing text-to-video (T2V), image-to-video (I2V), and video-to-video (V2V) models. - **Compare Closed-Source and Open-Source Models**: Compare 10 closed-source models and 3 open-source models, showcasing over 8,000 generated video cases. - **In-Depth Analysis of Model Capabilities**: Analyze the subjective capabilities of models from multiple dimensions (such as text alignment, visual and motion quality, composition, stability, creativity, etc.). - **Explore Practical Application Scenarios**: Based on survey results from hundreds of users, explore the performance of these models in ten practical applications. - **Propose Future Research Directions**: Discuss the challenges faced by current models and propose future research directions to advance video generation technology. ### Summary Through systematic evaluation and comparison, this paper aims to bridge the gap between academic research and industrial practice, providing a reference for the further development of video generation technology.