DreamFactory: Pioneering Multi-Scene Long Video Generation with a Multi-Agent Framework

Zhifei Xie,Daniel Tang,Dingwei Tan,Jacques Klein,Tegawend F. Bissyand,Saad Ezzini
2024-08-22
Abstract:Current video generation models excel at creating short, realistic clips, but struggle with longer, multi-scene videos. We introduce \texttt{DreamFactory}, an LLM-based framework that tackles this challenge. \texttt{DreamFactory} leverages multi-agent collaboration principles and a Key Frames Iteration Design Method to ensure consistency and style across long videos. It utilizes Chain of Thought (COT) to address uncertainties inherent in large language models. \texttt{DreamFactory} generates long, stylistically coherent, and complex videos. Evaluating these long-form videos presents a challenge. We propose novel metrics such as Cross-Scene Face Distance Score and Cross-Scene Style Consistency Score. To further research in this area, we contribute the Multi-Scene Videos Dataset containing over 150 human-rated videos.
Artificial Intelligence,Computation and Language,Computer Vision and Pattern Recognition,Software Engineering
What problem does this paper attempt to address?
### Problems the Paper Aims to Solve This paper aims to address the main challenges faced by current video generation models when generating long videos. Specifically, while existing video generation models can create short, realistic video clips, they perform poorly when generating multi-scene, long-duration videos. These issues include: 1. **Style Consistency**: Maintaining a consistent visual style and background across different scenes. 2. **Character Coherence**: Ensuring that characters' behavior and appearance remain consistent across different scenes. 3. **High-Quality Images**: Generating high-resolution, high-quality video clips. 4. **Complex Scene Management**: Handling the complex production steps involved in multi-scene video generation, rather than merely stitching together short video clips. To address these challenges, the paper introduces the **DreamFactory** framework. DreamFactory is based on large language models (LLM) and multi-agent collaboration technology, and it addresses the aforementioned issues through the following methods: - **Multi-Agent Collaboration**: Simulating a virtual animation production team where different agents play roles such as director, art director, screenwriter, etc., working together to complete script writing, storyboard creation, character design, keyframe development, and video synthesis. - **Keyframe Iterative Design**: Defining the concept of keyframes and proposing a keyframe iterative method to ensure continuity and consistency between video clips. - **Supervision Mechanism**: Introducing a monitoring role to ensure coherence between different frames. - **Integrated Image Vector Database**: Maintaining the stability of the creative process. Additionally, the paper proposes new evaluation metrics, such as the Cross-Scene Face Distance Score and the Cross-Scene Style Consistency Score, to assess the quality and consistency of the generated long videos. Through these methods, DreamFactory can automatically generate multi-scene, long-duration, and style-consistent high-quality videos.