StoryAgent: Customized Storytelling Video Generation via Multi-Agent Collaboration

Panwen Hu,Jin Jiang,Jianqi Chen,Mingfei Han,Shengcai Liao,Xiaojun Chang,Xiaodan Liang

2024-11-08

Abstract:The advent of AI-Generated Content (AIGC) has spurred research into automated video generation to streamline conventional processes. However, automating storytelling video production, particularly for customized narratives, remains challenging due to the complexity of maintaining subject consistency across shots. While existing approaches like Mora and AesopAgent integrate multiple agents for Story-to-Video (S2V) generation, they fall short in preserving protagonist consistency and supporting Customized Storytelling Video Generation (CSVG). To address these limitations, we propose StoryAgent, a multi-agent framework designed for CSVG. StoryAgent decomposes CSVG into distinct subtasks assigned to specialized agents, mirroring the professional production process. Notably, our framework includes agents for story design, storyboard generation, video creation, agent coordination, and result evaluation. Leveraging the strengths of different models, StoryAgent enhances control over the generation process, significantly improving character consistency. Specifically, we introduce a customized Image-to-Video (I2V) method, LoRA-BE, to enhance intra-shot temporal consistency, while a novel storyboard generation pipeline is proposed to maintain subject consistency across shots. Extensive experiments demonstrate the effectiveness of our approach in synthesizing highly consistent storytelling videos, outperforming state-of-the-art methods. Our contributions include the introduction of StoryAgent, a versatile framework for video generation tasks, and novel techniques for preserving protagonist consistency.

Computer Vision and Pattern Recognition,Artificial Intelligence,Multiagent Systems

What problem does this paper attempt to address?

The paper attempts to address the problem of maintaining the coherence and consistency of the main character across multiple shots in Customized Storytelling Video Generation (CSVG). Specifically, existing methods struggle to ensure the consistency of the main character's appearance and behavior in different shots when generating story videos. This results in visually incoherent videos, affecting the quality and viewing experience of the videos.

To tackle this challenge, the paper proposes a multi-agent framework named StoryAgent. This framework improves the coherence and consistency of generated videos by decomposing the CSVG task into multiple sub-tasks and assigning them to specialized agents. The main contributions include:

1. **Proposing the StoryAgent framework**: This is a multi-agent system that includes several agents such as a story designer, storyboard generator, video creator, agent manager, and observer. Each agent is responsible for specific tasks, working collaboratively to generate high-quality story videos.
2. **Introducing a customized image-to-video generation method (LoRA-BE)**: By using Low-Rank Adaptation with Block-wise Embeddings, this method enhances the consistency of the main character during the video generation process.
3. **Experimental validation**: Extensive experiments on public and custom datasets demonstrate the effectiveness of the proposed method, especially in generating highly consistent customized story videos.

These innovations enable StoryAgent to significantly improve the consistency and coherence of the main character across different shots when generating customized story videos, thereby enhancing the overall video quality.