VideoCrafter2: Overcoming Data Limitations for High-Quality Video Diffusion Models

Haoxin Chen,Yong Zhang,Xiaodong Cun,Menghan Xia,Xintao Wang,Chao Weng,Ying Shan

2024-01-17

Abstract:Text-to-video generation aims to produce a video based on a given prompt. Recently, several commercial video models have been able to generate plausible videos with minimal noise, excellent details, and high aesthetic scores. However, these models rely on large-scale, well-filtered, high-quality videos that are not accessible to the community. Many existing research works, which train models using the low-quality WebVid-10M dataset, struggle to generate high-quality videos because the models are optimized to fit WebVid-10M. In this work, we explore the training scheme of video models extended from Stable Diffusion and investigate the feasibility of leveraging low-quality videos and synthesized high-quality images to obtain a high-quality video model. We first analyze the connection between the spatial and temporal modules of video models and the distribution shift to low-quality videos. We observe that full training of all modules results in a stronger coupling between spatial and temporal modules than only training temporal modules. Based on this stronger coupling, we shift the distribution to higher quality without motion degradation by finetuning spatial modules with high-quality images, resulting in a generic high-quality video model. Evaluations are conducted to demonstrate the superiority of the proposed method, particularly in picture quality, motion, and concept composition.

Computer Vision and Pattern Recognition

What problem does this paper attempt to address?

This paper focuses on how to train high-quality video diffusion models without high-quality video data. Although existing video generation models can generate realistic videos, they rely on large-scale, high-quality video data, which is not easy to obtain for the community. Researchers analyzed the training strategy of Stable Diffusion-based video models and found that training the entire module leads to stronger coupling between the spatial and temporal modules, which helps improve video quality by fine-tuning the spatial module with high-quality images without reducing motion quality.

The method proposed by them is to separate motion and appearance at the data level, using low-quality videos to ensure motion consistency and high-quality images to ensure image quality and conceptual combination ability. Specifically, a basic video model is trained using low-quality videos, and then the spatial module is fine-tuned only with high-resolution images and complex conceptual combinations to obtain a high-quality video model.

Experimental results show that this method outperforms other methods in terms of image quality, motion consistency, and conceptual combination ability. The paper also compares different training strategies and module adjustment methods and finds that directly fine-tuning the spatial module with high-quality images achieves the best results. In addition, the paper explores the use of synthetic images to improve the model's conceptual combination ability and alleviate the difficulty of capturing both concepts and motion simultaneously.

In conclusion, this paper addresses the problem of training high-quality video generation models in the absence of high-quality video data and proposes a data-level decoupling method that combines low-quality videos and high-quality images.