VideoCrafter2: Overcoming Data Limitations for High-Quality Video Diffusion Models

Haoxin Chen,Yong Zhang,Xiaodong Cun,Menghan Xia,Xintao Wang,Chao Weng,Ying Shan

DOI: https://doi.org/10.48550/arXiv.2401.09047

2024-01-17

Computer Vision and Pattern Recognition

Abstract:Text-to-video generation aims to produce a video based on a given prompt. Recently, several commercial video models have been able to generate plausible videos with minimal noise, excellent details, and high aesthetic scores. However, these models rely on large-scale, well-filtered, high-quality videos that are not accessible to the community. Many existing research works, which train models using the low-quality WebVid-10M dataset, struggle to generate high-quality videos because the models are optimized to fit WebVid-10M. In this work, we explore the training scheme of video models extended from Stable Diffusion and investigate the feasibility of leveraging low-quality videos and synthesized high-quality images to obtain a high-quality video model. We first analyze the connection between the spatial and temporal modules of video models and the distribution shift to low-quality videos. We observe that full training of all modules results in a stronger coupling between spatial and temporal modules than only training temporal modules. Based on this stronger coupling, we shift the distribution to higher quality without motion degradation by finetuning spatial modules with high-quality images, resulting in a generic high-quality video model. Evaluations are conducted to demonstrate the superiority of the proposed method, particularly in picture quality, motion, and concept composition.

What problem does this paper attempt to address?

The problem that this paper attempts to solve is: how to train a high - quality video generation model in the absence of high - quality video data. Specifically, the existing high - quality video generation models rely on large - scale, high - quality, and carefully - screened video datasets, which are usually not accessible to the research community. Therefore, the author proposes a method to use low - quality videos and high - quality images to train a high - quality video generation model.

### Problem Background

1. **Scarcity of High - Quality Video Data**:
- It is difficult to obtain large - scale, high - quality video datasets, mainly due to copyright restrictions and high post - processing costs.
- Although the existing open - source video datasets (such as WebVid - 10M) are rich in content, the picture quality is poor, and most video resolutions are around 320p.

2. **Limitations of Existing Models**:
- Many existing video generation models rely on high - quality videos for training, resulting in poor generation effects when using low - quality videos.
- Some commercial models can generate high - quality videos, but the datasets they use are not public, which hinders the research progress in academia.

### Solution

The author proposes a new training method to overcome data limitations through the following steps:

1. **Decoupling Appearance and Motion**:
- Use low - quality videos to learn motion features, and at the same time use high - quality images to learn appearance and aesthetic features.
- By analyzing the connection between the spatial module and the temporal module, it is found that fully training all modules can enhance the coupling between the two, so as to better tolerate subsequent spatial module adjustments.

2. **Training Process**:
- First, use low - quality videos to comprehensively train the video model.
- Then, use high - quality images to fine - tune the spatial module to improve the picture quality and aesthetic score of the generated video.

3. **Improvement of Concept Combination Ability**:
- Use synthesized complex - concept images (such as images generated by Midjourney) for partial fine - tuning to enhance the concept combination ability of the video model and avoid the burden of simultaneously capturing concepts and motion.

### Main Contributions

- Proposed a method to train a high - quality video generation model without using high - quality videos.
- Conducted an in - depth analysis of the connection between the spatial module and the temporal module and discovered the key to improving the quality of generated videos.
- Designed an effective training pipeline, that is, first comprehensively train the video model, and then use high - quality images to fine - tune the spatial module.

Through this method, the author has successfully trained a video generation model with high visual quality, motion consistency, and concept combination ability only using low - quality videos and high - quality images.