Abstract:We present Stable Video Diffusion - a latent video diffusion model for high-resolution, state-of-the-art text-to-video and image-to-video generation. Recently, latent diffusion models trained for 2D image synthesis have been turned into generative video models by inserting temporal layers and finetuning them on small, high-quality video datasets. However, training methods in the literature vary widely, and the field has yet to agree on a unified strategy for curating video data. In this paper, we identify and evaluate three different stages for successful training of video LDMs: text-to-image pretraining, video pretraining, and high-quality video finetuning. Furthermore, we demonstrate the necessity of a well-curated pretraining dataset for generating high-quality videos and present a systematic curation process to train a strong base model, including captioning and filtering strategies. We then explore the impact of finetuning our base model on high-quality data and train a text-to-video model that is competitive with closed-source video generation. We also show that our base model provides a powerful motion representation for downstream tasks such as image-to-video generation and adaptability to camera motion-specific LoRA modules. Finally, we demonstrate that our model provides a strong multi-view 3D-prior and can serve as a base to finetune a multi-view diffusion model that jointly generates multiple views of objects in a feedforward fashion, outperforming image-based methods at a fraction of their compute budget. We release code and model weights at https://github.com/Stability-AI/generative-models .

Stable Diffusion Dataset Generation for Downstream Classification Tasks

Mind the Gap Between Synthetic and Real: Utilizing Transfer Learning to Probe the Boundaries of Stable Diffusion Generated Data

Generative Dataset Distillation Based on Diffusion Model

Stable Diffusion for Data Augmentation in COCO and Weed Datasets

Accelerating Dataset Distillation Via Model Augmentation

Data-Efficient Generation for Dataset Distillation

Evaluating a Synthetic Image Dataset Generated with Stable Diffusion

Controllable Image Synthesis of Industrial Data Using Stable Diffusion

Efficient Dataset Distillation via Minimax Diffusion

Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasets

Improving the Effectiveness of Deep Generative Data

Generative Dataset Distillation: Balancing Global Structure and Local Details

Synthetic Data from Diffusion Models Improves ImageNet Classification

InstaGen: Enhancing Object Detection by Training on Synthetic Dataset

Dataset Diffusion: Diffusion-based Synthetic Dataset Generation for Pixel-Level Semantic Segmentation

Enhancing Diffusion-Based Image Synthesis with Robust Classifier Guidance

Visual Car Brand Classification by Implementing a Synthetic Image Dataset Creation Pipeline

Toward Understanding Generative Data Augmentation

Improving Adversarial Robustness by Contrastive Guided Diffusion Process