Highly Detailed and Temporal Consistent Video Stylization via Synchronized Multi-Frame Diffusion

Minshan Xie,Hanyuan Liu,Chengze Li,Tien-Tsin Wong

2023-11-24

Abstract:Text-guided video-to-video stylization transforms the visual appearance of a source video to a different appearance guided on textual prompts. Existing text-guided image diffusion models can be extended for stylized video synthesis. However, they struggle to generate videos with both highly detailed appearance and temporal consistency. In this paper, we propose a synchronized multi-frame diffusion framework to maintain both the visual details and the temporal consistency. Frames are denoised in a synchronous fashion, and more importantly, information of different frames is shared since the beginning of the denoising process. Such information sharing ensures that a consensus, in terms of the overall structure and color distribution, among frames can be reached in the early stage of the denoising process before it is too late. The optical flow from the original video serves as the connection, and hence the venue for information sharing, among frames. We demonstrate the effectiveness of our method in generating high-quality and diverse results in extensive experiments. Our method shows superior qualitative and quantitative results compared to state-of-the-art video editing methods.

Computer Vision and Pattern Recognition

What problem does this paper attempt to address?

This paper attempts to address the problem of maintaining high detail and temporal consistency during video stylization. Existing text-guided image diffusion models, although extendable for video stylization, face difficulties in generating videos with high detail and temporal consistency. This paper proposes a synchronized multi-frame diffusion framework aimed at preserving visual details and temporal consistency by sharing information between different frames during the denoising process. Specifically, this method utilizes the optical flow of the original video as a connection for inter-frame information sharing, ensuring consistency in overall structure and color distribution at the early stages of the denoising process. This helps in generating high-quality and diverse video results.

The main contributions of the paper include:
1. Proposing a consensus-based approach where all frames participate equally and synchronously in generating stylized content, rather than the traditional "warp-and-patch" method.
2. Designing a multi-frame fusion module that unifies the visual content of intermediate generated video frames by adding temporal consistency constraints at each denoising step.

Through extensive qualitative and quantitative experiments, this method performs excellently on all evaluation metrics, capable of generating highly detailed video content while maintaining temporal consistency and adhering to user text prompts.