AVID: Any-Length Video Inpainting with Diffusion Model

Zhixing Zhang,Bichen Wu,Xiaoyan Wang,Yaqiao Luo,Luxin Zhang,Yinan Zhao,Peter Vajda,Dimitris Metaxas,Licheng Yu

2024-03-30

Abstract:Recent advances in diffusion models have successfully enabled text-guided image inpainting. While it seems straightforward to extend such editing capability into the video domain, there have been fewer works regarding text-guided video inpainting. Given a video, a masked region at its initial frame, and an editing prompt, it requires a model to do infilling at each frame following the editing guidance while keeping the out-of-mask region intact. There are three main challenges in text-guided video inpainting: ($i$) temporal consistency of the edited video, ($ii$) supporting different inpainting types at different structural fidelity levels, and ($iii$) dealing with variable video length. To address these challenges, we introduce Any-Length Video Inpainting with Diffusion Model, dubbed as AVID. At its core, our model is equipped with effective motion modules and adjustable structure guidance, for fixed-length video inpainting. Building on top of that, we propose a novel Temporal MultiDiffusion sampling pipeline with a middle-frame attention guidance mechanism, facilitating the generation of videos with any desired duration. Our comprehensive experiments show our model can robustly deal with various inpainting types at different video duration ranges, with high quality. More visualization results are made publicly available at <a class="link-external link-https" href="https://zhang-zx.github.io/AVID/" rel="external noopener nofollow">this https URL</a> .

Computer Vision and Pattern Recognition

What problem does this paper attempt to address?

The main problem that this paper attempts to solve is text - guided video inpainting, especially how to handle different types of video inpainting tasks while maintaining temporal coherence and support video inputs of any length. Specifically, the challenges faced by the paper include:

1. **Temporal Consistency**: The repaired video content needs to be temporally consistent. For example, if an object is replaced throughout the video, its features (such as color, shape, etc.) should remain unchanged from start to finish.
2. **Support for Different Types of Inpainting Tasks**: Video inpainting can include object replacement (such as replacing one car with another), texture reset (such as changing the color of leaves), and video expansion (such as filling the upper and lower boundaries of the video). Different tasks have different requirements for structural fidelity.
3. **Handling Videos of Any Length**: The length of the input video may vary, so the model needs to be able to flexibly handle videos of different lengths.

To address these challenges, the authors propose a framework named AVID (Any - Length Video Inpainting with Diffusion Model). The core innovations of this framework include:

- **Motion Module**: By introducing the motion module, ensure the temporal consistency of the content in the repaired area.
- **Structure - Guiding Module**: Adjust the fidelity of the output content to the original video structure according to different types of inpainting tasks.
- **Zero - Shot Generation Pipeline**: Combined with the intermediate - frame attention - guiding mechanism, enable the model to handle videos of any length without additional training.

Through these methods, AVID can flexibly handle various video inpainting tasks while maintaining temporal coherence and detail integrity.