Abstract:Large text-to-image diffusion models have achieved remarkable success in generating diverse, high-quality images. Additionally, these models have been successfully leveraged to edit input images by just changing the text prompt. But when these models are applied to videos, the main challenge is to ensure temporal consistency and coherence across frames. In this paper, we propose InFusion, a framework for zero-shot text-based video editing leveraging large pre-trained image diffusion models. Our framework specifically supports editing of multiple concepts with pixel-level control over diverse concepts mentioned in the editing prompt. Specifically, we inject the difference in features obtained with source and edit prompts from U-Net residual blocks of decoder layers. When these are combined with injected attention features, it becomes feasible to query the source contents and scale edited concepts along with the injection of unedited parts. The editing is further controlled in a fine-grained manner with mask extraction and attention fusion, which cut the edited part from the source and paste it into the denoising pipeline for the editing prompt. Our framework is a low-cost alternative to one-shot tuned models for editing since it does not require training. We demonstrated complex concept editing with a generalised image model (Stable Diffusion v1.5) using LoRA. Adaptation is compatible with all the existing image diffusion techniques. Extensive experimental results demonstrate the effectiveness of existing methods in rendering high-quality and temporally consistent videos.
What problem does this paper attempt to address?
### Problems the paper attempts to solve
This paper aims to solve several key challenges in text - guided multi - concept zero - shot video editing. Specifically, it attempts to solve the following problems:
1. **Temporal Consistency**: When existing large - scale pre - trained image diffusion models are applied to videos, the main challenge is to ensure the temporal consistency and coherence between video frames. That is, during the editing process, how to maintain the consistency between each frame of the video and avoid the phenomenon of jumps or incoherence.
2. **Zero - shot Editing**: To achieve a method for video editing without training or fine - tuning. Many existing methods require fine - tuning or training of pre - trained models, which is not only time - consuming but also requires a large amount of computing resources. Zero - shot methods can significantly reduce these requirements.
3. **Multi - concept Editing**: To support fine - grained control editing of multiple concepts. For example, it is possible to simultaneously change the vehicle type (such as from a silver Jeep to a Porsche) and the background environment (such as from the countryside to a landmark autumn scenery) in the same video. In addition, it is also necessary to be able to precisely control the editing range of each concept and avoid affecting the unedited parts.
### Solutions
To solve the above problems, the authors propose a framework named **InFusion**. The main features of this framework are as follows:
- **Inject (INJECT)**: By injecting the feature differences extracted from the residual blocks of the U - Net decoder layer and the attention features extracted from the source prompt and the editing prompt, the target concept is highlighted. These injection steps help to maintain the structure of the unedited parts during the editing process while amplifying the concept of the edited parts.
- **Attention Fusion (ATTENTION FUSION)**: By extracting masks from the cross - attention maps and combining the mixture of self - attention and cross - attention, the separation and fusion of the edited parts and the unedited parts are precisely controlled. This helps to preserve the structure and layout of the source content during the editing process.
- **Spatio - Temporal Attention (SPATIO - TEMPORAL ATTENTION)**: In order to ensure the temporal consistency of the video, a spatio - temporal attention mechanism is introduced. By initializing the spatial self - attention weights as the time attention weights and combining the key - frame attention, all frames are aligned with the key frames, thereby improving the coherence of the video.
### Experimental Results
The authors verified the effectiveness of the **InFusion** framework through a series of experiments. The experimental results show that this framework performs excellently in generating high - quality and temporally consistent videos, especially in multi - concept editing and fine - grained control. Specifically:
- **Qualitative Comparison**: Compared with several existing zero - shot methods (such as FateZero, vid2vid - zero, Pix2Video and Text2Video - Zero), **InFusion** can more accurately replace and edit objects and backgrounds in videos and maintain better temporal consistency.
- **Quantitative Evaluation**: Using the trained CLIP model for quantitative evaluation, the results show that **InFusion** is superior to other methods in terms of temporal consistency and per - frame editing accuracy.
- **User Study**: The practical application effect of **InFusion** was further verified through a user study, and users generally believed that this method performs best in terms of editing quality and temporal consistency.
In conclusion, the **InFusion** framework provides an efficient and flexible zero - shot text - guided video editing method, which can achieve fine - grained control editing of multiple concepts while maintaining the temporal consistency of the video.