VIA: Unified Spatiotemporal Video Adaptation Framework for Global and Local Video Editing

Jing Gu,Yuwei Fang,Ivan Skorokhodov,Peter Wonka,Xinya Du,Sergey Tulyakov,Xin Eric Wang

2024-10-16

Abstract:Video editing is a cornerstone of digital media, from entertainment and education to professional communication. However, previous methods often overlook the necessity of comprehensively understanding both global and local contexts, leading to inaccurate and inconsistent edits in the spatiotemporal dimension, especially for long videos. In this paper, we introduce VIA, a unified spatiotemporal Video Adaptation framework for global and local video editing, pushing the limits of consistently editing minute-long videos. First, to ensure local consistency within individual frames, we designed test-time editing adaptation to adapt a pre-trained image editing model for improving consistency between potential editing directions and the text instruction, and adapt masked latent variables for precise local control. Furthermore, to maintain global consistency over the video sequence, we introduce spatiotemporal adaptation that recursively gather consistent attention variables in key frames and strategically applies them across the whole sequence to realize the editing effects. Extensive experiments demonstrate that, compared to baseline methods, our VIA approach produces edits that are more faithful to the source videos, more coherent in the spatiotemporal context, and more precise in local control. More importantly, we show that VIA can achieve consistent long video editing in minutes, unlocking the potential for advanced video editing tasks over long video sequences.

Computer Vision and Pattern Recognition,Artificial Intelligence,Multimedia

What problem does this paper attempt to address?

### What problems does this paper attempt to solve?

This paper aims to solve several key challenges in video editing, especially maintaining global and local consistency in long - videos. Specifically:

1. **Understanding of global and local context**:
- **Deficiencies of existing methods**: Previous methods often overlooked the necessity of comprehensively understanding the global and local context of videos, resulting in inaccurate and inconsistent editing in the spatio - temporal dimension (i.e., time and space).
- **Challenges of long - videos**: Especially when dealing with long - time videos, it is particularly difficult to maintain long - term spatio - temporal consistency.

2. **Editing consistency**:
- **Inter - frame consistency**: Ensure that the editing results of each frame in the video are semantically consistent with the text instructions, and the non - target areas are not affected.
- **Temporal consistency**: Ensure that the editing effects in the entire video sequence transition smoothly, avoiding abrupt changes or visual inconsistencies.

3. **Efficient processing of long - videos**:
- **Limitations of existing methods**: Many existing video editing methods can usually only process short - time videos (less than 200 frames) due to model capacity and hardware limitations.
- **Innovation of VIA**: The paper proposes a new unified spatio - temporal video adaptation framework VIA (Unified Spatiotemporal Video Adaptation), which can achieve consistent and efficient editing in long - videos at the minute - level.

### Main contributions of VIA

- **Editing adaptation mechanism at test time**: Improve semantic understanding and editing consistency within a single frame by adjusting the pre - trained image editing model.
- **Spatio - temporal attention adaptation**: Introduce a two - step "collect - exchange" process to ensure global editing consistency throughout the video sequence.
- **Local latent adaptation**: Ensure inter - frame consistency by controlling local editing, especially for precise editing within specified areas.
- **Performance improvement**: Compared with existing technologies, VIA shows significantly better editing quality and efficiency in human evaluation and automatic evaluation.

Through these innovations, VIA can achieve more consistent and more precise editing effects in longer videos, promoting the development of video editing technology.