MotionFollower: Editing Video Motion via Lightweight Score-Guided Diffusion

Shuyuan Tu,Qi Dai,Zihao Zhang,Sicheng Xie,Zhi-Qi Cheng,Chong Luo,Xintong Han,Zuxuan Wu,Yu-Gang Jiang

2024-05-31

Abstract:Despite impressive advancements in diffusion-based video editing models in altering video attributes, there has been limited exploration into modifying motion information while preserving the original protagonist's appearance and background. In this paper, we propose MotionFollower, a lightweight score-guided diffusion model for video motion editing. To introduce conditional controls to the denoising process, MotionFollower leverages two of our proposed lightweight signal controllers, one for poses and the other for appearances, both of which consist of convolution blocks without involving heavy attention calculations. Further, we design a score guidance principle based on a two-branch architecture, including the reconstruction and editing branches, which significantly enhance the modeling capability of texture details and complicated backgrounds. Concretely, we enforce several consistency regularizers and losses during the score estimation. The resulting gradients thus inject appropriate guidance to the intermediate latents, forcing the model to preserve the original background details and protagonists' appearances without interfering with the motion modification. Experiments demonstrate the competitive motion editing ability of MotionFollower qualitatively and quantitatively. Compared with MotionEditor, the most advanced motion editing model, MotionFollower achieves an approximately 80% reduction in GPU memory while delivering superior motion editing performance and exclusively supporting large camera movements and actions.

Computer Vision and Pattern Recognition

What problem does this paper attempt to address?

The paper aims to address the issue of modifying motion information in video editing while keeping the appearance of the main character and the background unchanged. Specifically, existing diffusion models have made significant progress in changing video attributes (such as style transfer and manipulation of the background and main character's appearance), but they fall short when it comes to modifying motion information. To this end, the authors propose a lightweight score-guided diffusion model called MotionFollower.

The main contributions of the paper include:
1. Proposing two lightweight signal controllers (a pose controller and a reference controller) to replace the original complex network structure, thereby reducing computational costs.
2. Introducing a new score-guided principle that allows the model to retain the semantic details of the source video during inference, such as the background and camera movements.
3. Compared to the state-of-the-art motion editing model MotionEditor, MotionFollower achieves approximately 80% reduction in GPU memory usage and performs excellently when processing videos with extensive camera movements and complex backgrounds.

Through these improvements, MotionFollower not only enhances the quality of video motion editing but also significantly reduces the demand for computational resources.