Abstract:The emergence of diffusion models has greatly propelled the progress in image and video generation. Recently, some efforts have been made in controllable video generation, including text-to-video, image-to-video generation, video editing, and video motion control, among which camera motion control is an important topic. However, existing camera motion control methods rely on training a temporal camera module, and necessitate substantial computation resources due to the large amount of parameters in video generation models. Moreover, existing methods pre-define camera motion types during training, which limits their flexibility in camera control, preventing the realization of some specific camera controls, such as various camera movements in films. Therefore, to reduce training costs and achieve flexible camera control, we propose COMD, a novel training-free video motion transfer model, which disentangles camera motions and object motions in source videos and transfers the extracted camera motions to new videos. We first propose a one-shot camera motion disentanglement method to extract camera motion from a single source video, which separates the moving objects from the background and estimates the camera motion in the moving objects region based on the motion in the background by solving a Poisson equation. Furthermore, we propose a few-shot camera motion disentanglement method to extract the common camera motion from multiple videos with similar camera motions, which employs a window-based clustering technique to extract the common features in temporal attention maps of multiple videos. Finally, we propose a motion combination method to combine different types of camera motions together, enabling our model a more controllable and flexible camera control. Extensive experiments demonstrate that our training-free approach can effectively decouple camera-object motion and apply the decoupled camera motion to a wide range of controllable video generation tasks, achieving flexible and diverse camera motion control. Project Page: https://sjtuplayer.github.io/projects/MotionMaster.

MotionFlow: Attention-Driven Motion Transfer in Video Diffusion Models

Video Motion Transfer with Diffusion Transformers

MotionShop: Zero-Shot Motion Transfer in Video Diffusion Models with Mixture of Score Guidance

OnlyFlow: Optical Flow based Motion Conditioning for Video Diffusion Models

COMD: Training-free Video Motion Transfer with Camera-Object Motion Disentanglement

Optical Flow as Spatial-Temporal Attention Learners

MoVideo: Motion-Aware Video Generation with Diffusion Models

Monkey See, Monkey Do: Harnessing Self-attention in Motion Diffusion for Zero-shot Motion Transfer

MoTrans: Customized Motion Transfer with Text-driven Video Diffusion Models

Video Diffusion Models are Training-free Motion Interpreter and Controller

MotionEditor: Editing Video Motion via Content-Aware Diffusion

Video Frame Interpolation with Flow Transformer

MV-Diffusion: Motion-aware Video Diffusion Model

Animate Your Motion: Turning Still Images into Dynamic Videos

RecMoDiffuse: Recurrent Flow Diffusion for Human Motion Generation

TokenFlow: Consistent Diffusion Features for Consistent Video Editing

MotionFollower: Editing Video Motion via Lightweight Score-Guided Diffusion

Human Motion Diffusion as a Generative Prior

Dreamix: Video Diffusion Models are General Video Editors

TransFlow: Transformer as Flow Learner