MotionStone: Decoupled Motion Intensity Modulation with Diffusion Transformer for Image-to-Video Generation

Shuwei Shi,Biao Gong,Xi Chen,Dandan Zheng,Shuai Tan,Zizheng Yang,Yuyuan Li,Jingwen He,Kecheng Zheng,Jingdong Chen,Ming Yang,Yinqiang Zheng

2024-12-08

Abstract:The image-to-video (I2V) generation is conditioned on the static image, which has been enhanced recently by the motion intensity as an additional control signal. These motion-aware models are appealing to generate diverse motion patterns, yet there lacks a reliable motion estimator for training such models on large-scale video set in the wild. Traditional metrics, e.g., SSIM or optical flow, are hard to generalize to arbitrary videos, while, it is very tough for human annotators to label the abstract motion intensity neither. Furthermore, the motion intensity shall reveal both local object motion and global camera movement, which has not been studied before. This paper addresses the challenge with a new motion estimator, capable of measuring the decoupled motion intensities of objects and cameras in video. We leverage the contrastive learning on randomly paired videos and distinguish the video with greater motion intensity. Such a paradigm is friendly for annotation and easy to scale up to achieve stable performance on motion estimation. We then present a new I2V model, named MotionStone, developed with the decoupled motion estimator. Experimental results demonstrate the stability of the proposed motion estimator and the state-of-the-art performance of MotionStone on I2V generation. These advantages warrant the decoupled motion estimator to serve as a general plug-in enhancer for both data processing and video generation training.

Computer Vision and Pattern Recognition

What problem does this paper attempt to address?

The problem that this paper attempts to solve is: in the image - to - video (I2V) generation task, how to achieve accurate modeling and control of the object and camera motion intensities in the video. Specifically, existing methods face the following challenges when estimating motion intensities:

1. **Lack of a reliable motion estimator**: Existing motion - aware models lack a motion estimator that can be reliably trained on large - scale video datasets.
2. **Traditional metrics are difficult to generalize**: Traditional metrics such as SSIM or optical flow are difficult to generalize to arbitrary videos.
3. **Difficult for human annotation**: It is very difficult for human annotators to annotate abstract motion intensities.
4. **Complex motion patterns**: Videos in the real world contain complex motion patterns, including local object motion and global camera motion, and existing methods cannot distinguish these different types of motion.

To solve these problems, the paper proposes a new I2V generation model - MotionStone, whose core is an independent motion estimator that can decouple and predict the motion intensities of objects and cameras in the video respectively. This motion estimator is trained by a contrastive learning strategy and applied to the diffusion model to achieve accurate control of motion intensities.

### Main contributions

1. **Proposed a decoupled motion estimator**: It can predict the motion intensities of objects and cameras respectively, solving the problem that traditional methods cannot distinguish different types of motion.
2. **Introduced a contrastive learning strategy**: By contrasting video pairs to train the motion estimator, the annotation process is simplified and the generalization ability of the model is improved.
3. **Designed a new I2V generation framework**: Combining the decoupled motion estimator and the diffusion model, accurate control of the motion patterns in the generated video is achieved.
4. **Demonstrated superior experimental results**: Through quantitative and qualitative experiments, the superior performance of MotionStone in the I2V generation task is proved.

### Solutions

The paper solves the above problems through the following steps:

1. **Construct training data**: Developed a simple and intuitive annotation method, allowing annotators to compare the relative motion intensities between video pairs instead of assigning specific scores to each video.
2. **Design motion estimator architecture**: Use TAdaConv as the backbone network and combine two MLP heads to predict the motion intensities of objects and cameras.
3. **Training configuration**: Use contrastive learning and regression loss to jointly train the motion estimator to ensure that it can accurately predict relative motion intensities and has sufficient discrimination.
4. **Condition injection design**: During the diffusion model training process, the predicted motion intensity values are mapped into high - dimensional vectors by two independent MLPs and added to time step t to keep the two motion types decoupled.

Through these methods, MotionStone can achieve accurate control of the motion intensities of objects and cameras when generating videos, thereby generating more realistic and controllable video content.