Abstract:Diffusion models have achieved great progress in image animation due to powerful generative capabilities. However, maintaining spatio-temporal consistency with detailed information from the input static image over time (e.g., style, background, and object of the input static image) and ensuring smoothness in animated video narratives guided by textual prompts still remains challenging. In this paper, we introduce Cinemo, a novel image animation approach towards achieving better motion controllability, as well as stronger temporal consistency and smoothness. In general, we propose three effective strategies at the training and inference stages of Cinemo to accomplish our goal. At the training stage, Cinemo focuses on learning the distribution of motion residuals, rather than directly predicting subsequent via a motion diffusion model. Additionally, a structural similarity index-based strategy is proposed to enable Cinemo to have better controllability of motion intensity. At the inference stage, a noise refinement technique based on discrete cosine transformation is introduced to mitigate sudden motion changes. Such three strategies enable Cinemo to produce highly consistent, smooth, and motion-controllable results. Compared to previous methods, Cinemo offers simpler and more precise user controllability. Extensive experiments against several state-of-the-art methods, including both commercial tools and research approaches, across multiple metrics, demonstrate the effectiveness and superiority of our proposed approach.

What problem does this paper attempt to address?

The problems that this paper attempts to solve are two main challenges in Image Animation: **Image Consistency** and **Motion Controllability**. Specifically: 1. **Image Consistency**: - During the generation process of animated videos, the consistency of detailed information (such as shape, color, texture, etc.) with the input static image may be lost, resulting in significant differences between the generated video frames and the original image, which affects the overall sense of reality and coherence. 2. **Motion Controllability**: - Existing methods may not be able to accurately respond to given text prompts when generating videos, causing the generated motion sequences to deviate from the context described in the text and unable to accurately reflect the motion patterns expected by users. To solve these problems, the author proposes a new model named **Cinemo**, which achieves better image consistency and motion controllability through the following three innovative strategies: ### 1. Motion Residual Learning - **Problem**: Existing methods directly predict subsequent frames, which easily leads to distortion of video content and it is difficult to maintain the amount of detail in the input image. - **Solution**: Cinemo introduces a new learning strategy that focuses on learning motion residuals instead of directly predicting subsequent frames. Specifically, in each diffusion time step, the appearance information of the input static image is added to the noisy motion residuals, and it is spliced with the appearance information and then input into the model. This method can more effectively guide the model to generate motion residuals aligned with the prompts. ### 2. Structure Similarity Index (SSIM) for Motion Intensity Control - **Problem**: Existing methods usually rely on frame rate (FPS) or optical flow to adjust motion intensity, but these methods either cannot accurately reflect motion intensity or have too high a computational cost. - **Solution**: Cinemo proposes a strategy based on the Structure Similarity Index (SSIM) for fine - grained control of motion intensity. By calculating the average SSIM value between adjacent frames, Cinemo can better control the motion intensity of the generated video, ensuring that the motion changes meet the user's expectations. ### 3. DCT - based Inference Noise Refinement - **Problem**: The noise inputs in the training and inference stages are different, which may lead to the accumulation of inference errors, especially in the low - frequency components. - **Solution**: Cinemo introduces the DCTInit strategy, which uses the Discrete Cosine Transform (DCT) to extract low - frequency components to optimize the initial inference noise. This helps to stabilize the generation process, reduce sudden motion changes, and improve the quality of the generated video. Through these strategies, Cinemo can generate high - quality animated videos that are highly consistent with the input static image and have controllable motion, showing better performance and flexibility compared to existing methods.

Cinemo: Consistent and Controllable Image Animation with Motion Diffusion Models

Controllable Longer Image Animation with Diffusion Models

Animate Your Motion: Turning Still Images into Dynamic Videos

Animate Anyone: Consistent and Controllable Image-to-Video Synthesis for Character Animation

Continuous Piecewise-Affine Based Motion Model for Image Animation

Motion-Conditioned Diffusion Model for Controllable Video Synthesis

Motion Consistency Model: Accelerating Video Diffusion with Disentangled Motion-Appearance Distillation

DynamiCrafter: Animating Open-domain Images with Video Diffusion Priors

COMD: Training-free Video Motion Transfer with Camera-Object Motion Disentanglement

Flexible Motion In-betweening with Diffusion Models

Enhanced Fine-Grained Motion Diffusion for Text-Driven Human Motion Synthesis

MOFA-Video: Controllable Image Animation via Generative Motion Field Adaptions in Frozen Image-to-Video Diffusion Model

MagicAnimate: Temporally Consistent Human Image Animation using Diffusion Model

Cinematographic Camera Diffusion Model

AnimateDiff: Animate Your Personalized Text-to-Image Diffusion Models without Specific Tuning

MotionCrafter: One-Shot Motion Customization of Diffusion Models

Understanding Text-driven Motion Synthesis with Keyframe Collaboration via Diffusion Models

Robust Diffusion‐based Motion In‐betweening