Abstract:Image animation is to animate a still image of the object of interest using poses extracted from another video sequence. Through training on a large-scale video dataset, most existing approaches aim to explore disentangled appearance and pose representations of training frames. Then, the desired output with a specific appearance and pose can be synthesized via recombining learned representations. However, in some real-world applications, test images may lack the corresponding video ground-truth or follow a different distribution than the distribution of the training video frames (i.e., different domains), which largely limit the performance of existing methods. In this paper, we propose domain-independent pose representations that are compatible with and accessible by still images from a different domain. Specifically, we devise a two-stage self-supervised pose adaptation framework for general image animation tasks. A domain-independent pose adaptation generative adversarial network (DIPA-GAN) and a shuffle-patch generative adversarial network (Shuffle-patch GAN) are proposed to penalize the rationality of the synthesized frame's pose and appearance, respectively. Finally, experiments evaluated on various image animation tasks, which include same/cross-domain moving objects, facial expression transfer and human pose retargeting, demonstrate the superiority of the proposed framework over prior literature. Impact Statement—Image animation is a popular technology in video production. Benefiting from the rapid development of artificial intelligence (AI), recent image animation algorithms have been widely used in real-world applications, such as virtual AI news anchor, virtual try-on, and face swapping. However, most existing methods are designed for specific cases. To animate a new portrait, users are asked to collect hundreds of images of the same person and train a new model. The technology proposed in this paper overcomes these training limitations and generalizes image animations. In the challenging cross-domain facial expression transfer task, the user study demonstrated that our technology achieved more than 20% increase in animation success rate. The proposed technology could benefit users in a wide variety of industries including movie production, virtual reality, social media and online retail.

Image Animation via Joint Attention Mechanism

Visual Attention Based on Long-Short Term Memory Model for Image Caption Generation

Generating Video Animation from Single Still Image in Social Media Based on Intelligent Computing.

Thin-Plate Spline Motion Model for Image Animation

Coherent Image Animation Using Spatial-Temporal Correspondence

Motion Transformer for Unsupervised Image Animation

Video-driven state-aware facial animation

Animate Anyone: Consistent and Controllable Image-to-Video Synthesis for Character Animation

DynamiCrafter: Animating Open-domain Images with Video Diffusion Priors

Controllable Longer Image Animation with Diffusion Models

AnimateAnything: Fine-Grained Open Domain Image Animation with Motion Guidance

High Quality Human Image Animation using Regional Supervision and Motion Blur Condition

Multi-scale unsupervised network for infrared and visible image fusion based on joint attention mechanism

Self-Supervised Pose Adaptation for Cross-Domain Image Animation.

Face Animation with Multiple Source Images

Image-to-Video Generation via 3D Facial Dynamics

Facial Expression Animation Through Action Units Transfer in Latent Space.

Animating Landscape: Self-Supervised Learning of Decoupled Motion and Appearance for Single-Image Video Synthesis

Improve Image Captioning By Self-Attention

Variational Joint Self‐attention for Image Captioning

Multi-Scale Attention Generative Adversarial Networks for Video Frame Interpolation