Abstract:In this paper, we propose Occlusion-Aware Warping GAN (OAW-GAN), a unified Human Video Synthesis (HVS) framework that can uniformly tackle human video motion transfer, attribute editing, as well as inpainting. This is the first work to our knowledge that can handle all these tasks within a one-time trained model. Although existing GAN-based HVS methods have achieved great success, they either can’t preserve appearance details due to the loss of spatial consistency between the synthesized target frames and the input source images, or generate incoherent video results due to the loss of temporal consistency among frames. Besides, most of them lack the ability to create new contents while keeping existing ones, failing especially when some regions in the target are invisible in the source due to self-occlusion. To address these limitations, we first introduce Coarse-to-Fine Flow Warping Network (C2F-FWN) to estimate spatial-temporal consistent transformation between source and target, as well as occlusion mask indicating which parts in the target are invisible in the source. Then, the flow and the mask are scaled and fed into the pyramidal stages of our OAW-GAN, guiding Occlusion-Aware Synthesis (OAS) that can be abstracted into visible part re-utilization and invisible part inpainting at the feature level, which effectively alleviates the self-occlusion problem. Extensive experiments conducted on both human video (i.e., iPER, SoloDance)Keywords are desired. please provide if necessary. and image (i.e., DeepFashion) datasets demonstrate the superiority of our approach to existing state-of-the-arts. We also show that, besides motion transfer task that previous works concern, our framework can further achieve attribute editing and texture inpainting, which paves the way towards unified HVS.

Autoregressive GAN for Semantic Unconditional Head Motion Generation

OAW-GAN: Occlusion-Aware Warping GAN for Unified Human Video Synthesis

GANHead: Towards Generative Animatable Neural Head Avatars

Audio-driven Talking Face Video Generation with Natural Head Pose

A Comprehensive Multi-scale Approach for Speech and Dynamics Synchrony in Talking Head Generation

Realistic talking face animation with speech-induced head motion

End-to-End Speech-Driven Facial Animation with Temporal GANs

GANimator: Neural Motion Synthesis from a Single Sequence

Audio-Semantic Enhanced Pose-Driven Talking Head Generation

Realistic Speech-Driven Facial Animation with GANs

Talking-head video generation with long short-term contextual semantics

GANimation: Anatomically-aware Facial Animation from a Single Image

Generative Model for Skeletal Human Movements Based on Conditional DC-GAN Applied to Pseudo-Images

Dynamic Facial Expression Generation on Hilbert Hypersphere With Conditional Wasserstein Generative Adversarial Nets

Towards Practical Single-shot Motion Synthesis

G3FA: Geometry-guided GAN for Face Animation

Audio2Head: Audio-driven One-shot Talking-head Generation with Natural Head Motion

High-Fidelity and Freely Controllable Talking Head Video Generation

GANemotion: Increase Vitality of Characters in Videos by Generative Adversary Networks

Talking-head Generation with Rhythmic Head Motion

Language-Guided Face Animation by Recurrent StyleGAN-based Generator