Abstract:In this paper, we propose Occlusion-Aware Warping GAN (OAW-GAN), a unified Human Video Synthesis (HVS) framework that can uniformly tackle human video motion transfer, attribute editing, as well as inpainting. This is the first work to our knowledge that can handle all these tasks within a one-time trained model. Although existing GAN-based HVS methods have achieved great success, they either can’t preserve appearance details due to the loss of spatial consistency between the synthesized target frames and the input source images, or generate incoherent video results due to the loss of temporal consistency among frames. Besides, most of them lack the ability to create new contents while keeping existing ones, failing especially when some regions in the target are invisible in the source due to self-occlusion. To address these limitations, we first introduce Coarse-to-Fine Flow Warping Network (C2F-FWN) to estimate spatial-temporal consistent transformation between source and target, as well as occlusion mask indicating which parts in the target are invisible in the source. Then, the flow and the mask are scaled and fed into the pyramidal stages of our OAW-GAN, guiding Occlusion-Aware Synthesis (OAS) that can be abstracted into visible part re-utilization and invisible part inpainting at the feature level, which effectively alleviates the self-occlusion problem. Extensive experiments conducted on both human video (i.e., iPER, SoloDance)Keywords are desired. please provide if necessary. and image (i.e., DeepFashion) datasets demonstrate the superiority of our approach to existing state-of-the-arts. We also show that, besides motion transfer task that previous works concern, our framework can further achieve attribute editing and texture inpainting, which paves the way towards unified HVS.

NUWA-Infinity: Autoregressive over Autoregressive Generation for Infinite Visual Synthesis

OAW-GAN: Occlusion-Aware Warping GAN for Unified Human Video Synthesis

3D-Aware Image Synthesis Via Learning Structural and Textural Representations

N\"UWA: Visual Synthesis Pre-training for Neural visUal World creAtion

Infinity: Scaling Bitwise AutoRegressive Modeling for High-Resolution Image Synthesis

NUWA-XL: Diffusion over Diffusion for eXtremely Long Video Generation

Infinite Photorealistic Worlds using Procedural Generation

DragNUWA: Fine-grained Control in Video Generation by Integrating Text, Image, and Trajectory

Autoregressive Omni-Aware Outpainting for Open-Vocabulary 360-Degree Image Generation

Video-Infinity: Distributed Long Video Generation

Randomized Autoregressive Visual Generation

UniReal: Universal Image Generation and Editing via Learning Real-world Dynamics

L-MAGIC: Language Model Assisted Generation of Images with Coherence

UniAnimate: Taming Unified Video Diffusion Models for Consistent Human Image Animation

Unsupervised Image Generation with Infinite Generative Adversarial Networks

UniControl: A Unified Diffusion Model for Controllable Visual Generation In the Wild

Generating Videos with Dynamics-aware Implicit Generative Adversarial Networks

CD-NGP: A Fast Scalable Continual Representation for Dynamic Scenes

UniAudio: An Audio Foundation Model Toward Universal Audio Generation

AV-NeRF: Learning Neural Fields for Real-World Audio-Visual Scene Synthesis

$\infty$-Brush: Controllable Large Image Synthesis with Diffusion Models in Infinite Dimensions