Abstract:Recent advancements in human video synthesis have enabled the generation of high-quality videos through the application of stable diffusion models. However, existing methods predominantly concentrate on animating solely the human element (the foreground) guided by pose information, while leaving the background entirely static. Contrary to this, in authentic, high-quality videos, backgrounds often dynamically adjust in harmony with foreground movements, eschewing stagnancy. We introduce a technique that concurrently learns both foreground and background dynamics by segregating their movements using distinct motion representations. Human figures are animated leveraging pose-based motion, capturing intricate actions. Conversely, for backgrounds, we employ sparse tracking points to model motion, thereby reflecting the natural interaction between foreground activity and environmental changes. Training on real-world videos enhanced with this innovative motion depiction approach, our model generates videos exhibiting coherent movement in both foreground subjects and their surrounding contexts. To further extend video generation to longer sequences without accumulating errors, we adopt a clip-by-clip generation strategy, introducing global features at each step. To ensure seamless continuity across these segments, we ingeniously link the final frame of a produced clip with input noise to spawn the succeeding one, maintaining narrative flow. Throughout the sequential generation process, we infuse the feature representation of the initial reference image into the network, effectively curtailing any cumulative color inconsistencies that may otherwise arise. Empirical evaluations attest to the superiority of our method in producing videos that exhibit harmonious interplay between foreground actions and responsive background dynamics, surpassing prior methodologies in this regard.

Generating Realistic Videos from Keyframes with Concatenated GANs

OAW-GAN: Occlusion-Aware Warping GAN for Unified Human Video Synthesis

Towards Smooth Video Composition

Video Content Swapping Using GAN

To Create What You Tell: Generating Videos from Captions

Pose Guided Human Video Generation

Scripted Video Generation With a Bottom-Up Generative Adversarial Network

Diverse Video Generation from a Single Video

Generating fluent tubes in video synopsis

Generative Adversarial Networks for Image and Video Synthesis: Algorithms and Applications

Towards High Resolution Video Generation with Progressive Growing of Sliced Wasserstein GANs

MotionVideoGAN: A Novel Video Generator Based on the Motion Space Learned from Image Pairs

Generating Videos with Dynamics-aware Implicit Generative Adversarial Networks

Disentangling Foreground and Background Motion for Enhanced Realism in Human Video Generation

Realistic Dynamic Facial Textures from a Single Image Using GANs

Temporally Consistent Semantic Video Editing

Realistic Speech-Driven Facial Animation with GANs

Video Generation Beyond a Single Clip

IRC-GAN: Introspective Recurrent Convolutional GAN for Text-to-video Generation.

xGen-VideoSyn-1: High-fidelity Text-to-Video Synthesis with Compressed Representations

Point-to-Point Video Generation