Abstract:Human motion copy is an intriguing yet challenging task in artificial intelligence and computer vision, which strives to generate a fake video of a target person performing the motion of a source person. The problem is inherently challenging due to the subtle human-body texture details to be generated and the temporal consistency to be considered. Existing approaches typically adopt a conventional GAN with an L1 or L2 loss to produce the target fake video, which intrinsically necessitates a large number of training samples that are challenging to acquire. Meanwhile, current methods still have difficulties in attaining realistic image details and temporal consistency, which unfortunately can be easily perceived by human observers. Motivated by this, we try to tackle the issues from three aspects: (1) We constrain pose-to-appearance generation with a perceptual loss and a theoretically motivated Gromov-Wasserstein loss to bridge the gap between pose and appearance. (2) We present an episodic memory module in the pose-to-appearance generation to propel continuous learning that helps the model learn from its past poor generations. We also utilize geometrical cues of the face to optimize facial details and refine each key body part with a dedicated local GAN. (3) We advocate generating the foreground in a sequence-to-sequence manner rather than a single-frame manner, explicitly enforcing temporal inconsistency. Empirical results on five datasets, iPER, ComplexMotion, SoloDance, Fish, and Mouse datasets, demonstrate that our method is capable of generating realistic target videos while precisely copying motion from a source video. Our method significantly outperforms state-of-the-art approaches and gains 7.2% and 12.4% improvements in PSNR and FID respectively.

Skeleton-Aided Articulated Motion Generation

Generative Model for Skeletal Human Movements Based on Conditional DC-GAN Applied to Pseudo-Images

3D Articulated Skeleton Extraction Using a Single Consumer-Grade Depth Camera.

Sequential Articulated Motion Reconstruction from a Monocular Image Sequence

Unsupervised Articulated Skeleton Extraction from Point Set Sequences Captured by a Single Depth Camera

3D Skeleton-based Human Motion Prediction with Manifold-Aware GAN

GANimator: Neural Motion Synthesis from a Single Sequence

Human Motion Generation Via Cross-Space Constrained Sampling.

Generative Tweening: Long-term Inbetweening of 3D Human Motions

GAN-Poser: an Improvised Bidirectional GAN Model for Human Motion Prediction

Multi-person/Group Interactive Video Generation

Pose Guided Human Video Generation

Human Action Transfer Based on 3D Model Reconstruction.

Do As I Do: Pose Guided Human Motion Copy

3D Information Guided Motion Transfer via Sequential Image based Human Model Refinement and Face-Attention GAN

MimicMotion: High-Quality Human Motion Video Generation with Confidence-aware Pose Guidance

Animated 3D Human Avatars from a Single Image with GAN-based Texture Inference.

Multi-person/Group Interactive Video Generation

Ponymation: Learning Articulated 3D Animal Motions from Unlabeled Online Videos

Action2Motion: Conditioned Generation of 3D Human Motions