Abstract:Human motion copy is an intriguing yet challenging task in artificial intelligence and computer vision, which strives to generate a fake video of a target person performing the motion of a source person. The problem is inherently challenging due to the subtle human-body texture details to be generated and the temporal consistency to be considered. Existing approaches typically adopt a conventional GAN with an L1 or L2 loss to produce the target fake video, which intrinsically necessitates a large number of training samples that are challenging to acquire. Meanwhile, current methods still have difficulties in attaining realistic image details and temporal consistency, which unfortunately can be easily perceived by human observers. Motivated by this, we try to tackle the issues from three aspects: (1) We constrain pose-to-appearance generation with a perceptual loss and a theoretically motivated Gromov-Wasserstein loss to bridge the gap between pose and appearance. (2) We present an episodic memory module in the pose-to-appearance generation to propel continuous learning that helps the model learn from its past poor generations. We also utilize geometrical cues of the face to optimize facial details and refine each key body part with a dedicated local GAN. (3) We advocate generating the foreground in a sequence-to-sequence manner rather than a single-frame manner, explicitly enforcing temporal inconsistency. Empirical results on five datasets, iPER, ComplexMotion, SoloDance, Fish, and Mouse datasets, demonstrate that our method is capable of generating realistic target videos while precisely copying motion from a source video. Our method significantly outperforms state-of-the-art approaches and gains 7.2% and 12.4% improvements in PSNR and FID respectively.

Articulated Human Motion Tracking Using Sequential Immune Genetic Algorithm

Generative Tracking of Human Motion by Sequential Clonal Selection Algorithm

Generative tracking of 3D human motion in latent space by sequential clonal selection algorithm

An improved clonal selection algorithm for articulated human motion tracking

Articulated Human Motion Tracking by Sequential Annealed Particle Swarm Optimization.

Vision-Based Hand Motion Capture Using Genetic Algorithm

Human motion tracking by temporal-spatial local gaussian process experts

Human motion tracking based on shape analysis

Gesture Recognition Based on Adaptive Genetic Algorithm

An Anthropomorphic Motion Retargeting Method Based on an Adaptive Genetic Algorithm

MimicMotion: High-Quality Human Motion Video Generation with Confidence-aware Pose Guidance

Tracking Generic Human Motion Via Fusion of Low- and High-Dimensional Approaches

Motion Mamba: Efficient and Long Sequence Motion Generation

Human Motion Transfer from Poses in the Wild

Simultaneous 3-D Human-Motion Tracking and Voxel Reconstruction

A Top-down Approach to Articulated Human Pose Estimation and Tracking

Learning a Tracking and Estimation Integrated Graphical Model for Human Pose Tracking.

Reconstruction of 3D Human Motion Using Multiple View L1 Trackers

A Human Motion Feature Based on Semi-Supervised Learning of GMM

Do As I Do: Pose Guided Human Motion Copy

Coupling Top-down and Bottom-up Methods for 3D Human Pose and Shape Estimation from Monocular Image Sequences