Abstract:We aim to tackle the interesting yet challenging problem of generating videos of diverse and natural human motions from prescribed action categories. The key issue lies in the ability to synthesize multiple distinct motion sequences that are realistic in their visual appearances. It is achieved in this paper by a two-step process that maintains internal 3D pose and shape representations, action2motion and motion2video. Action2motion stochastically generates plausible 3D pose sequences of a prescribed action category, which are processed and rendered by motion2video to form 2D videos. Specifically, the Lie algebraic theory is engaged in representing natural human motions following the physical law of human kinematics; a temporal variational auto-encoder is developed that encourages diversity of output motions. Moreover, given an additional input image of a clothed human character, an entire pipeline is proposed to extract his/her 3D detailed shape, and to render in videos the plausible motions from different views. This is realized by improving existing methods to extract 3D human shapes and textures from single 2D images, rigging, animating, and rendering to form 2D videos of human motions. It also necessitates the curation and reannotation of 3D human motion datasets for training purpose. Thorough empirical experiments including ablation study, qualitative and quantitative evaluations manifest the applicability of our approach, and demonstrate its competitiveness in addressing related tasks, where components of our approach are compared favorably to the state-of-the-arts.

Action2Motion: Conditioned Generation of 3D Human Motions

Action-Conditioned 3D Human Motion Synthesis with Transformer VAE

Action2video: Generating Videos of Human 3D Actions

Move-in-2D: 2D-Conditioned Human Motion Generation

A Unified 3D Human Motion Synthesis Model Via Conditional Variational Auto-Encoder

Action-guided 3D Human Motion Prediction.

ActFormer: A GAN-based Transformer Towards General Action-Conditioned 3D Human Motion Generation

Human motion segmentation using collaborative representations of 3D skeletal sequences.

HUMANISE: Language-conditioned Human Motion Generation in 3D Scenes

Human Action Recognition Based on Kinematic Similarity in Real Time.

3D Human motion anticipation and classification

Human Motion Generation: A Survey

Action Recognition Based on Motion Representing and Reconstructed Phase Spaces Matching of 3D Joint Positions

Human Motion Generation Via Cross-Space Constrained Sampling.

A Human-Like Action Learning Process: Progressive Pose Generation for Motion Prediction

Human 3D Model-based 2D Action Recognition

Human Action Recognition Using Key Poses and Atomic Motions

3d Human Action Recognition Based On The Spatial-Temporal Moving Skeleton Descriptor

Conditional Temporal Variational AutoEncoder for Action Video Prediction