Abstract:Creating pose-driven human avatars is about modeling the mapping from the low-frequency driving pose to high-frequency dynamic human appearances, so an effective pose encoding method that can encode high-fidelity human details is essential to human avatar modeling. To this end, we present PoseVocab, a novel pose encoding method that encourages the network to discover the optimal pose embeddings for learning the dynamic human appearance. Given multi-view RGB videos of a character, PoseVocab constructs key poses and latent embeddings based on the training poses. To achieve pose generalization and temporal consistency, we sample key rotations in $so(3)$ of each joint rather than the global pose vectors, and assign a pose embedding to each sampled key rotation. These joint-structured pose embeddings not only encode the dynamic appearances under different key poses, but also factorize the global pose embedding into joint-structured ones to better learn the appearance variation related to the motion of each joint. To improve the representation ability of the pose embedding while maintaining memory efficiency, we introduce feature lines, a compact yet effective 3D representation, to model more fine-grained details of human appearances. Furthermore, given a query pose and a spatial position, a hierarchical query strategy is introduced to interpolate pose embeddings and acquire the conditional pose feature for dynamic human synthesis. Overall, PoseVocab effectively encodes the dynamic details of human appearance and enables realistic and generalized animation under novel poses. Experiments show that our method outperforms other state-of-the-art baselines both qualitatively and quantitatively in terms of synthesis quality. Code is available at <a class="link-external link-https" href="https://github.com/lizhe00/PoseVocab" rel="external noopener nofollow">this https URL</a>.

Pose Embeddings: A Deep Architecture for Learning to Match Human Poses

View-Invariant, Occlusion-Robust Probabilistic Embedding for Human Pose

Modelling Human Body Pose for Action Recognition Using Deep Neural Networks

PoseVocab: Learning Joint-structured Pose Embeddings for Human Avatar Modeling

Human Pose Estimation from Depth Images via Inference Embedded Multi-task Learning

PoseEmbroider: Towards a 3D, Visual, Semantic-aware Human Pose Representation

Adversarial learning for viewpoints invariant 3D human pose estimation.

RePose: Learning Deep Kinematic Priors for Fast Human Pose Estimation

DeepIM: Deep Iterative Matching for 6D Pose Estimation

3D Human Pose Estimation = 2D Pose Estimation + Matching

Human Pose Estimation Using Deep Structure Guided Learning.

Poseur: Direct Human Pose Regression with Transformers.

Imitation Learning for Human Pose Prediction

Deep Kinematic Pose Regression.

AnatPose: Bidirectionally learning anatomy-aware heatmaps for human pose estimation

Back to the Future: Joint Aware Temporal Deep Learning 3D Human Pose Estimation

Deep3DPose: Realtime Reconstruction of Arbitrarily Posed Human Bodies from Single RGB Images

Learning Pose Controllable Human Reconstruction with Dynamic Implicit Fields from a Single Image

3D Human Pose Estimation from Deep Multi-View 2D Pose

3D Human Pose Estimation with Siamese Equivariant Embedding

GS-Pose: Category-Level Object Pose Estimation via Geometric and Semantic Correspondence