Pose-to-Motion: Cross-Domain Motion Retargeting with Pose Prior

Qingqing Zhao,Peizhuo Li,Wang Yifan,Olga Sorkine-Hornung,Gordon Wetzstein

2023-10-31

Abstract:Creating believable motions for various characters has long been a goal in computer graphics. Current learning-based motion synthesis methods depend on extensive motion datasets, which are often challenging, if not impossible, to obtain. On the other hand, pose data is more accessible, since static posed characters are easier to create and can even be extracted from images using recent advancements in computer vision. In this paper, we utilize this alternative data source and introduce a neural motion synthesis approach through retargeting. Our method generates plausible motions for characters that have only pose data by transferring motion from an existing motion capture dataset of another character, which can have drastically different skeletons. Our experiments show that our method effectively combines the motion features of the source character with the pose features of the target character, and performs robustly with small or noisy pose data sets, ranging from a few artist-created poses to noisy poses estimated directly from images. Additionally, a conducted user study indicated that a majority of participants found our retargeted motion to be more enjoyable to watch, more lifelike in appearance, and exhibiting fewer artifacts. Project page: <a class="link-external link-https" href="https://cyanzhao42.github.io/pose2motion" rel="external noopener nofollow">this https URL</a>

Computer Vision and Pattern Recognition,Graphics,Machine Learning

What problem does this paper attempt to address?

### The Problem This Paper Attempts to Solve

This paper aims to address the issue of generating believable actions for different characters in computer graphics. Specifically, it attempts to overcome the following challenges:

1. **Scarcity of High-Quality Motion Data**: Current learning-based methods rely on large motion datasets, but these datasets are often difficult to obtain, especially for non-humanoid creatures.
2. **Utilizing Pose Data**: Pose data is relatively easy to obtain and can be created by artists or extracted from images. The paper proposes a method to generate motions using this easily accessible data source.

### Research Contributions

1. **Pose-to-Motion Method**: A new motion retargeting method is proposed, which generates reasonable motions by transferring motion data from the source domain to pose data in the target domain.
2. **Asymmetric Cycle Consistency Learning**: By using an asymmetric cycle consistency learning framework combined with soft constraint synthesis of root transformations, the paper addresses the root ambiguity problem in pose-to-motion settings.
3. **Performance Evaluation**: Detailed experimental evaluations were conducted on multiple datasets, demonstrating superior performance across various characters and poses.

### Experimental Results

- **Mixamo Dataset**: Quantitative evaluations show that the Pose-to-Motion method outperforms baseline methods in terms of joint position jitter and contact consistency.
- **Animal Dataset**: Even with significant domain gaps (e.g., from dog to T-Rex, hamster, etc.), the Pose-to-Motion method can still generate high-quality motions.

Through these contributions, the paper demonstrates how to generate reasonable and realistic motions using pose data in the absence of high-quality motion data.