High-Quality Animatable Dynamic Garment Reconstruction from Monocular Videos

Xiongzheng Li,Jinsong Zhang,Yu-Kun Lai,Jingyu Yang,Kun Li

2023-11-02

Abstract:Much progress has been made in reconstructing garments from an image or a video. However, none of existing works meet the expectations of digitizing high-quality animatable dynamic garments that can be adjusted to various unseen poses. In this paper, we propose the first method to recover high-quality animatable dynamic garments from monocular videos without depending on scanned data. To generate reasonable deformations for various unseen poses, we propose a learnable garment deformation network that formulates the garment reconstruction task as a pose-driven deformation problem. To alleviate the ambiguity estimating 3D garments from monocular videos, we design a multi-hypothesis deformation module that learns spatial representations of multiple plausible deformations. Experimental results on several public datasets demonstrate that our method can reconstruct high-quality dynamic garments with coherent surface details, which can be easily animated under unseen poses. The code will be provided for research purposes.

Computer Vision and Pattern Recognition

What problem does this paper attempt to address?

The problem this paper attempts to address is how to reconstruct high-quality, animatable dynamic clothing from monocular video. Existing methods, although making some progress in reconstructing clothing from images or videos, fail to meet the need for digitizing high-quality, animatable dynamic clothing, especially in adapting to various unseen poses. Additionally, many existing methods rely on expensive scanning data for training and face domain gap issues, making it difficult to generalize well to inputs outside the training dataset.

To overcome these issues, the authors propose a new weakly supervised framework aimed at recovering high-quality, animatable dynamic clothing from monocular video without relying on scanning data. This framework achieves this goal through the following points:

1. **Weakly Supervised Learning**: Reduces the dependence on large amounts of scanning data, lowering the time cost of data preparation and model deployment.
2. **Learnable Clothing Deformation Network**: Based on human priors, models the clothing reconstruction task as a pose-driven deformation problem, enabling the model to generate reasonable deformations suitable for various unseen poses.
3. **Multi-Hypothesis Displacement Module**: By learning multiple possible deformations, alleviates the depth ambiguity when estimating 3D clothing from monocular video, thereby generating more reasonable final 3D clothing.

These innovations collectively address the shortcomings of existing methods in high-quality, animatable dynamic clothing reconstruction, providing new solutions for related applications.