Qianli Ma,Jinlong Yang,Michael J. Black,Siyu Tang
Abstract:Parametric 3D body models like SMPL only represent minimally-clothed people and are hard to extend to clothing because they have a fixed mesh topology and resolution. To address these limitations, recent work uses implicit surfaces or point clouds to model clothed bodies. While not limited by topology, such methods still struggle to model clothing that deviates significantly from the body, such as skirts and dresses. This is because they rely on the body to canonicalize the clothed surface by reposing it to a reference shape. Unfortunately, this process is poorly defined when clothing is far from the body. Additionally, they use linear blend skinning to pose the body and the skinning weights are tied to the underlying body parts. In contrast, we model the clothing deformation in a local coordinate space without canonicalization. We also relax the skinning weights to let multiple body parts influence the surface. Specifically, we extend point-based methods with a coarse stage, that replaces canonicalization with a learned pose-independent "coarse shape" that can capture the rough surface geometry of clothing like skirts. We then refine this using a network that infers the linear blend skinning weights and pose dependent displacements from the coarse representation. The approach works well for garments that both conform to, and deviate from, the body. We demonstrate the usefulness of our approach by learning person-specific avatars from examples and then show how they can be animated in new poses and motions. We also show that the method can learn directly from raw scans with missing data, greatly simplifying the process of creating realistic avatars. Code is available for research purposes at {\small\url{<a class="link-external link-https" href="https://qianlim.github.io/SkiRT" rel="external noopener nofollow">this https URL</a>}}.
What problem does this paper attempt to address?
The problem that this paper attempts to solve is to generate high - quality 3D human body models on challenging clothing types such as skirts and dresses. Traditional methods have significant problems when dealing with such clothing, for example, generating unnatural structures or having uneven point densities and crack - like artifacts in the predicted shapes. The paper proposes a point - based method to solve these problems by predicting body - related Linear Blend Skinning (LBS) weights and using a novel coarse - shape representation. Specifically, the paper introduces a new method named SkiRT, which can achieve state - of - the - art modeling accuracy on different types of clothing, especially in pose - dependent shape - modeling tasks.
### Key Innovation Points
1. **Coarse - Shape Representation**: SkiRT introduces a learned, pose - independent "coarse shape" that can capture the approximate surface geometry of the clothing. This coarse shape is similar to the clothing template in the traditional pipeline, but does not require a specific mesh topology, so it can flexibly represent various clothing types.
2. **Local Coordinate Space**: SkiRT models clothing deformation in the local coordinate space instead of relying on the normalization process of re - positioning the clothing surface to the reference shape. This enables the method to better handle clothing types that are far from the body, such as skirts and dresses.
3. **Predicting LBS Weights**: SkiRT predicts the LBS weights for each point through a neural network. These weights allow multiple body parts to influence the surface, thereby reducing the problem of surface discontinuities.
4. **Adaptive Regularization Technique**: In order to train the entire pipeline, SkiRT introduces an adaptive regularization technique that encourages the generation of more evenly distributed point clouds, further improving the model's accuracy.
### Method Overview
The training process of SkiRT is divided into two stages:
1. **Coarse - Shape Network Training**: First, train a coarse - shape network that predicts a pose - independent clothing shape according to the SMPL model. This coarse shape provides the basis for the subsequent fine - tuning stage and is used to obtain the pre - diffused LBS weights.
2. **Fine - Shape Network Training**: Then train a fine - shape network that predicts pose - dependent detailed clothing offsets and also predicts the transformation form of local coordinates (i.e., LBS weights). These offsets are added to the coarse shape and pose - adjusted using the predicted transformation, and finally the complete clothing - human prediction is generated.
### Experimental Evaluation
The paper conducted extensive experiments on the ReSynth dataset, which contains rich geometric details and significant pose - dependent clothing deformations. Special attention was paid to subjects wearing skirts and dresses of different styles, lengths, and tightness levels, and non - skirt clothing types were also evaluated. The experimental results show that SkiRT outperforms the existing state - of - the - art methods when dealing with challenging clothing types, especially in pose - dependent clothing - modeling tasks.
### Conclusion
By introducing the coarse - shape representation, local - coordinate - space modeling, predicting LBS weights, and adaptive regularization technique, SkiRT has successfully solved the shortcomings of existing methods in dealing with complex clothing types such as skirts and dresses, and has achieved higher - quality 3D human - body - model generation.