LayGA: Layered Gaussian Avatars for Animatable Clothing Transfer

Siyou Lin,Zhe Li,Zhaoqi Su,Zerong Zheng,Hongwen Zhang,Yebin Liu

DOI: https://doi.org/10.48550/arXiv.2405.07319

2024-05-13

Abstract:Animatable clothing transfer, aiming at dressing and animating garments across characters, is a challenging problem. Most human avatar works entangle the representations of the human body and clothing together, which leads to difficulties for virtual try-on across identities. What's worse, the entangled representations usually fail to exactly track the sliding motion of garments. To overcome these limitations, we present Layered Gaussian Avatars (LayGA), a new representation that formulates body and clothing as two separate layers for photorealistic animatable clothing transfer from multi-view videos. Our representation is built upon the Gaussian map-based avatar for its excellent representation power of garment details. However, the Gaussian map produces unstructured 3D Gaussians distributed around the actual surface. The absence of a smooth explicit surface raises challenges in accurate garment tracking and collision handling between body and garments. Therefore, we propose two-stage training involving single-layer reconstruction and multi-layer fitting. In the single-layer reconstruction stage, we propose a series of geometric constraints to reconstruct smooth surfaces and simultaneously obtain the segmentation between body and clothing. Next, in the multi-layer fitting stage, we train two separate models to represent body and clothing and utilize the reconstructed clothing geometries as 3D supervision for more accurate garment tracking. Furthermore, we propose geometry and rendering layers for both high-quality geometric reconstruction and high-fidelity rendering. Overall, the proposed LayGA realizes photorealistic animations and virtual try-on, and outperforms other baseline methods. Our project page is <a class="link-external link-https" href="https://jsnln.github.io/layga/index.html" rel="external noopener nofollow">this https URL</a>.

Computer Vision and Pattern Recognition

What problem does this paper attempt to address?

The problem that this paper attempts to solve is the challenge of virtual try - on and clothing animation transfer between different characters. Specifically, most existing human avatar works entangle the representations of the human body and clothing, which leads to difficulties in cross - identity virtual try - on, and such entangled representations usually cannot accurately track the sliding movement of clothing. To overcome these limitations, the paper proposes Layered Gaussian Avatars (LayGA), a new representation method that treats the body and clothing as two independent layers to achieve photorealistic animatable clothing transfer.

### Main Contributions:
1. **Proposing Layered Gaussian Avatars (LayGA)**: This is the first layered human avatar representation based on 3D Gaussian Splatting (3DGS) for animatable clothing transfer.
2. **Geometric Constraints**: Geometric constraints are introduced to reconstruct smooth surfaces and support collision handling between the body and clothing in the layered representation.
3. **Multi - layer Learning**: In multi - layer learning, the previously segmented reconstruction is used as supervision to track clothing boundaries more accurately. In addition, a rendering layer is introduced to alleviate the problem of degraded rendering quality caused by geometric constraints.

### Method Overview:
- **Single - layer Reconstruction Phase**: By introducing a series of geometric constraints, the 3D Gaussian distribution is forced onto the actual geometric surface, while obtaining the segmentation of the body and clothing.
- **Multi - layer Fitting Phase**: Two independent models are trained to represent the body and clothing respectively. The previously reconstructed clothing geometry is used as 3D supervision to track clothing movement more accurately.
- **Separating the Geometric Layer and the Rendering Layer**: To maintain the smoothness of the geometric layer and avoid the negative impact of geometric constraints on the rendering quality, two offsets are introduced for the geometric layer and the rendering layer respectively.
- **Geometric Supervision**: The movement of clothing is directly supervised through the Chamfer distance loss.
- **Segmentation Loss**: The segmentation loss is maintained to avoid the situation where clothing is optimized to be transparent.

### Application Scenarios:
- **Virtual Try - on**: The clothing of one character can be transferred to another character, achieving cross - identity clothing transfer.
- **Clothing Animation**: Generate realistic animations in new poses while handling collision problems.

### Summary:
LayGA solves the limitations of existing methods in virtual try - on and clothing animation through layered modeling and geometric constraints, achieving more accurate clothing transfer and high - quality rendering effects.