ClotheDreamer: Text-Guided Garment Generation with 3D Gaussians

Yufei Liu,Junshu Tang,Chu Zheng,Shijie Zhang,Jinkun Hao,Junwei Zhu,Dongjin Huang
2024-06-25
Abstract:High-fidelity 3D garment synthesis from text is desirable yet challenging for digital avatar creation. Recent diffusion-based approaches via Score Distillation Sampling (SDS) have enabled new possibilities but either intricately couple with human body or struggle to reuse. We introduce ClotheDreamer, a 3D Gaussian-based method for generating wearable, production-ready 3D garment assets from text prompts. We propose a novel representation Disentangled Clothe Gaussian Splatting (DCGS) to enable separate optimization. DCGS represents clothed avatar as one Gaussian model but freezes body Gaussian splats. To enhance quality and completeness, we incorporate bidirectional SDS to supervise clothed avatar and garment RGBD renderings respectively with pose conditions and propose a new pruning strategy for loose clothing. Our approach can also support custom clothing templates as input. Benefiting from our design, the synthetic 3D garment can be easily applied to virtual try-on and support physically accurate animation. Extensive experiments showcase our method's superior and competitive performance. Our project page is at <a class="link-external link-https" href="https://ggxxii.github.io/clothedreamer" rel="external noopener nofollow">this https URL</a>.
Computer Vision and Pattern Recognition
What problem does this paper attempt to address?
The paper aims to address the issue of high-fidelity 3D clothing generation in the creation of digital virtual characters. Specifically, the paper proposes a new method called ClotheDreamer, which is based on a 3D Gaussian model and can generate wearable and production-ready 3D clothing assets from text prompts. ### Main Problems the Paper Attempts to Solve: 1. **High Fidelity and Controllability**: Traditional methods often require a lot of manual design to generate high-quality 3D clothing, and the types of clothing generated are limited by the mesh models in the training dataset. ClotheDreamer improves the realism and controllability of the generated results by decoupling the human body and clothing parts. 2. **Reusability and Editability**: Existing methods generate clothing that is usually tightly integrated with the human body, making it difficult to further edit or replace. ClotheDreamer separates the human body and clothing, allowing the generated clothing to be easily applied to virtual try-ons and supporting physically accurate animation simulations. 3. **Efficient Rendering and Detail Capture**: Existing 3D generation methods have trade-offs between rendering quality and efficiency. ClotheDreamer combines the advantages of the 3D Gaussian model to achieve efficient rendering and complex appearance detail capture. 4. **Integrity of Loose Clothing**: For loose clothing (such as long skirts and gowns), existing methods are prone to detail loss or incompleteness. ClotheDreamer proposes a new pruning strategy to maintain the integrity of loose clothing. ### Main Contributions: 1. Proposed the Disentangled Clothe Gaussian Splatting (DCGS) representation, which optimizes the separation of clothing and the human body through ID initialization. 2. Introduced a bidirectional Score Distillation Sampling (SDS) guidance mechanism, supervising the RGBD rendering of clothing and the human body separately, and proposed a new pruning strategy to enhance the integrity of loose clothing. 3. Supports template-guided customized clothing generation and can accurately animate the generated clothing. With these improvements, ClotheDreamer can generate diverse, high-quality 3D clothing assets based on text descriptions, which are easy to edit and apply.