Garment Animation NeRF with Color Editing

Renke Wang,Meng Zhang,Jun Li,Jian Yan

2024-07-29

Abstract:Generating high-fidelity garment animations through traditional workflows, from modeling to rendering, is both tedious and expensive. These workflows often require repetitive steps in response to updates in character motion, rendering viewpoint changes, or appearance edits. Although recent neural rendering offers an efficient solution for computationally intensive processes, it struggles with rendering complex garment animations containing fine wrinkle details and realistic garment-and-body occlusions, while maintaining structural consistency across frames and dense view rendering. In this paper, we propose a novel approach to directly synthesize garment animations from body motion sequences without the need for an explicit garment proxy. Our approach infers garment dynamic features from body motion, providing a preliminary overview of garment structure. Simultaneously, we capture detailed features from synthesized reference images of the garment's front and back, generated by a pre-trained image model. These features are then used to construct a neural radiance field that renders the garment animation video. Additionally, our technique enables garment recoloring by decomposing its visual elements. We demonstrate the generalizability of our method across unseen body motions and camera views, ensuring detailed structural consistency. Furthermore, we showcase its applicability to color editing on both real and synthetic garment data. Compared to existing neural rendering techniques, our method exhibits qualitative and quantitative improvements in garment dynamics and wrinkle detail modeling. Code is available at \url{<a class="link-external link-https" href="https://github.com/wrk226/GarmentAnimationNeRF" rel="external noopener nofollow">this https URL</a>}.

Computer Vision and Pattern Recognition

What problem does this paper attempt to address?

The paper proposes a novel method aimed at synthesizing high-quality, detail-rich clothing animations directly from human motion sequences without the need for explicit clothing proxies. Specifically, the paper addresses the following key issues:

1. **Simplifying Traditional Workflows**: Traditional clothing animation generation processes involve multiple steps such as modeling, simulation, and rendering. These steps are both cumbersome and expensive, requiring repetitive work when updating character motions, rendering viewpoint changes, or editing appearances.

2. **Efficient Neural Rendering**: Although recent neural rendering techniques can effectively handle computationally intensive tasks, they struggle with rendering complex clothing animations that include fine wrinkle details and realistic body-clothing occlusion effects. Additionally, maintaining structural consistency across different views and frames is challenging.

3. **Direct Synthesis of Clothing Animation**: The proposed method can directly synthesize clothing animations from human motion sequences without the need for predefined or explicitly reconstructed clothing proxies. This approach can infer clothing dynamic features influenced by human motion and ensure detailed structural consistency across different views and frames.

4. **Generalization Capability**: The proposed method demonstrates good generalization capability on unseen human motions and camera views, ensuring the presentation of complex wrinkle details and structural consistency.

5. **Color Editing Functionality**: In addition to synthesizing clothing animations, the method also supports artifact-free color editing of the target clothing appearance.

Through the aforementioned methods, the research in the paper significantly improves the modeling quality of clothing dynamics and wrinkle details, showcasing quantitative and qualitative improvements over existing neural rendering techniques. Furthermore, the paper provides a detailed description of the specific implementation and technical details of the method.