Gaussian Garments: Reconstructing Simulation-Ready Clothing with Photorealistic Appearance from Multi-View Video

Boxiang Rong,Artur Grigorev,Wenbo Wang,Michael J. Black,Bernhard Thomaszewski,Christina Tsalicoglou,Otmar Hilliges
2024-09-13
Abstract:We introduce Gaussian Garments, a novel approach for reconstructing realistic simulation-ready garment assets from multi-view videos. Our method represents garments with a combination of a 3D mesh and a Gaussian texture that encodes both the color and high-frequency surface details. This representation enables accurate registration of garment geometries to multi-view videos and helps disentangle albedo textures from lighting effects. Furthermore, we demonstrate how a pre-trained graph neural network (GNN) can be fine-tuned to replicate the real behavior of each garment. The reconstructed Gaussian Garments can be automatically combined into multi-garment outfits and animated with the fine-tuned GNN.
Computer Vision and Pattern Recognition,Graphics
What problem does this paper attempt to address?
The problem that this paper attempts to solve is how to reconstruct clothing assets with realistic appearance and simulation - ready characteristics from multi - view videos. Specifically, the paper focuses on three key aspects: 1. **Reconstruction of 3D geometric structure**: Accurately capture the overall structure and detailed details of clothing. 2. **Appearance reproduction**: Precisely reflect the color and texture of clothing. 3. **Behavior simulation**: Realistically imitate the dynamic performance of clothing under different actions. To achieve these goals, the authors introduced a new method called "Gaussian Garments", which combines 3D meshes and 3D Gaussian point lattices (Gaussian splatting) to represent clothing. This method can not only accurately represent the geometric shape of clothing, but also capture its high - frequency details, such as plush materials. In addition, the reconstructed clothing can be re - positioned on a new body model, resized to fit different body types, and simulated by a neural network, thereby producing realistic animation effects. ### Main contributions - **Comprehensive reconstruction process**: Reconstruct the shape, appearance, and behavior of clothing from multi - view videos. - **Registration algorithm based on Gaussian point lattices**: Align the clothing mesh with multi - view videos. - **Gaussian Garment representation**: Combine triangular meshes and Gaussian textures to capture a realistic appearance and serve as a fully controllable 3D asset. Through these innovations, Gaussian Garments provide high - quality digital clothing assets for applications such as virtual fitting systems, film production, and video games.