GaussianBody: Clothed Human Reconstruction via 3d Gaussian Splatting

Mengtian Li,Shengxiang Yao,Zhifeng Xie,Keyu Chen

2024-01-27

Abstract:In this work, we propose a novel clothed human reconstruction method called GaussianBody, based on 3D Gaussian Splatting. Compared with the costly neural radiance based models, 3D Gaussian Splatting has recently demonstrated great performance in terms of training time and rendering quality. However, applying the static 3D Gaussian Splatting model to the dynamic human reconstruction problem is non-trivial due to complicated non-rigid deformations and rich cloth details. To address these challenges, our method considers explicit pose-guided deformation to associate dynamic Gaussians across the canonical space and the observation space, introducing a physically-based prior with regularized transformations helps mitigate ambiguity between the two spaces. During the training process, we further propose a pose refinement strategy to update the pose regression for compensating the inaccurate initial estimation and a split-with-scale mechanism to enhance the density of regressed point clouds. The experiments validate that our method can achieve state-of-the-art photorealistic novel-view rendering results with high-quality details for dynamic clothed human bodies, along with explicit geometry reconstruction.

Computer Vision and Pattern Recognition

What problem does this paper attempt to address?

### Problems the Paper Attempts to Solve

The paper proposes a new method called **GaussianBody** for reconstructing dynamic dressed human models from monocular videos. Specifically, the paper aims to address the following issues:

1. **High-Fidelity Reconstruction**:
- Traditional methods rely on complex capture systems or the tedious work of 3D artists, which are time-consuming and costly, making large-scale applications difficult.
- Existing mesh-based methods (such as SMPL) can quickly reconstruct human shapes but struggle to capture complex geometric details and rich clothing features.

2. **Real-Time Rendering and Efficient Training**:
- Implicit methods (such as NeRF) can improve reconstruction fidelity and rendering quality, but the complex volumetric rendering process leads to long training times, making real-time applications difficult.
- Models based on implicit methods lack effective deformation schemes when dealing with complex body movements in dynamic sequences.

3. **Non-Rigid Deformation and Detail Capture**:
- Dynamic dressed human models exhibit complex non-rigid deformations and rich clothing details, making it challenging for existing methods to accurately capture these details.

To address the above challenges, the paper proposes the following key techniques:

- **3D Gaussian Splatting**: Utilizes 3D Gaussian point cloud representation for efficient rendering and introduces explicit pose-guided deformation to map the Gaussian point cloud from canonical space to observation space.
- **Physics-Based Priors**: Introduces physics-based priors to regularize Gaussian parameters in the observation space, avoiding overfitting issues.
- **Pose Optimization and Point Cloud Enhancement**: Proposes a pose optimization strategy to correct initial pose estimation errors and employs a hierarchical mechanism to enhance point cloud density.

Experimental results show that this method excels in detail reconstruction and geometric recovery, with short training times (about 1 hour) and near real-time rendering speed. Additionally, ablation experiments validate the effectiveness of each component.