OccGaussian: 3D Gaussian Splatting for Occluded Human Rendering

Jingrui Ye,Zongkai Zhang,Yujiao Jiang,Qingmin Liao,Wenming Yang,Zongqing Lu

2024-04-15

Abstract:Rendering dynamic 3D human from monocular videos is crucial for various applications such as virtual reality and digital entertainment. Most methods assume the people is in an unobstructed scene, while various objects may cause the occlusion of body parts in real-life scenarios. Previous method utilizing NeRF for surface rendering to recover the occluded areas, but it requiring more than one day to train and several seconds to render, failing to meet the requirements of real-time interactive applications. To address these issues, we propose OccGaussian based on 3D Gaussian Splatting, which can be trained within 6 minutes and produces high-quality human renderings up to 160 FPS with occluded input. OccGaussian initializes 3D Gaussian distributions in the canonical space, and we perform occlusion feature query at occluded regions, the aggregated pixel-align feature is extracted to compensate for the missing information. Then we use Gaussian Feature MLP to further process the feature along with the occlusion-aware loss functions to better perceive the occluded area. Extensive experiments both in simulated and real-world occlusions, demonstrate that our method achieves comparable or even superior performance compared to the state-of-the-art method. And we improving training and inference speeds by 250x and 800x, respectively. Our code will be available for research purposes.

Computer Vision and Pattern Recognition

What problem does this paper attempt to address?

The paper aims to address the problem of rendering occluded human bodies from monocular video. Specifically, existing methods (such as NeRF-based methods) suffer from significant performance degradation when dealing with occluded areas, and their training and rendering speeds are slow, failing to meet the needs of real-time interactive applications. To solve these issues, the paper proposes the OccGaussian method, which is based on 3D Gaussian Splatting, allowing training to be completed within minutes and high-quality human rendering at speeds of up to 160 FPS.

The main contributions of the paper are as follows:
1. Proposes OccGaussian, which for the first time applies 3D Gaussian Splatting to human rendering in occluded scenes, achieving fast training (6 to 13 minutes) and real-time rendering (up to 169 FPS).
2. Introduces K-nearest neighbor feature querying in occluded areas, combines aggregated pixel-aligned features, and designs occlusion loss and consistency loss to better handle textures in occluded regions.
3. Experimental results show that OccGaussian outperforms the state-of-the-art method OccNeRF on two datasets, significantly improving training and rendering speeds while maintaining or enhancing rendering quality.