Superpoint Gaussian Splatting for Real-Time High-Fidelity Dynamic Scene Reconstruction

Diwen Wan,Ruijie Lu,Gang Zeng

2024-06-06

Abstract:Rendering novel view images in dynamic scenes is a crucial yet challenging task. Current methods mainly utilize NeRF-based methods to represent the static scene and an additional time-variant MLP to model scene deformations, resulting in relatively low rendering quality as well as slow inference speed. To tackle these challenges, we propose a novel framework named Superpoint Gaussian Splatting (SP-GS). Specifically, our framework first employs explicit 3D Gaussians to reconstruct the scene and then clusters Gaussians with similar properties (e.g., rotation, translation, and location) into superpoints. Empowered by these superpoints, our method manages to extend 3D Gaussian splatting to dynamic scenes with only a slight increase in computational expense. Apart from achieving state-of-the-art visual quality and real-time rendering under high resolutions, the superpoint representation provides a stronger manipulation capability. Extensive experiments demonstrate the practicality and effectiveness of our approach on both synthetic and real-world datasets. Please see our project page at <a class="link-external link-https" href="https://dnvtmf.github.io/SP_GS.github.io" rel="external noopener nofollow">this https URL</a>.

Computer Vision and Pattern Recognition

What problem does this paper attempt to address?

### What problem does this paper attempt to solve?

This paper aims to address the key challenge of high-fidelity novel view image rendering in dynamic scenes. Specifically:

1. **Problems with existing methods**:
- Current methods primarily utilize NeRF-based approaches to represent static scenes and model scene deformations through additional time-variant multi-layer perceptrons (MLPs), which result in lower rendering quality and slower inference speed.

2. **Proposed new framework**:
- The paper proposes a new framework called "Superpoint Gaussian Splatting (SP-GS)" that reconstructs scenes through explicit 3D Gaussian distributions and clusters Gaussians with similar attributes (such as rotation, translation, and position) into superpoints. This method extends 3D Gaussian distributions to dynamic scenes with only a slight increase in computational overhead.

3. **Goals**:
- Achieve real-time high-fidelity dynamic scene reconstruction while maintaining efficient and high-quality rendering performance. Additionally, the superpoint representation offers stronger operational capabilities, facilitating downstream applications such as scene editing.

Through the above methods, the paper demonstrates the effectiveness and practicality of its approach on both synthetic and real datasets.