Abstract:Recently, a range of neural network-based methods for image rendering have been introduced. One such widely-researched neural radiance field (NeRF) relies on a neural network to represent 3D scenes, allowing for realistic view synthesis from a small number of 2D images. However, most NeRF models are constrained by long training and inference times. In comparison, Gaussian Splatting (GS) is a novel, state-of-the-art technique for rendering points in a 3D scene by approximating their contribution to image pixels through Gaussian distributions, warranting fast training and swift, real-time rendering. A drawback of GS is the absence of a well-defined approach for its conditioning due to the necessity to condition several hundred thousand Gaussian components. To solve this, we introduce the Gaussian Mesh Splatting (GaMeS) model, which allows modification of Gaussian components in a similar way as meshes. We parameterize each Gaussian component by the vertices of the mesh face. Furthermore, our model needs mesh initialization on input or estimated mesh during training. We also define Gaussian splats solely based on their location on the mesh, allowing for automatic adjustments in position, scale, and rotation during animation. As a result, we obtain a real-time rendering of editable GS.
What problem does this paper attempt to address?
The main problem this paper attempts to address is the challenges present in existing Gaussian Splatting (GS) methods when modifying and adjusting Gaussian components. Specifically, while GS methods are fast to train and capable of real-time rendering, they lack effective conditional methods for adjusting the position, scale, and rotation of Gaussian components, especially when dealing with tens of thousands of Gaussian components. Additionally, existing methods like SuGaR, although able to solve some issues, require an expensive preprocessing stage.
To address these problems, the paper introduces the Gaussian MeshSplatting (GaMeS) model. GaMeS parameterizes Gaussian components as vertices of mesh faces, allowing Gaussian components to be modified like a mesh. This approach not only enables real-time rendering but also allows automatic adjustment of the position, scale, and rotation of Gaussian components during animation, achieving editable Gaussian splatting.
### Main Contributions:
1. **Hybrid Representation**: A hybrid representation method that seamlessly combines mesh and Gaussian splatting is proposed.
2. **Pseudo Mesh**: By using a simple pseudo mesh, Gaussian components can be directly estimated from a single training session without the need for expensive preprocessing steps.
3. **Efficient Rendering**: Relying only on basic vector operations, it can render dynamic scenes at speeds similar to static scenes.
### Experimental Validation:
- **Existing Mesh Scenario**: Using provided meshes, Gaussian components are placed on their surfaces, allowing vertex movement. Experimental results show that GaMeS performs excellently in reconstruction and simple object animation.
- **No Mesh Scenario**: When there is no mesh, GS is first trained with flat Gaussians, then a pseudo mesh is constructed. Experimental results show that GaMeS is also effective in handling unbounded scenes.
- **FLAME Initialized Mesh**: Using the FLAME framework to generate controllable face meshes, GaMeS demonstrates advantages in face fitting and expression generation.
### Summary:
The GaMeS model addresses the challenges of modifying and adjusting Gaussian components in existing Gaussian splatting methods by combining Gaussian components with meshes, achieving efficient real-time rendering and editability. This method performs excellently in various scenarios, especially when dealing with complex and dynamic scenes.