Mesh-based Gaussian Splatting for Real-time Large-scale Deformation

Lin Gao,Jie Yang,Bo-Tao Zhang,Jia-Mu Sun,Yu-Jie Yuan,Hongbo Fu,Yu-Kun Lai
2024-02-07
Abstract:Neural implicit representations, including Neural Distance Fields and Neural Radiance Fields, have demonstrated significant capabilities for reconstructing surfaces with complicated geometry and topology, and generating novel views of a scene. Nevertheless, it is challenging for users to directly deform or manipulate these implicit representations with large deformations in the real-time fashion. Gaussian Splatting(GS) has recently become a promising method with explicit geometry for representing static scenes and facilitating high-quality and real-time synthesis of novel views. However,it cannot be easily deformed due to the use of discrete Gaussians and lack of explicit topology. To address this, we develop a novel GS-based method that enables interactive deformation. Our key idea is to design an innovative mesh-based GS representation, which is integrated into Gaussian learning and manipulation. 3D Gaussians are defined over an explicit mesh, and they are bound with each other: the rendering of 3D Gaussians guides the mesh face split for adaptive refinement, and the mesh face split directs the splitting of 3D Gaussians. Moreover, the explicit mesh constraints help regularize the Gaussian distribution, suppressing poor-quality Gaussians(e.g. misaligned Gaussians,long-narrow shaped Gaussians), thus enhancing visual quality and avoiding artifacts during deformation. Based on this representation, we further introduce a large-scale Gaussian deformation technique to enable deformable GS, which alters the parameters of 3D Gaussians according to the manipulation of the associated mesh. Our method benefits from existing mesh deformation datasets for more realistic data-driven Gaussian deformation. Extensive experiments show that our approach achieves high-quality reconstruction and effective deformation, while maintaining the promising rendering results at a high frame rate(65 FPS on average).
Computer Vision and Pattern Recognition
What problem does this paper attempt to address?
The problem that this paper attempts to solve is to achieve large - scale real - time deformation based on 3D Gaussian Splatting (3DGS) while maintaining high - quality rendering. Specifically, although traditional 3DGS methods can achieve efficient and high - quality real - time rendering, they have difficulties in large - scale deformation because they are based on discrete Gaussian kernels and lack explicit topological information, resulting in visual artifacts that are prone to occur during the deformation process. To solve these problems, the paper proposes a Mesh - based Gaussian Splatting representation method. By binding 3D Gaussian kernels to explicit meshes and using the topological structure of the meshes to guide the splitting and deformation of Gaussian kernels, high - quality, large - scale real - time deformation is achieved. ### Main contributions of the paper: 1. **Innovative Mesh - based Gaussian Splatting representation**: Bind 3D Gaussian kernels to explicit meshes and use the topological information of the meshes to guide the learning and splitting of Gaussian kernels, improving the quality of 3DGS. 2. **Large - scale Gaussian deformation technology**: Introduce a large - scale Gaussian deformation method, which not only uses vertex positions but also uses deformation gradients to guide the deformation of Gaussian kernels, combining the advantages of mesh deformation methods while maintaining real - time rendering and high - quality appearance. 3. **Extensive experimental verification**: Through a large number of experiments on public datasets and self - collected scenes, it is proved that this method is superior to existing methods in terms of efficiency and quality. ### Formula display: - **Gaussian function**: \[ g(x)=e^{-\frac{1}{2}(x - \mu)^T\Sigma^{-1}(x - \mu)} \] where \(\mu\in\mathbb{R}^3\) is the central position and \(\Sigma\in\mathbb{R}^{3\times3}\) is the covariance matrix, which can be decomposed into a rotation matrix \(R\) and a scaling matrix \(S\), that is, \(\Sigma = RSS^TR^T\). - **Regularization loss**: \[ L_r=\sum_{g\in G}\max(\max(s_i)-\gamma R_i, 0) \] where \(s_i\) is the 3D scaling vector of each Gaussian kernel, \(R_i\) is the circumradius of the triangle where the Gaussian kernel is located, and \(\gamma\) is a hyperparameter that controls the degree to which the size of the Gaussian kernel is affected by neighboring triangles. - **Deformed Gaussian kernel**: \[ g'(x)=e^{-\frac{1}{2}(x - (\mu+\Delta P))^T(T_i\Sigma T_i^T)^{-1}(x - (\mu+\Delta P))} \] where \(\Delta P\) is the relative displacement after deformation and \(T_i\) is the deformation gradient matrix. ### Conclusion: Through the above methods, the paper successfully achieves large - scale real - time deformation based on 3D Gaussian Splatting while maintaining high - quality rendering effects. This provides a new solution for real - time editing and interaction in 3D scenes.