Projecting Gaussian Ellipsoids While Avoiding Affine Projection Approximation

Han Qi,Tao Cai,Xiyue Han

2024-11-12

Abstract:Recently, 3D Gaussian Splatting has dominated novel-view synthesis with its real-time rendering speed and state-of-the-art rendering quality. However, during the rendering process, the use of the Jacobian of the affine approximation of the projection transformation leads to inevitable errors, resulting in blurriness, artifacts and a lack of scene consistency in the final rendered images. To address this issue, we introduce an ellipsoid-based projection method to calculate the projection of Gaussian ellipsoid on the image plane, witch is the primitive of 3D Gaussian Splatting. As our proposed ellipsoid-based projection method cannot handle Gaussian ellipsoids with camera origins inside them or parts lying below $z=0$ plane in the camera space, we designed a pre-filtering strategy. Experiments over multiple widely adopted benchmark datasets show that using our ellipsoid-based projection method can enhance the rendering quality of 3D Gaussian Splatting and its extensions.

Computer Vision and Pattern Recognition,Graphics

What problem does this paper attempt to address?

The problem that this paper attempts to solve is the error problem introduced by the local affine approximation of the projection transformation during the rendering process of 3D Gaussian Splatting (3DGS). Specifically, these problems lead to a decline in the quality of the rendered image, manifested as blurring, artifacts, and insufficient scene consistency.

### Specific manifestations of the problem:
1. **Blurring and artifacts**: Since the Jacobian matrix of the local affine approximation is used to approximate the projection transformation during the rendering process, this results in relatively large errors at positions far from the center of the Gaussian ellipsoid, especially when dealing with distant objects.
2. **Insufficient scene consistency**: These errors not only affect the quality of a single image but also lead to scene consistency problems between different viewing angles, especially with a poorer performance in the test set than in the training set.

### Solution:
To solve the above - mentioned problems, the author proposes an ellipsoid - based projection method, which directly calculates the projection of the Gaussian ellipsoid on the image plane instead of relying on the local affine approximation. This method can represent the projection of the Gaussian ellipsoid more accurately, thereby improving the rendering quality and enhancing the scene consistency.

### Main contributions:
1. **Proposing an ellipsoid - based projection method**: By accurately calculating the projection of the Gaussian ellipsoid on the image plane, the error problem introduced by the affine approximation is solved, and the rendering quality and scene consistency are improved.
2. **Designing a pre - filtering strategy**: For two types of Gaussian ellipsoids that cannot be projected correctly (the camera origin is inside the ellipsoid or the ellipsoid is partially located below the z = 0 plane), a pre - filtering algorithm is designed, which enhances the robustness of the system and speeds up the rendering.
3. **Experimental verification**: Experiments on multiple benchmark data sets show that this method not only improves the rendering quality but also further accelerates the rendering speed.

Through these improvements, the author's method outperforms existing 3D Gaussian Splatting methods in multiple metrics, especially performing excellently in quality evaluation metrics such as PSNR, SSIM, and LPIPS.