Ref-GS: Directional Factorization for 2D Gaussian Splatting

Youjia Zhang,Anpei Chen,Yumin Wan,Zikai Song,Junqing Yu,Yawei Luo,Wei Yang

2024-12-02

Abstract:In this paper, we introduce Ref-GS, a novel approach for directional light factorization in 2D Gaussian splatting, which enables photorealistic view-dependent appearance rendering and precise geometry recovery. Ref-GS builds upon the deferred rendering of Gaussian splatting and applies directional encoding to the deferred-rendered surface, effectively reducing the ambiguity between orientation and viewing angle. Next, we introduce a spherical Mip-grid to capture varying levels of surface roughness, enabling roughness-aware Gaussian shading. Additionally, we propose a simple yet efficient geometry-lighting factorization that connects geometry and lighting via the vector outer product, significantly reducing renderer overhead when integrating volumetric attributes. Our method achieves superior photorealistic rendering for a range of open-world scenes while also accurately recovering geometry.

Computer Vision and Pattern Recognition,Graphics

What problem does this paper attempt to address?

The main problem that this paper attempts to solve is the limitations of existing 3D reconstruction and rendering methods when dealing with complex materials (such as reflection and refraction), especially in terms of geometric precision and realistic rendering of view - dependent effects. Specifically:

1. **Realistic Rendering of View - Dependent Effects**: Existing NeRF and its derivative methods mainly focus on the light propagation from the light source to the camera, while ignoring the reflection and refraction of light on the object surface, resulting in significant artifacts in the geometric structure and rendered images when reconstructing scenes with complex materials.

2. **Geometric Precision Problem**: Although some methods can generate high - quality view synthesis, they often have difficulty accurately recovering the geometric structure when dealing with highly reflective or transparent objects.

To solve these problems, the paper proposes Ref - GS (Directional Factorization for 2D Gaussian Splatting). By introducing directional factorization and deferred rendering techniques, more accurate geometric recovery and realistic rendering of view - dependent effects are achieved. Specific improvements include:

- **Deferred Rendering and Directional Encoding**: Through the deferred rendering technique, shading is performed after fusing Gaussian attributes, effectively reducing the ambiguity in directional queries.
- **Multi - level Spherical Feature Grid**: The Sph - Mip encoding is introduced to capture different levels of surface roughness, thereby enabling the modeling of far - field illumination.
- **Decomposition of Geometry and Illumination**: A simple geometry - illumination decomposition method is proposed. By using the vector outer product to connect geometry and illumination, the rendering overhead during volume attribute integration is significantly reduced.

These improvements enable Ref - GS to achieve high - quality reflection and refraction reconstruction in a variety of open - world scenes and maintain efficient real - time rendering performance.

### Formula Summary

1. **Definition of Gaussian Distribution**:
\[
G(x|\mu, \Sigma) = \exp\left(-\frac{1}{2}(x - \mu)^T \Sigma^{-1} (x - \mu)\right)
\]
where the covariance matrix \(\Sigma\) is decomposed into a rotation matrix \(R\) and a scaling matrix \(S\):
\[
\Sigma = R S S^T R^T
\]

2. **Color Calculation**:
\[
c(r) = \sum_{i = 1}^{N} c_i \alpha_i G_i(u) T_i
\]
where \(T_i\) is the cumulative transmittance, defined as:
\[
T_i = \prod_{j = 1}^{i - 1} (1 - \alpha_j G_j(u))
\]

3. **Final Color Prediction**:
\[
I = I_d + f_\Theta(S, K\otimes S)
\]
where \(\otimes\) represents the per - pixel outer product, \(K\) is the spatial feature, and \(S\) is the directional feature.

4. **Spherical Coordinate Transformation**:
\[
\theta = \arccos\left(\frac{\omega_z}{\sqrt{\omega_x^2+\omega_y^2+\omega_z^2}}\right)\in[0, \pi]
\]
\[
\phi = \arctan2\left(\frac{\omega_y}{\omega_x}\right)\in[-\pi, \pi]
\]

Through these formulas and techniques, Ref - GS achieves high - quality reconstruction of view - dependent effects and geometric recovery.