Abstract:3D Gaussian splatting has achieved very impressive performance in real-time novel view synthesis. However, it often suffers from over-reconstruction during Gaussian densification where high-variance image regions are covered by a few large Gaussians only, leading to blur and artifacts in the rendered images. We design a progressive frequency regularization (FreGS) technique to tackle the over-reconstruction issue within the frequency space. Specifically, FreGS performs coarse-to-fine Gaussian densification by exploiting low-to-high frequency components that can be easily extracted with low-pass and high-pass filters in the Fourier space. By minimizing the discrepancy between the frequency spectrum of the rendered image and the corresponding ground truth, it achieves high-quality Gaussian densification and alleviates the over-reconstruction of Gaussian splatting effectively. Experiments over multiple widely adopted benchmarks (e.g., Mip-NeRF360, Tanks-and-Temples and Deep Blending) show that FreGS achieves superior novel view synthesis and outperforms the state-of-the-art consistently.
What problem does this paper attempt to address?
The problem that this paper attempts to solve is the over - reconstruction problem that often occurs during Gaussian densification in 3D Gaussian Splatting (3D - GS). Specifically, when performing Gaussian densification on high - variance image regions, these regions are often covered by a few large Gaussian functions, resulting in blurring and artifacts in the rendered image.
To solve this problem, the author proposes a new method named FreGS (3D Gaussian Splatting with Progressive Frequency Regularization). FreGS alleviates the over - reconstruction problem by introducing progressive frequency regularization techniques, thereby improving the quality of Gaussian densification and generating higher - quality novel view - synthesis images.
### Key points of the solution:
1. **Progressive frequency regularization**: FreGS utilizes the step - by - step regularization of low - to - high - frequency signals to gradually optimize the Gaussian densification process from low to high frequencies.
2. **Frequency annealing technique**: Extract low - and high - frequency components in Fourier space through low - pass filters and dynamic high - pass filters to achieve coarse - to - fine Gaussian densification.
3. **Frequency spectrum difference minimization**: Minimize the frequency spectrum difference between the rendered image and the real image to ensure that the details and structures in the Gaussian densification process are more accurate.
### Formula summary:
- Frequency representation:
\[
F(u, v)=\sum_{x = 0}^{H - 1}\sum_{y = 0}^{W - 1}I(x, y)\cdot e^{-i2\pi(ux/H+vy/W)}
\]
- Magnitude and phase:
\[
\|F(u, v)\|=\sqrt{\text{Re}(u, v)^{2}+\text{Im}(u, v)^{2}}
\]
\[
\angle F(u, v)=\arctan\left(\frac{\text{Im}(u, v)}{\text{Re}(u, v)}\right)
\]
- Magnitude difference:
\[
d_{a}=\frac{1}{\sqrt{HW}}\sum_{x = 0}^{H - 1}\sum_{y = 0}^{W - 1}\left|\|F(u, v)\|-\|\hat{F}(u, v)\|\right|
\]
- Phase difference:
\[
d_{p}=\frac{1}{\sqrt{HW}}\sum_{x = 0}^{H - 1}\sum_{y = 0}^{W - 1}\left|\angle F(u, v)-\angle\hat{F}(u, v)\right|
\]
Through these methods, FreGS effectively solves the over - reconstruction problem in 3D Gaussian Splatting and improves the quality of the rendered image, reducing blurring and artifacts.