Multi-Scale 3D Gaussian Splatting for Anti-Aliased Rendering

Zhiwen Yan,Weng Fei Low,Yu Chen,Gim Hee Lee

2024-05-29

Abstract:3D Gaussians have recently emerged as a highly efficient representation for 3D reconstruction and rendering. Despite its high rendering quality and speed at high resolutions, they both deteriorate drastically when rendered at lower resolutions or from far away camera position. During low resolution or far away rendering, the pixel size of the image can fall below the Nyquist frequency compared to the screen size of each splatted 3D Gaussian and leads to aliasing effect. The rendering is also drastically slowed down by the sequential alpha blending of more splatted Gaussians per pixel. To address these issues, we propose a multi-scale 3D Gaussian splatting algorithm, which maintains Gaussians at different scales to represent the same scene. Higher-resolution images are rendered with more small Gaussians, and lower-resolution images are rendered with fewer larger Gaussians. With similar training time, our algorithm can achieve 13\%-66\% PSNR and 160\%-2400\% rendering speed improvement at 4$\times$-128$\times$ scale rendering on Mip-NeRF360 dataset compared to the single scale 3D Gaussian splitting. Our code and more results are available on our project website <a class="link-external link-https" href="https://jokeryan.github.io/projects/ms-gs/" rel="external noopener nofollow">this https URL</a>

Computer Vision and Pattern Recognition

What problem does this paper attempt to address?

This paper addresses the severe aliasing (anti-aliasing) and performance degradation issues of 3D Gaussian scattering in low-resolution or long-distance rendering. 3D Gaussian scattering is an efficient new view synthesis representation, but at low resolutions, the insufficient pixel sampling frequency leads to inaccurate capturing of continuous signals, resulting in distortion (aliasing). At the same time, the rendering speed significantly decreases as the number of Gaussian bodies scattered on each pixel increases.

To address these issues, the paper proposes a multiscale 3D Gaussian scattering algorithm. This algorithm represents the same scene by maintaining Gaussian bodies of different sizes at different levels of detail (LOD). More small Gaussian bodies are used at higher resolutions, while fewer large Gaussian bodies are used at lower resolutions. This allows for an improvement in the fidelity (PSNR) of low-resolution rendering by 13%-66% and a speed increase of 160%-2400%, while keeping the training time similar.

The paper also discusses the limitations of traditional anti-aliasing techniques (such as supersampling, multisampling, and fast approximate anti-aliasing) in 3D Gaussian scattering and analyzes the causes of aliasing effects in 3D Gaussian scattering. The authors found that the concentration of small Gaussian bodies results in color dominance by the front Gaussian bodies and a decrease in rendering speed.

To address the above issues, the paper proposes a pixel coverage-based multiscale Gaussian body approach. During the training phase, small Gaussian bodies are aggregated and enlarged into large Gaussian bodies, and the appropriate scale of Gaussian bodies is selected for rendering based on the pixel coverage range. This approach reduces the influence of small Gaussian bodies on pixel colors while improving rendering efficiency.

The experiments show that compared to the original 3D Gaussian scattering, this method significantly improves rendering quality and speed at low resolutions, with minimal impact on rendering quality at the original resolution. Through quantitative and qualitative comparisons, the paper demonstrates the effectiveness of the proposed method.