Abstract:Neural Radiance Fields (NeRFs) have demonstrated remarkable potential in capturing complex 3D scenes with high fidelity. However, one persistent challenge that hinders the widespread adoption of NeRFs is the computational bottleneck due to the volumetric rendering. On the other hand, 3D Gaussian splatting (3DGS) has recently emerged as an alternative representation that leverages a 3D Gaussisan-based representation and adopts the rasterization pipeline to render the images rather than volumetric rendering, achieving very fast rendering speed and promising image quality. However, a significant drawback arises as 3DGS entails a substantial number of 3D Gaussians to maintain the high fidelity of the rendered images, which requires a large amount of memory and storage. To address this critical issue, we place a specific emphasis on two key objectives: reducing the number of Gaussian points without sacrificing performance and compressing the Gaussian attributes, such as view-dependent color and covariance. To this end, we propose a learnable mask strategy that significantly reduces the number of Gaussians while preserving high performance. In addition, we propose a compact but effective representation of view-dependent color by employing a grid-based neural field rather than relying on spherical harmonics. Finally, we learn codebooks to compactly represent the geometric attributes of Gaussian by vector quantization. With model compression techniques such as quantization and entropy coding, we consistently show over 25$\times$ reduced storage and enhanced rendering speed, while maintaining the quality of the scene representation, compared to 3DGS. Our work provides a comprehensive framework for 3D scene representation, achieving high performance, fast training, compactness, and real-time rendering. Our project page is available at <a class="link-external link-https" href="https://maincold2.github.io/c3dgs/" rel="external noopener nofollow">this https URL</a>.

Learning-based Ray Sampling Strategy for Computation Efficient Neural Radiance Field Generation

Efficient Ray Sampling for Radiance Fields Reconstruction

Improving Neural Radiance Field using Near-Surface Sampling with Point Cloud Generation

Area-NeRF: Area-based Neural Radiance Fields

AdaNeRF: Adaptive Sampling for Real-time Rendering of Neural Radiance Fields

Adaptive Incident Radiance Field Sampling and Reconstruction Using Deep Reinforcement Learning.

NeuSample: Neural Sample Field for Efficient View Synthesis

Efficient Neural Radiance Fields with Learned Depth-Guided Sampling

SG-NeRF: Semantic-guided Point-based Neural Radiance Fields

ProNeRF: Learning Efficient Projection-Aware Ray Sampling for Fine-Grained Implicit Neural Radiance Fields

Efficient NeRF Optimization -- Not All Samples Remain Equally Hard

Accelerated Neural Radiation Field Based on Optimal Ray Sampling

ActRay: Online Active Ray Sampling for Radiance Fields.

Efficient View Synthesis with Neural Radiance Distribution Field

Sampling Neural Radiance Fields for Refractive Objects.

Adaptive Multi-NeRF: Exploit Efficient Parallelism in Adaptive Multiple Scale Neural Radiance Field Rendering

ACFNeRF: Accelerating and Cache-Free Neural Rendering Via Point Cloud-Based Distance Fields.

Where and How: Mitigating Confusion in Neural Radiance Fields from Sparse Inputs

Rendering stable features improves sampling-based localisation with Neural radiance fields

Compact 3D Gaussian Representation for Radiance Field