3D Convex Splatting: Radiance Field Rendering with 3D Smooth Convexes

Jan Held,Renaud Vandeghen,Abdullah Hamdi,Adrien Deliege,Anthony Cioppa,Silvio Giancola,Andrea Vedaldi,Bernard Ghanem,Marc Van Droogenbroeck

2024-11-22

Abstract:Recent advances in radiance field reconstruction, such as 3D Gaussian Splatting (3DGS), have achieved high-quality novel view synthesis and fast rendering by representing scenes with compositions of Gaussian primitives. However, 3D Gaussians present several limitations for scene reconstruction. Accurately capturing hard edges is challenging without significantly increasing the number of Gaussians, creating a large memory footprint. Moreover, they struggle to represent flat surfaces, as they are diffused in space. Without hand-crafted regularizers, they tend to disperse irregularly around the actual surface. To circumvent these issues, we introduce a novel method, named 3D Convex Splatting (3DCS), which leverages 3D smooth convexes as primitives for modeling geometrically-meaningful radiance fields from multi-view images. Smooth convex shapes offer greater flexibility than Gaussians, allowing for a better representation of 3D scenes with hard edges and dense volumes using fewer primitives. Powered by our efficient CUDA-based rasterizer, 3DCS achieves superior performance over 3DGS on benchmarks such as Mip-NeRF360, Tanks and Temples, and Deep Blending. Specifically, our method attains an improvement of up to 0.81 in PSNR and 0.026 in LPIPS compared to 3DGS while maintaining high rendering speeds and reducing the number of required primitives. Our results highlight the potential of 3D Convex Splatting to become the new standard for high-quality scene reconstruction and novel view synthesis. Project page: <a class="link-external link-http" href="http://www.convexsplatting.com" rel="external noopener nofollow">this http URL</a>.

Computer Vision and Pattern Recognition

What problem does this paper attempt to address?

The problem that this paper attempts to solve is the limitations of current scene reconstruction methods based on Gaussian distribution when representing complex geometric structures. Specifically:

1. **Gaussian distribution lacks clear physical boundaries**: Due to its smoothness and circular characteristics, the Gaussian distribution is difficult to accurately represent planes or achieve physically meaningful scene decomposition. This makes them perform poorly in capturing hard edges and geometric structures.

2. **Gaussian distribution is difficult to represent hard edges and planes**: Each Gaussian distribution is similar to an ellipsoid and has the characteristics of symmetric distribution, and it is difficult to adapt to angular boundaries or planar surfaces. This inherent limitation is particularly evident in the sphere - packing problem, that is, densely arranged spheres or ellipsoids will leave gaps at flat or sharp corners, resulting in low coverage efficiency. Therefore, a large number of Gaussian distributions are required to fill the space without gaps, which will increase memory consumption and computational overhead.

To overcome these limitations, the paper proposes a new method - 3D Convex Splatting (3DCS), which uses 3D smooth convex shapes as basic units to model and reconstruct geometrically accurate radiation fields extracted from multi - view images. 3D smooth convex shapes provide greater flexibility than Gaussian distributions and can accurately capture hard edges and detailed surfaces with fewer basic units to form a dense volume.

### Main contributions:
1. **Introducing 3D Convex Splatting (3DCS)**: Using 3D smooth convex shapes as new basic units, it solves the limitations of Gaussian distribution in capturing dense volume features.
2. **Developing an optimization framework and a fast differentiable GPU rendering pipeline**: High - quality 3D scene representation and high - speed rendering are achieved.
3. **Surpassing existing methods on multiple datasets**: On the Mip - NeRF360, Tanks and Temples, and Deep Blending datasets, 3DCS outperforms 3D Gaussian Splatting (3DGS) with fewer basic units.

Through these contributions, 3DCS is expected to become a new standard for high - quality scene reconstruction and new view synthesis.