Zian Wang,Tianchang Shen,Merlin Nimier-David,Nicholas Sharp,Jun Gao,Alexander Keller,Sanja Fidler,Thomas Müller,Zan Gojcic
Abstract:Neural radiance fields achieve unprecedented quality for novel view synthesis, but their volumetric formulation remains expensive, requiring a huge number of samples to render high-resolution images. Volumetric encodings are essential to represent fuzzy geometry such as foliage and hair, and they are well-suited for stochastic optimization. Yet, many scenes ultimately consist largely of solid surfaces which can be accurately rendered by a single sample per pixel. Based on this insight, we propose a neural radiance formulation that smoothly transitions between volumetric- and surface-based rendering, greatly accelerating rendering speed and even improving visual fidelity. Our method constructs an explicit mesh envelope which spatially bounds a neural volumetric representation. In solid regions, the envelope nearly converges to a surface and can often be rendered with a single sample. To this end, we generalize the NeuS formulation with a learned spatially-varying kernel size which encodes the spread of the density, fitting a wide kernel to volume-like regions and a tight kernel to surface-like regions. We then extract an explicit mesh of a narrow band around the surface, with width determined by the kernel size, and fine-tune the radiance field within this band. At inference time, we cast rays against the mesh and evaluate the radiance field only within the enclosed region, greatly reducing the number of samples required. Experiments show that our approach enables efficient rendering at very high fidelity. We also demonstrate that the extracted envelope enables downstream applications such as animation and simulation.
What problem does this paper attempt to address?
The main goal of this paper is to improve the performance of Neural Radiance Fields (NeRF) in efficient rendering while maintaining or enhancing its visual fidelity. Specifically, the paper addresses the following issues:
1. **Improving rendering efficiency**: Traditional NeRF methods, although capable of high-quality novel view synthesis, require a large number of sampling points to generate high-resolution images due to their volumetric rendering formula, leading to very high computational costs.
2. **Optimizing rendering strategies for different scene content**: The paper observes that different scene regions are suited to different rendering methods—complex transparent or fuzzy surfaces like hair benefit most from detailed volumetric rendering, while smooth opaque surfaces can be well represented with single sampling.
To address the above issues, the paper proposes a new method called "Adaptive Shell." This method combines the advantages of explicit geometric information and neural implicit representation to achieve efficient rendering performance. The main contributions include:
- **Introducing spatially varying kernel width**: The paper extends the NeuS framework by learning a spatially varying kernel width that can automatically adjust according to the surface type in the scene (e.g., hair, skin). For fuzzy surfaces (like hair), the kernel width is wider; for smooth surfaces (like skin), the kernel width tends towards a pulse function, thus reducing unnecessary sampling.
- **Extracting narrow band shells**: Utilizing the learned spatially varying kernel width, the paper proposes a method to extract a narrow band shell closely adhering to the object's surface. The width of this shell can adaptively adjust according to the scene's complexity. The narrow band shell, serving as an acceleration data structure, can effectively skip empty space and only sample in regions that significantly contribute to rendering.
- **Efficient rendering strategy**: During the inference phase, by projecting rays onto the narrow band shell and evaluating the radiance field only within the shell's internal region, the required number of samples is greatly reduced. For surface-like regions, the narrow band shell allows rendering with a single sample, while for fuzzy surfaces, a wider kernel width and local volumetric rendering are employed.
In summary, this research significantly improves the rendering efficiency of Neural Radiance Fields by introducing adaptive shells and spatially varying kernel widths, while maintaining or enhancing visual quality. This is of great significance for achieving real-time high-quality rendering.