Abstract:Neural Radiance Fields (NeRFs) have proven to be powerful 3D representations, capable of high quality novel view synthesis of complex scenes. While NeRFs have been applied to graphics, vision, and robotics, problems with slow rendering speed and characteristic visual artifacts prevent adoption in many use cases. In this work, we investigate combining an autoencoder (AE) with a NeRF, in which latent features (instead of colours) are rendered and then convolutionally decoded. The resulting latent-space NeRF can produce novel views with higher quality than standard colour-space NeRFs, as the AE can correct certain visual artifacts, while rendering over three times faster. Our work is orthogonal to other techniques for improving NeRF efficiency. Further, we can control the tradeoff between efficiency and image quality by shrinking the AE architecture, achieving over 13 times faster rendering with only a small drop in performance. We hope that our approach can form the basis of an efficient, yet high-fidelity, 3D scene representation for downstream tasks, especially when retaining differentiability is useful, as in many robotics scenarios requiring continual learning.

What problem does this paper attempt to address?

The paper aims to address two main issues faced by Neural Radiance Fields (NeRF) in practical applications: slow rendering speed and the presence of specific visual artifacts. Specifically, the paper proposes a method called Reconstructive Latent-Space Neural Radiance Fields (ReLS-NeRF). This method combines Autoencoder (AE) and NeRF technologies, enabling the generation of high-quality novel view images with higher efficiency and reducing the visual artifacts originally present in NeRF. The key innovations of ReLS-NeRF include: 1. **Latent Space Rendering**: Unlike traditional NeRF which directly outputs color information, ReLS-NeRF outputs latent feature vectors during the rendering process. These latent features are then converted into high-resolution RGB images through a decoder. This design reduces the number of computationally intensive Multi-Layer Perceptron (MLP) calls, significantly improving rendering speed. 2. **Role of Autoencoder**: By using a pre-trained autoencoder, ReLS-NeRF not only improves rendering efficiency but also enhances image quality. This is because the decoder can act as an image prior, helping to correct some of the visual artifacts produced by direct NeRF rendering. 3. **Trade-off Between Performance and Efficiency**: The study also demonstrates how to balance image quality and rendering speed by adjusting the architecture of the autoencoder. For example, reducing the size of the autoencoder can achieve more than 13 times rendering acceleration with only a slight decrease in quality. Additionally, to evaluate the effectiveness of this new method, the authors introduce a new evaluation metric—Reprojective Colour Consistency (RCC), which is used to detect local appearance changes due to view inconsistency. Experimental results show that compared to standard color space NeRF, ReLS-NeRF performs better on multiple image and video quality metrics while maintaining faster rendering speed.

Reconstructive Latent-Space Neural Radiance Fields for Efficient 3D Scene Representations

Baking Neural Radiance Fields for Real-Time View Synthesis

Enhance-NeRF: Multiple Performance Evaluation for Neural Radiance Fields

Neural Radiance Fields From Sparse RGB-D Images for High-Quality View Synthesis

HyperNeRF: A Higher-Dimensional Representation for Topologically Varying Neural Radiance Fields

NeRFs: The Search for the Best 3D Representation

NeRF: Neural Radiance Field in 3D Vision, A Comprehensive Review

Instant Continual Learning of Neural Radiance Fields

Neural Radiance Fields (NeRFs): A Review and Some Recent Developments

Fast Radiance Field Reconstruction from Sparse Inputs

Learning Neural Duplex Radiance Fields for Real-Time View Synthesis

NeRFusion: Fusing Radiance Fields for Large-Scale Scene Reconstruction

NeRFMeshing: Distilling Neural Radiance Fields into Geometrically-Accurate 3D Meshes

NeRF-SDP: Efficient Generalizable Neural Radiance Field with Scene Depth Perception.

BioNeRF: Biologically Plausible Neural Radiance Fields for View Synthesis

Adaptive Multi-NeRF: Exploit Efficient Parallelism in Adaptive Multiple Scale Neural Radiance Field Rendering

Ced-NeRF: A Compact and Efficient Method for Dynamic Neural Radiance Fields

NeRFuser: Large-Scale Scene Representation by NeRF Fusion

Tetra-NeRF: Representing Neural Radiance Fields Using Tetrahedra

DA4NeRF: Depth-aware augmentation technique for neural radiance fields