Abstract:The recently proposed neural radiance fields (NeRF) use a continuous function formulated as a multi-layer perceptron (MLP) to model the appearance and geometry of a 3D scene. This enables realistic synthesis of novel views, even for scenes with view dependent appearance. Many follow-up works have since extended NeRFs in different ways. However, a fundamental restriction of the method remains that it requires a large number of images captured from densely placed viewpoints for high-quality synthesis and the quality of the results quickly degrades when the number of captured views is insufficient. To address this problem, we propose a novel NeRF-based framework capable of high-quality view synthesis using only a sparse set of RGB-D images, which can be easily captured using cameras and LiDAR sensors on current consumer devices. First, a geometric proxy of the scene is reconstructed from the captured RGB-D images. Renderings of the reconstructed scene along with precise camera parameters can then be used to pre-train a network. Finally, the network is fine-tuned with a small number of real captured images. We further introduce a patch discriminator to supervise the network under novel views during fine-tuning, as well as a 3D color prior to improve synthesis quality. We demonstrate that our method can generate arbitrary novel views of a 3D scene from as few as 6 RGB-D images. Extensive experiments show the improvements of our method compared with the existing NeRF-based methods, including approaches that also aim to reduce the number of input images.

Neural Radiance Flow for 4D View Synthesis and Video Processing

Forward Flow for Novel View Synthesis of Dynamic Scenes

NeRFPlayer: A Streamable Dynamic Scene Representation with Decomposed Neural Radiance Fields

Neural Radiance Fields From Sparse RGB-D Images for High-Quality View Synthesis

Neural 3D Video Synthesis from Multi-view Video

BeNeRF: Neural Radiance Fields from a Single Blurry Image and Event Stream

Space-time Neural Irradiance Fields for Free-Viewpoint Video

Gaussian-Flow: 4D Reconstruction with Dynamic 3D Gaussian Particle

Depth Guided Fast Rendering Neural Radiance Fields

Reconstructive Latent-Space Neural Radiance Fields for Efficient 3D Scene Representations

Learning Neural Duplex Radiance Fields for Real-Time View Synthesis

SLS4D: Sparse Latent Space for 4D Novel View Synthesis

Dense Depth Priors for Neural Radiance Fields from Sparse Input Views

Detachable Novel Views Synthesis of Dynamic Scenes Using Distribution-Driven Neural Radiance Fields

NeVRF: Neural Video-based Radiance Fields for Long-duration Sequences

Deep Flow Rendering: View Synthesis via Layer-aware Reflection Flow

UHDNeRF: Ultra-High-Definition Neural Radiance Fields

Fast Neural Scene Flow

4D Facial Avatar Reconstruction From Monocular Video via Efficient and Controllable Neural Radiance Fields

Semantic Flow: Learning Semantic Field of Dynamic Scenes from Monocular Videos

Dynamic Neural Radiance Field From Defocused Monocular Video