Abstract:We present a novel neural algorithm for performing high-quality, high-resolution, real-time novel view synthesis. From a sparse set of input RGB images or videos streams, our network both reconstructs the 3D scene and renders novel views at 1080p resolution at 30fps on an NVIDIA A100. Our feed-forward network generalizes across a wide variety of datasets and scenes and produces state-of-the-art quality for a real-time method. Our quality approaches, and in some cases surpasses, the quality of some of the top offline methods. In order to achieve these results we use a novel combination of several key concepts, and tie them together into a cohesive and effective algorithm. We build on previous works that represent the scene using semi-transparent layers and use an iterative learned render-and-refine approach to improve those layers. Instead of flat layers, our method reconstructs layered depth maps (LDMs) that efficiently represent scenes with complex depth and occlusions. The iterative update steps are embedded in a multi-scale, UNet-style architecture to perform as much compute as possible at reduced resolution. Within each update step, to better aggregate the information from multiple input views, we use a specialized Transformer-based network component. This allows the majority of the per-input image processing to be performed in the input image space, as opposed to layer space, further increasing efficiency. Finally, due to the real-time nature of our reconstruction and rendering, we dynamically create and discard the internal 3D geometry for each frame, generating the LDM for each view. Taken together, this produces a novel and effective algorithm for view synthesis. Through extensive evaluation, we demonstrate that we achieve state-of-the-art quality at real-time rates. Project page: <a class="link-external link-https" href="https://quark-3d.github.io/" rel="external noopener nofollow">this https URL</a>

Neural 3D Video Synthesis from Multi-view Video

MPS-NeRF: Generalizable 3D Human Rendering from Multiview Images

Neural 3D Video Synthesis

High-Fidelity and Real-Time Novel View Synthesis for Dynamic Scenes.

DynPoint: Dynamic Neural Point For View Synthesis

D-NPC: Dynamic Neural Point Clouds for Non-Rigid View Synthesis from Monocular Video

Mixed Neural Voxels for Fast Multi-view Video Synthesis

Quark: Real-time, High-resolution, and General Neural View Synthesis

ViewCrafter: Taming Video Diffusion Models for High-fidelity Novel View Synthesis

A Portable Multiscopic Camera for Novel View and Time Synthesis in Dynamic Scenes

View Synthesis of Dynamic Scenes based on Deep 3D Mask Volume

NeuLF: Efficient Novel View Synthesis with Neural 4D Light Field.

DynMF: Neural Motion Factorization for Real-time Dynamic View Synthesis with 3D Gaussian Splatting

IBRNet: Learning Multi-View Image-Based Rendering

Neural Body: Implicit Neural Representations with Structured Latent Codes for Novel View Synthesis of Dynamic Humans

Neural Radiance Flow for 4D View Synthesis and Video Processing

Dynamic Multi-View Scene Reconstruction Using Neural Implicit Surface

Diffusion Priors for Dynamic View Synthesis from Monocular Videos

Multi-View Stereo and Depth Priors Guided NeRF for View Synthesis.

Collaborative neural radiance fields for novel view synthesis