Abstract:Novel view synthesis from an in-the-wild video is difficult due to challenges like scene dynamics and lack of parallax. While existing methods have shown promising results with implicit neural radiance fields, they are slow to train and render. This paper revisits explicit video representations to synthesize high-quality novel views from a monocular video efficiently. We treat static and dynamic video content separately. Specifically, we build a global static scene model using an extended plane-based scene representation to synthesize temporally coherent novel video. Our plane-based scene representation is augmented with spherical harmonics and displacement maps to capture view-dependent effects and model non-planar complex surface geometry. We opt to represent the dynamic content as per-frame point clouds for efficiency. While such representations are inconsistency-prone, minor temporal inconsistencies are perceptually masked due to motion. We develop a method to quickly estimate such a hybrid video representation and render novel views in real time. Our experiments show that our method can render high-quality novel views from an in-the-wild video with comparable quality to state-of-the-art methods while being 100x faster in training and enabling real-time rendering.

What problem does this paper attempt to address?

The paper aims to address the problem of efficiently synthesizing high-quality novel views from monocular videos, with a particular focus on handling scene dynamics and the challenge of lack of parallax. Specifically, the paper proposes a hybrid representation approach that processes static and dynamic video content separately, enabling rapid training from monocular videos and real-time rendering of high-quality novel views. For static content, the paper adopts an extended plane-based scene representation, using a collection of planes (soup-of-planes) to model static scenes, combined with spherical harmonics and displacement mapping to capture view-dependent effects and complex non-planar surface geometries. This method is not only suitable for low-parallax scenes but also effectively models general static scenes. For dynamic content, considering the complex motion of nearby objects, the paper opts to use per-frame point clouds for representation to improve efficiency. Although this representation may introduce temporal inconsistencies, small inconsistencies are often not perceived due to motion masking effects. The paper also developed a set of loss functions and optimization algorithms for estimating this hybrid video representation from monocular videos, achieving optimization on a single GPU in just 15 minutes. Experimental results show that the quality of the novel views generated by this method on NVIDIA and DAVIS datasets is comparable to the current state-of-the-art (SOTA) Neural Radiance Fields (NeRF) methods, but is over 100 times faster in training and rendering speed. In summary, the contributions of the paper include: 1. Proposing a hybrid explicit non-neural representation that can model static and dynamic video content, support view-dependent effects and complex surface geometries, while enabling real-time rendering; 2. Developing a video-specific optimization algorithm, combined with carefully designed loss functions, for estimating the hybrid video representation from monocular videos; 3. Extensive evaluation on NVIDIA and DAVIS datasets shows that the method can generate novel views of comparable quality to SOTA NeRF methods while significantly reducing training and rendering time.

Fast View Synthesis of Casual Videos with Soup-of-Planes

Ivs-Net: Learning Human View Synthesis from Internet Videos

Pseudo-Generalized Dynamic View Synthesis from a Video

ViewCrafter: Taming Video Diffusion Models for High-fidelity Novel View Synthesis

Novel View Synthesis of Dynamic Human with Sparse Cameras.

Diffusion Priors for Dynamic View Synthesis from Monocular Videos

Real-time High-resolution View Synthesis of Complex Scenes with Explicit 3D Visibility Reasoning

Neural 3D Video Synthesis from Multi-view Video

ViVid-1-to-3: Novel View Synthesis with Video Diffusion Models

View Synthesis of Dynamic Scenes based on Deep 3D Mask Volume

Deep Video‐Based Performance Synthesis from Sparse Multi‐View Capture

Novel View Synthesis of Human Interactions from Sparse Multi-view Videos

Decoupling Dynamic Monocular Videos for Dynamic View Synthesis

A Construct-Optimize Approach to Sparse View Synthesis without Camera Pose

DynPoint: Dynamic Neural Point For View Synthesis

High-Fidelity and Real-Time Novel View Synthesis for Dynamic Scenes.

View Synthesis with Sculpted Neural Points

R4D-planes: Remapping Planes for Novel View Synthesis and Self-Supervised Decoupling of Monocular Videos

Mixed Neural Voxels for Fast Multi-view Video Synthesis

A Fast Free-viewpoint Video Synthesis Algorithm for Sports Scenes