Abstract:Rendering photo-realistic novel-view images of complex scenes has been a long-standing challenge in computer graphics. In recent years, great research progress has been made on enhancing rendering quality and accelerating rendering speed in the realm of view synthesis. However, when rendering complex dynamic scenes with sparse views, the rendering quality remains limited due to occlusion problems. Besides, for rendering high-resolution images on dynamic scenes, the rendering speed is still far from real-time. In this work, we propose a generalizable view synthesis method that can render high-resolution novel-view images of complex static and dynamic scenes in real-time from sparse views. To address the occlusion problems arising from the sparsity of input views and the complexity of captured scenes, we introduce an explicit 3D visibility reasoning approach that can efficiently estimate the visibility of sampled 3D points to the input views. The proposed visibility reasoning approach is fully differentiable and can gracefully fit inside the volume rendering pipeline, allowing us to train our networks with only multi-view images as supervision while refining geometry and texture simultaneously. Besides, each module in our pipeline is carefully designed to bypass the time-consuming MLP querying process and enhance the rendering quality of high-resolution images, enabling us to render high-resolution novel-view images in real-time.Experimental results show that our method outperforms previous view synthesis methods in both rendering quality and speed, particularly when dealing with complex dynamic scenes with sparse views.

Depth-Assisted Full Resolution Network for Single Image-Based View Synthesis.

Ivs-Net: Learning Human View Synthesis from Internet Videos

Asymmetric Bidirectional View Synthesis for Free Viewpoint and Three-Dimensional Video

Hierarchical Hole Filling for Novel View Synthesis

DCL: Differential Contrastive Learning for Geometry-Aware Depth Synthesis

A neural refinement network for single image view synthesis

Multi-Scale Feature Fusion for Single Image Novel View Synthesis

Real-Time Unsupervised Multi-View Depth Estimation Network for Virtual View Synthesis

Depth self-supervision for single image novel view synthesis

Real-time High-resolution View Synthesis of Complex Scenes with Explicit 3D Visibility Reasoning

Novel View Synthesis Using Feature-Preserving Depth Map Resampling

A Novel Method of Multi-view Virtual Image Synthesis for Auto-stereoscopic Display

Novel View Synthesis from only a 6-DoF Camera Pose by Two-stage Networks

Novel View Synthesis from a Single Unposed Image via Unsupervised Learning

Self-Supervised Visibility Learning for Novel View Synthesis

Single-View View Synthesis with Self-Rectified Pseudo-Stereo

Depth-image Based View Synthesis for Threedimensional Television

Robust View Synthesis in Wide-baseline Complex Geometric Environments

View synthesis with multiplane images from computationally generated RGB-D light fields

ReShader: View-Dependent Highlights for Single Image View-Synthesis

High-Fidelity and Real-Time Novel View Synthesis for Dynamic Scenes.