Abstract:Rendering photo-realistic novel-view images of complex scenes has been a long-standing challenge in computer graphics. In recent years, great research progress has been made on enhancing rendering quality and accelerating rendering speed in the realm of view synthesis. However, when rendering complex dynamic scenes with sparse views, the rendering quality remains limited due to occlusion problems. Besides, for rendering high-resolution images on dynamic scenes, the rendering speed is still far from real-time. In this work, we propose a generalizable view synthesis method that can render high-resolution novel-view images of complex static and dynamic scenes in real-time from sparse views. To address the occlusion problems arising from the sparsity of input views and the complexity of captured scenes, we introduce an explicit 3D visibility reasoning approach that can efficiently estimate the visibility of sampled 3D points to the input views. The proposed visibility reasoning approach is fully differentiable and can gracefully fit inside the volume rendering pipeline, allowing us to train our networks with only multi-view images as supervision while refining geometry and texture simultaneously. Besides, each module in our pipeline is carefully designed to bypass the time-consuming MLP querying process and enhance the rendering quality of high-resolution images, enabling us to render high-resolution novel-view images in real-time.Experimental results show that our method outperforms previous view synthesis methods in both rendering quality and speed, particularly when dealing with complex dynamic scenes with sparse views.

Realistic View Synthesis of a Structured Traffic Environment Via Adversarial Training

3D-Aware Image Synthesis Via Learning Structural and Textural Representations

Virtual View Generation Based on 3D-Dense-attentive GAN Networks

Synthesizing Traffic Datasets using Graph Neural Networks

View Independent Generative Adversarial Network for Novel View Synthesis

CRTGAN: Controllable Road Network Graphs Generation Via Transformer Based GAN

Generating Traffic Scene with Deep Convolutional Generative Adversarial Networks

Style Transfer of Urban Road Images Using Generative Adversarial Networks with Structural Details

Unsupervised Traffic Scene Generation with Synthetic 3D Scene Graphs

Novel View Synthesis from only a 6-DoF Camera Pose by Two-stage Networks

SST-GAN: Single Sample-based Realistic Traffic Image Generation for Parallel Vision

Structure-aware Roadview Synthesis for Testing Autonomous Perception

TL-GAN: Improving Traffic Light Recognition via Data Synthesis for Autonomous Driving

FreeVS: Generative View Synthesis on Free Driving Trajectory

Exploring Generative AI for Sim2Real in Driving Data Synthesis

Realistic Image Synthesis with Configurable 3D Scene Layouts

Realistic Data-Driven Traffic Flow Animation Using Texture Synthesis.

Inferring Intersection Traffic Patterns With Sparse Video Surveillance Information: An ST-GAN Method

Real-time High-resolution View Synthesis of Complex Scenes with Explicit 3D Visibility Reasoning

Generative Novel View Synthesis with 3D-Aware Diffusion Models

StackGAN++: Realistic Image Synthesis with Stacked Generative Adversarial Networks