Abstract:Novel view synthesis is a long-standing problem that revolves around rendering frames of scenes from novel camera viewpoints. Volumetric approaches provide a solution for modeling occlusions through the explicit 3D representation of the camera frustum. Multi-plane Images (MPI) are volumetric methods that represent the scene using front-parallel planes at distinct depths but suffer from depth discretization leading to a 2.D scene representation. Another line of approach relies on implicit 3D scene representations. Neural Radiance Fields (NeRF) utilize neural networks for encapsulating the continuous 3D scene structure within the network weights achieving photorealistic synthesis results, however, methods are constrained to per-scene optimization settings which are inefficient in practice. Multi-plane Neural Radiance Fields (MINE) open the door for combining implicit and explicit scene representations. It enables continuous 3D scene representations, especially in the depth dimension, while utilizing the input image features to avoid per-scene optimization. The main drawback of the current literature work in this domain is being constrained to single-view input, limiting the synthesis ability to narrow viewpoint ranges. In this work, we thoroughly examine the performance, generalization, and efficiency of single-view multi-plane neural radiance fields. In addition, we propose a new multiplane NeRF architecture that accepts multiple views to improve the synthesis results and expand the viewing range. Features from the input source frames are effectively fused through a proposed attention-aware fusion module to highlight important information from different viewpoints. Experiments show the effectiveness of attention-based fusion and the promising outcomes of our proposed method when compared to multi-view NeRF and MPI techniques.

Plane-based multi-view inpainting for image-based rendering in large scenes

Single-Mask Inpainting for Voxel-Based Neural Radiance Fields

Scalable Image-Based Indoor Scene Rendering with Reflections

Interactive arbitrary-view rendering system using fast IBR and depth correction

Multi-view Inverse Rendering for Large-scale Real-world Indoor Scenes

MVInpainter: Learning Multi-View Consistent Inpainting to Bridge 2D and 3D Editing

Depth-Aided Inpainting for Disocclusion Restoration of Multi-View Images Using Depth-Image-Based Rendering

MAIR: Multi-view Attention Inverse Rendering with 3D Spatially-Varying Lighting Estimation

A Novel DIBR Method Based on Image Inpainting

iNVS: Repurposing Diffusion Inpainters for Novel View Synthesis

A New Method of DIBR Based on Background Inpainting

RIC: Rotate-Inpaint-Complete for Generalizable Scene Reconstruction

PhyIR: Physics-based Inverse Rendering for Panoramic Indoor Images

Multi-Plane Neural Radiance Fields for Novel View Synthesis

An IBR Method: Panoramic Mosaics of Slit Images with Depth

RefFusion: Reference Adapted Diffusion Models for 3D Scene Inpainting

DVI: Depth Guided Video Inpainting for Autonomous Driving

Layout-Guided Indoor Panorama Inpainting with Plane-Aware Normalization

SPIn-NeRF: Multiview Segmentation and Perceptual Inpainting with Neural Radiance Fields

Efficient View Synthesis and 3D-based Multi-Frame Denoising with Multiplane Feature Representations

Inverse Rendering for Complex Indoor Scenes: Shape, Spatially-Varying Lighting and SVBRDF from a Single Image