Abstract:Despite the emergence of successful NeRF inpainting methods built upon explicit RGB and depth 2D inpainting supervisions, these methods are inherently constrained by the capabilities of their underlying 2D inpainters. This is due to two key reasons: (i) independently inpainting constituent images results in view-inconsistent imagery, and (ii) 2D inpainters struggle to ensure high-quality geometry completion and alignment with inpainted RGB images.

What problem does this paper attempt to address?

The problem that this paper attempts to solve is multi - view 3D inpainting in Neural Radiance Field (NeRF) scenes. Specifically, when dealing with NeRF scene inpainting, existing methods mainly rely on explicit RGB and depth inpainting priors. These priors usually inpaint each view independently through 2D image inpainters and then attempt to solve multi - view inconsistency. However, this method has several limitations: 1. **View Inconsistency**: Independently inpainting each constituent image will lead to view - inconsistent images, especially when the change in viewing angles is significant. 2. **Geometric Completeness and Alignment Problems**: 2D image inpainters have difficulty ensuring high - quality geometric completion and alignment with the inpainted RGB images. To overcome these problems, the paper proposes a new method - MVIP - NeRF (Multi - view 3D Inpainting on NeRF Scenes via Diffusion Prior), which utilizes the potential of diffusion models to achieve inpainting of NeRF scenes while focusing on both appearance and geometric aspects. The main features of MVIP - NeRF include: - **Joint Multi - view Inpainting**: Through an iterative optimization process based on Score Distillation Sampling (SDS), joint inpainting is carried out on multiple views to achieve a consistent solution. - **Geometric Representation**: In addition to restoring the rendered RGB images, normal maps are also extracted as geometric representations, and a normal SDS loss is defined to encourage accurate geometric inpainting and alignment with the appearance. - **Multi - view Score Function**: To handle large changes in viewing angles, a multi - view score function is proposed to ensure that generative priors are simultaneously extracted from different - view images, thereby ensuring consistent visual completion when dealing with large - angle - of - view changes. Through these innovations, MVIP - NeRF can achieve more consistent and more realistic inpainting results under multi - view conditions without explicit RGB or depth supervision.

MVIP-NeRF: Multi-view 3D Inpainting on NeRF Scenes via Diffusion Prior

Single-Mask Inpainting for Voxel-Based Neural Radiance Fields

MPS-NeRF: Generalizable 3D Human Rendering from Multiview Images

Three-Dimensional-Consistent Scene Inpainting via Uncertainty-Aware Neural Radiance Field

Inpaint4DNeRF: Promptable Spatio-Temporal NeRF Inpainting with Generative Diffusion Models

NeRF-In: Free-Form NeRF Inpainting with RGB-D Priors

Inpaint3D: 3D Scene Content Generation using 2D Inpainting Diffusion

SPIn-NeRF: Multiview Segmentation and Perceptual Inpainting with Neural Radiance Fields

3D Scene De-occlusion in Neural Radiance Fields: A Framework for Obstacle Removal and Realistic Inpainting

MVInpainter: Learning Multi-View Consistent Inpainting to Bridge 2D and 3D Editing

IE-NeRF: Inpainting Enhanced Neural Radiance Fields in the Wild

PERF: Panoramic Neural Radiance Field From a Single Panorama

Taming Latent Diffusion Model for Neural Radiance Field Inpainting

Multi-View Stereo and Depth Priors Guided NeRF for View Synthesis.

HDPNERF: Hybrid Depth Priors for Neural Radiance Fields from Sparse Input Views

ExtraNeRF: Visibility-Aware View Extrapolation of Neural Radiance Fields with Diffusion Models

NeRF-In: Free-Form Inpainting for Pretrained NeRF With RGB-D Priors

NeRFail: Neural Radiance Fields-Based Multiview Adversarial Attack

Prior-Driven NeRF: Prior Guided Rendering

Enhancing Neural Radiance Fields with Depth and Normal Completion Priors from Sparse Views