Abstract:Neural Radiance Fields (NeRFs) have emerged as a popular approach for novel view synthesis. While NeRFs are quickly being adapted for a wider set of applications, intuitively editing NeRF scenes is still an open challenge. One important editing task is the removal of unwanted objects from a 3D scene, such that the replaced region is visually plausible and consistent with its context. We refer to this task as 3D inpainting. In 3D, solutions must be both consistent across multiple views and geometrically valid. In this paper, we propose a novel 3D inpainting method that addresses these challenges. Given a small set of posed images and sparse annotations in a single input image, our framework first rapidly obtains a 3D segmentation mask for a target object. Using the mask, a perceptual optimizationbased approach is then introduced that leverages learned 2D image inpainters, distilling their information into 3D space, while ensuring view consistency. We also address the lack of a diverse benchmark for evaluating 3D scene inpainting methods by introducing a dataset comprised of challenging real-world scenes. In particular, our dataset contains views of the same scene with and without a target object, enabling more principled benchmarking of the 3D inpainting task. We first demonstrate the superiority of our approach on multiview segmentation, comparing to NeRFbased methods and 2D segmentation approaches. We then evaluate on the task of 3D inpainting, establishing state-ofthe-art performance against other NeRF manipulation algorithms, as well as a strong 2D image inpainter baseline. Project Page: <a class="link-external link-https" href="https://spinnerf3d.github.io" rel="external noopener nofollow">this https URL</a>

Three-Dimensional-Consistent Scene Inpainting via Uncertainty-Aware Neural Radiance Field

Single-Mask Inpainting for Voxel-Based Neural Radiance Fields

SPIn-NeRF: Multiview Segmentation and Perceptual Inpainting with Neural Radiance Fields

MVIP-NeRF: Multi-view 3D Inpainting on NeRF Scenes via Diffusion Prior

Inpaint4DNeRF: Promptable Spatio-Temporal NeRF Inpainting with Generative Diffusion Models

NeRF-In: Free-Form NeRF Inpainting with RGB-D Priors

Inpaint3D: 3D Scene Content Generation using 2D Inpainting Diffusion

RefFusion: Reference Adapted Diffusion Models for 3D Scene Inpainting

Taming Latent Diffusion Model for Neural Radiance Field Inpainting

IE-NeRF: Inpainting Enhanced Neural Radiance Fields in the Wild

Estimating 3D Uncertainty Field: Quantifying Uncertainty for Neural Radiance Fields

3D-Consistent Image Inpainting with Diffusion Models

NeRF-In: Free-Form Inpainting for Pretrained NeRF With RGB-D Priors

ConsistentNeRF: Enhancing Neural Radiance Fields with 3D Consistency for Sparse View Synthesis

DerainNeRF: 3D Scene Estimation with Adhesive Waterdrop Removal

Enhancing Neural Radiance Fields with Depth and Normal Completion Priors from Sparse Views

OR-NeRF: Object Removing from 3D Scenes Guided by Multiview Segmentation with Neural Radiance Fields

RePaint-NeRF: NeRF Editting via Semantic Masks and Diffusion Models

Removing Objects From Neural Radiance Fields

Clean-NeRF: Reformulating NeRF to account for View-Dependent Observations

Diverse Image Inpainting with Disentangled Uncertainty