Abstract:Neural Radiance Fields (NeRFs) have emerged as a popular approach for novel view synthesis. While NeRFs are quickly being adapted for a wider set of applications, intuitively editing NeRF scenes is still an open challenge. One important editing task is the removal of unwanted objects from a 3D scene, such that the replaced region is visually plausible and consistent with its context. We refer to this task as 3D inpainting. In 3D, solutions must be both consistent across multiple views and geometrically valid. In this paper, we propose a novel 3D inpainting method that addresses these challenges. Given a small set of posed images and sparse annotations in a single input image, our framework first rapidly obtains a 3D segmentation mask for a target object. Using the mask, a perceptual optimizationbased approach is then introduced that leverages learned 2D image inpainters, distilling their information into 3D space, while ensuring view consistency. We also address the lack of a diverse benchmark for evaluating 3D scene inpainting methods by introducing a dataset comprised of challenging real-world scenes. In particular, our dataset contains views of the same scene with and without a target object, enabling more principled benchmarking of the 3D inpainting task. We first demonstrate the superiority of our approach on multiview segmentation, comparing to NeRFbased methods and 2D segmentation approaches. We then evaluate on the task of 3D inpainting, establishing state-ofthe-art performance against other NeRF manipulation algorithms, as well as a strong 2D image inpainter baseline. Project Page: <a class="link-external link-https" href="https://spinnerf3d.github.io" rel="external noopener nofollow">this https URL</a>

3D Scene De-occlusion in Neural Radiance Fields: A Framework for Obstacle Removal and Realistic Inpainting

Single-Mask Inpainting for Voxel-Based Neural Radiance Fields

Three-Dimensional-Consistent Scene Inpainting via Uncertainty-Aware Neural Radiance Field

SPIn-NeRF: Multiview Segmentation and Perceptual Inpainting with Neural Radiance Fields

Occlusion-Free Scene Recovery Via Neural Radiance Fields

NeRF-In: Free-Form NeRF Inpainting with RGB-D Priors

OR-NeRF: Object Removing from 3D Scenes Guided by Multiview Segmentation with Neural Radiance Fields

IE-NeRF: Inpainting Enhanced Neural Radiance Fields in the Wild

Removing Objects From Neural Radiance Fields

DM-NeRF: 3D Scene Geometry Decomposition and Manipulation from 2D Images

Clean-NeRF: Reformulating NeRF to account for View-Dependent Observations

SN 2 eRF: A Framework for Neural Radiance Fields given Sparse and Noisy Poses

Enhancing Neural Radiance Fields with Depth and Normal Completion Priors from Sparse Views

Semantic-aware Occlusion Filtering Neural Radiance Fields in the Wild

Drone-NeRF: Efficient NeRF based 3D scene reconstruction for large-scale drone survey

An Exploration of Neural Radiance Field Scene Reconstruction: Synthetic, Real-world and Dynamic Scenes

NeRF-In: Free-Form Inpainting for Pretrained NeRF With RGB-D Priors

PERF: Panoramic Neural Radiance Field From a Single Panorama

IOVS4NeRF:Incremental Optimal View Selection for Large-Scale NeRFs

ReconFusion: 3D Reconstruction with Diffusion Priors

Inpaint4DNeRF: Promptable Spatio-Temporal NeRF Inpainting with Generative Diffusion Models