Abstract:Recent advances in NeRF inpainting have leveraged pretrained diffusion models to enhance performance. However, these methods often yield suboptimal results due to their ineffective utilization of 2D diffusion priors. The limitations manifest in two critical aspects: the inadequate capture of geometric information by pretrained diffusion models and the suboptimal guidance provided by existing Score Distillation Sampling (SDS) methods. To address these problems, we introduce GB-NeRF, a novel framework that enhances NeRF inpainting through improved utilization of 2D diffusion priors. Our approach incorporates two key innovations: a fine-tuning strategy that simultaneously learns appearance and geometric priors and a specialized normal distillation loss that integrates these geometric priors into NeRF inpainting. We propose a technique called Balanced Score Distillation (BSD) that surpasses existing methods such as Score Distillation (SDS) and the improved version, Conditional Score Distillation (CSD). BSD offers improved inpainting quality in appearance and geometric aspects. Extensive experiments show that our method provides superior appearance fidelity and geometric consistency compared to existing approaches.

What problem does this paper attempt to address?

The problem that this paper attempts to solve is that the existing NeRF (Neural Radiance Field) inpainting methods have poor performance when using 2D diffusion model priors. Specifically, the existing methods are insufficient in geometric information capture and optimization guidance, resulting in the generated NeRF models having poor performance in geometric accuracy and appearance consistency. These problems are mainly reflected in: 1. **Insufficient geometric information capture**: Pretrained diffusion models cannot effectively capture geometric information, especially when dealing with normal maps. 2. **Inadequate optimization guidance**: The existing Score Distillation Sampling (SDS) and Conditional Score Distillation (CSD) methods, due to the presence of random noise and unconditional noise prediction terms, lead to unnecessary changes during the optimization process, affecting the quality of the inpainting area. To solve these problems, the authors propose the GB - NeRF framework, which improves the effect of NeRF inpainting through the following two key innovations: 1. **Fine - tuning strategy**: A special fine - tuning strategy is introduced, enabling the diffusion model to learn appearance and geometric priors simultaneously. By training with high - quality RGB images and corresponding normal map pairs, the geometric understanding ability of the model is enhanced. 2. **Balanced Score Distillation (BSD)**: A new optimization technique BSD is proposed. By eliminating high - variability terms, it provides a more stable and consistent supervision signal, thereby improving the optimization efficiency and reconstruction quality. These improvements enable GB - NeRF to provide higher geometric accuracy and appearance fidelity when dealing with occluded areas, thus achieving significant performance improvement in the NeRF inpainting task.

NeRF Inpainting with Geometric Diffusion Prior and Balanced Score Distillation

Single-Mask Inpainting for Voxel-Based Neural Radiance Fields

Learn to Optimize Denoising Scores for 3D Generation: A Unified and Improved Diffusion Prior on NeRF and 3D Gaussian Splatting

Taming Latent Diffusion Model for Neural Radiance Field Inpainting

Inpaint4DNeRF: Promptable Spatio-Temporal NeRF Inpainting with Generative Diffusion Models

NeRF-In: Free-Form NeRF Inpainting with RGB-D Priors

NeRF-In: Free-Form Inpainting for Pretrained NeRF With RGB-D Priors

IE-NeRF: Inpainting Enhanced Neural Radiance Fields in the Wild

Rethinking Score Distillation as a Bridge Between Image Distributions

Depth-guided NeRF Training via Earth Mover's Distance

Edit-DiffNeRF: Editing 3D Neural Radiance Fields using 2D Diffusion Model

DiffusioNeRF: Regularizing Neural Radiance Fields with Denoising Diffusion Models

GD^2-NeRF: Generative Detail Compensation via GAN and Diffusion for One-shot Generalizable Neural Radiance Fields

MVIP-NeRF: Multi-view 3D Inpainting on NeRF Scenes via Diffusion Prior

Enhancing Neural Radiance Fields with Depth and Normal Completion Priors from Sparse Views

NeRF-SDP: Efficient Generalizable Neural Radiance Field with Scene Depth Perception.

RePaint-NeRF: NeRF Editting via Semantic Masks and Diffusion Models

ED-NeRF: Efficient Text-Guided Editing of 3D Scene with Latent Space NeRF

SGCNeRF: Few-Shot Neural Rendering via Sparse Geometric Consistency Guidance

DiSR-NeRF: Diffusion-Guided View-Consistent Super-Resolution NeRF

MD-NeRF: Enhancing Large-Scale Scene Rendering and Synthesis With Hybrid Point Sampling and Adaptive Scene Decomposition