Abstract:Neural Radiance Field (NeRF) has been widely recognized for its excellence in novel view synthesis and 3D scene reconstruction. However, their effectiveness is inherently tied to the assumption of static scenes, rendering them susceptible to undesirable artifacts when confronted with transient distractors such as moving objects or shadows. In this work, we propose a novel paradigm, namely "Heuristics-Guided Segmentation" (HuGS), which significantly enhances the separation of static scenes from transient distractors by harmoniously combining the strengths of hand-crafted heuristics and state-of-the-art segmentation models, thus significantly transcending the limitations of previous solutions. Furthermore, we delve into the meticulous design of heuristics, introducing a seamless fusion of Structure-from-Motion (SfM)-based heuristics and color residual heuristics, catering to a diverse range of texture profiles. Extensive experiments demonstrate the superiority and robustness of our method in mitigating transient distractors for NeRFs trained in non-static scenes. Project page:

What problem does this paper attempt to address?

The paper primarily addresses the challenges faced by Neural Radiance Fields (NeRF) in non-static scenes, particularly how to effectively handle transient distractors (such as moving people or vehicles) that cause unwanted artifacts in the reconstructed 3D geometry. The proposed solution is a new paradigm called "Heuristic-Guided Segmentation" (HuGS), aimed at improving NeRF's performance in non-static scenes. Specifically, HuGS combines the advantages of hand-crafted heuristic rules and advanced segmentation models to achieve precise separation between static scenes and transient distractors. The core contributions of the HuGS method include: 1. **Proposing the Heuristic-Guided Segmentation (HuGS) paradigm**: This method leverages hand-crafted heuristic rules to provide rough indications of static elements and uses modern segmentation models (such as the Segment Anything Model, SAM) to generate precise object boundaries, resulting in high-quality static maps. This approach avoids the limitations of relying solely on a single method and instead improves accuracy by combining the strengths of both. 2. **In-depth exploration of heuristic design**: The paper proposes two complementary heuristic methods—Structure from Motion (SfM)-based heuristics and color residual-based heuristics. SfM heuristics effectively identify static objects with high-frequency texture patterns, while color residual heuristics excel at detecting static elements with low-frequency textures. The combination of these two heuristics allows the method to cover a wide range of different texture types of static scene elements, enhancing the ability to handle transient distractors. 3. **Experimental results demonstration**: Extensive experiments on multiple datasets demonstrate that the HuGS method significantly improves NeRF's performance in non-static scenes, particularly in eliminating transient distractors. The experiments not only showcase the superiority and robustness of the method but also indicate that high-quality static and transient separation results can be achieved even when applied to partially trained models. In summary, the paper addresses the key issue of handling transient distractors in non-static scenes for NeRF. By proposing the HuGS paradigm and detailed heuristic design, it significantly enhances NeRF's performance in practical application scenarios.

NeRF-HuGS: Improved Neural Radiance Fields in Non-static Scenes Using Heuristics-Guided Segmentation

HG3-NeRF: Hierarchical Geometric, Semantic, and Photometric Guided Neural Radiance Fields for Sparse View Inputs

GSNeRF: Generalizable Semantic Neural Radiance Fields with Enhanced 3D Scene Understanding

An Exploration of Neural Radiance Field Scene Reconstruction: Synthetic, Real-world and Dynamic Scenes

Global-guided Focal Neural Radiance Field for Large-scale Scene Rendering

NeRF synthesis with shading guidance

Enhance-NeRF: Multiple Performance Evaluation for Neural Radiance Fields

IE-NeRF: Inpainting Enhanced Neural Radiance Fields in the Wild

Fast High Dynamic Range Radiance Fields for Dynamic Scenes

NeRF On-the-go: Exploiting Uncertainty for Distractor-free NeRFs in the Wild

NeRF-DS: Neural Radiance Fields for Dynamic Specular Objects

SN 2 eRF: A Framework for Neural Radiance Fields given Sparse and Noisy Poses

AG-NeRF: Attention-guided Neural Radiance Fields for Multi-height Large-scale Outdoor Scene Rendering

NeRFReN: Neural Radiance Fields with Reflections

SG-NeRF: Neural Surface Reconstruction with Scene Graph Optimization

NeRF-SOS: Any-View Self-supervised Object Segmentation on Complex Scenes

SG-NeRF: Semantic-guided Point-based Neural Radiance Fields

NeRF-SDP: Efficient Generalizable Neural Radiance Field with Scene Depth Perception.

NeRF-SR: High Quality Neural Radiance Fields Using Supersampling

S-NeRF: Neural Radiance Fields for Street Views

Enhancing Neural Radiance Fields with Adaptive Multi-Exposure Fusion: A Bilevel Optimization Approach for Novel View Synthesis