Abstract:Neural volumetric representations have shown the potential that Multi-layer Perceptrons (MLPs) can be optimized with multi-view calibrated images to represent scene geometry and appearance, without explicit 3D supervision. Object segmentation can enrich many downstream applications based on the learned radiance field. However, introducing hand-crafted segmentation to define regions of interest in a complex real-world scene is non-trivial and expensive as it acquires per view annotation. This paper carries out the exploration of self-supervised learning for object segmentation using NeRF for complex real-world scenes. Our framework, called NeRF with Self-supervised Object Segmentation NeRF-SOS, couples object segmentation and neural radiance field to segment objects in any view within a scene. By proposing a novel collaborative contrastive loss in both appearance and geometry levels, NeRF-SOS encourages NeRF models to distill compact geometry-aware segmentation clusters from their density fields and the self-supervised pre-trained 2D visual features. The self-supervised object segmentation framework can be applied to various NeRF models that both lead to photo-realistic rendering results and convincing segmentation maps for both indoor and outdoor scenarios. Extensive results on the LLFF, Tank & Temple, and BlendedMVS datasets validate the effectiveness of NeRF-SOS. It consistently surpasses other 2D-based self-supervised baselines and predicts finer semantics masks than existing supervised counterparts. Please refer to the video on our project page for more details:<a class="link-external link-https" href="https://zhiwenfan.github.io/NeRF-SOS" rel="external noopener nofollow">this https URL</a>.

Geometry Aware Field-to-field Transformations for 3D Semantic Segmentation

SegNeRF: 3D Part Segmentation with Neural Radiance Fields

Rethinking Open-Vocabulary Segmentation of Radiance Fields in 3D Space

DiscoNeRF: Class-Agnostic Object Field for 3D Object Discovery

NeRF-SOS: Any-View Self-supervised Object Segmentation on Complex Scenes

ONeRF: Unsupervised 3D Object Segmentation from Multiple Views

View-Consistent Hierarchical 3D Segmentation Using Ultrametric Feature Fields

DM-NeRF: 3D Scene Geometry Decomposition and Manipulation from 2D Images

GSNeRF: Generalizable Semantic Neural Radiance Fields with Enhanced 3D Scene Understanding

Sem2NeRF: Converting Single-View Semantic Masks to Neural Radiance Fields

Fast and Efficient: Mask Neural Fields for 3D Scene Segmentation

Unsupervised Multi-View Object Segmentation Using Radiance Field Propagation

Semantic Segmentation of Point Cloud With Novel Neural Radiation Field Convolution

Instance Neural Radiance Field

Points2NeRF: Generating Neural Radiance Fields from 3D point cloud

3DSAM: Segment Anything in NeRF

DatasetNeRF: Efficient 3D-aware Data Factory with Generative Radiance Fields

Nerflets: Local Radiance Fields for Efficient Structure-Aware 3D Scene Representation from 2D Supervision

Depth-supervised NeRF: Fewer Views and Faster Training for Free

SG-NeRF: Semantic-guided Point-based Neural Radiance Fields

Three-Dimensional Object Segmentation Method based on YOLO, SAM, and NeRF