Abstract:Training perception systems for self-driving cars requires substantial annotations. However, manual labeling in 2D images is highly labor-intensive. While existing datasets provide rich annotations for pre-recorded sequences, they fall short in labeling rarely encountered viewpoints, potentially hampering the generalization ability for perception models. In this paper, we present PanopticNeRF-360, a novel approach that combines coarse 3D annotations with noisy 2D semantic cues to generate consistent panoptic labels and high-quality images from any viewpoint. Our key insight lies in exploiting the complementarity of 3D and 2D priors to mutually enhance geometry and semantics. Specifically, we propose to leverage noisy semantic and instance labels in both 3D and 2D spaces to guide geometry optimization. Simultaneously, the improved geometry assists in filtering noise present in the 3D and 2D annotations by merging them in 3D space via a learned semantic field. To further enhance appearance, we combine MLP and hash grids to yield hybrid scene features, striking a balance between high-frequency appearance and predominantly contiguous semantics. Our experiments demonstrate PanopticNeRF-360's state-of-the-art performance over existing label transfer methods on the challenging urban scenes of the KITTI-360 dataset. Moreover, PanopticNeRF-360 enables omnidirectional rendering of high-fidelity, multi-view and spatiotemporally consistent appearance, semantic and instance labels. We make our code and data available at <a class="link-external link-https" href="https://github.com/fuxiao0719/PanopticNeRF" rel="external noopener nofollow">this https URL</a>

LabelFusion: A Pipeline for Generating Ground Truth Labels for Real RGBD Data of Cluttered Scenes

EasyLabel: A Semi-Automatic Pixel-wise Object Annotation Tool for Creating Robotic RGB-D Datasets

NeuralLabeling: A versatile toolset for labeling vision datasets using Neural Radiance Fields

Rapid Pose Label Generation through Sparse Representation of Unknown Objects

A Pipeline for Segmenting and Structuring RGB-D Data for Robotics Applications

Chunkfusion: A Learning-Based RGB-D 3D Reconstruction Framework Via Chunk-Wise Integration

Labeling 3D scenes for Personal Assistant Robots

LABELMAKER: Automatic Semantic Label Generation from RGB-D Trajectories

LDLS: 3-D Object Segmentation Through Label Diffusion From 2-D Images

3DFusion, A real-time 3D object reconstruction pipeline based on streamed instance segmented data

FusionVision: A comprehensive approach of 3D object reconstruction and segmentation from RGB-D cameras using YOLO and fast segment anything

Scale Invariant Semantic Segmentation with RGB-D Fusion

Robotic picking in dense clutter via domain invariant learning from synthetic dense cluttered rendering

PanopticNeRF-360: Panoramic 3D-to-2D Label Transfer in Urban Scenes

Self-Supervised Learning of Object Segmentation from Unlabeled RGB-D Videos

Panoptic NeRF: 3D-to-2d Label Transfer for Panoptic Urban Scene Segmentation

Automated Multimodal Data Annotation via Calibration With Indoor Positioning System

3DSFLabelling: Boosting 3D Scene Flow Estimation by Pseudo Auto-labelling

All You Need is LUV: Unsupervised Collection of Labeled Images using Invisible UV Fluorescent Indicators

RGB-D Object Detection and Semantic Segmentation for Autonomous Manipulation in Clutter

Multimodal deep learning for robust RGB-D object recognition