Abstract:Neural Radiance Fields (NeRF) are widely used for novel-view synthesis and have been adapted for 3D Object Detection (3DOD), offering a promising approach to 3DOD through view-synthesis representation. However, NeRF faces inherent limitations: (i) limited representational capacity for 3DOD due to its implicit nature, and (ii) slow rendering speeds. Recently, 3D Gaussian Splatting (3DGS) has emerged as an explicit 3D representation that addresses these limitations. Inspired by these advantages, this paper introduces 3DGS into 3DOD for the first time, identifying two main challenges: (i) Ambiguous spatial distribution of Gaussian blobs: 3DGS primarily relies on 2D pixel-level supervision, resulting in unclear 3D spatial distribution of Gaussian blobs and poor differentiation between objects and background, which hinders 3DOD; (ii) Excessive background blobs: 2D images often include numerous background pixels, leading to densely reconstructed 3DGS with many noisy Gaussian blobs representing the background, negatively affecting detection. To tackle the challenge (i), we leverage the fact that 3DGS reconstruction is derived from 2D images, and propose an elegant and efficient solution by incorporating 2D Boundary Guidance to significantly enhance the spatial distribution of Gaussian blobs, resulting in clearer differentiation between objects and their background. To address the challenge (ii), we propose a Box-Focused Sampling strategy using 2D boxes to generate object probability distribution in 3D spaces, allowing effective probabilistic sampling in 3D to retain more object blobs and reduce noisy background blobs. Benefiting from our designs, our 3DGS-DET significantly outperforms the SOTA NeRF-based method, NeRF-Det, achieving improvements of +6.6 on mAP@0.25 and +8.1 on mAP@0.5 for the ScanNet dataset, and impressive +31.5 on mAP@0.25 for the ARKITScenes dataset.

NeRF-DetS: Enhanced Adaptive Spatial-wise Sampling and View-wise Fusion Strategies for NeRF-based Indoor Multi-view 3D Object Detection

NeRF-Det++: Incorporating Semantic Cues and Perspective-aware Depth Supervision for Indoor Multi-View 3D Detection

NeRF-Det: Learning Geometry-Aware Volumetric Representation for Multi-View 3D Object Detection

3D-SSD: Learning Hierarchical Features from RGB-D Images for Amodal 3D Object Detection

3DGS-DET: Empower 3D Gaussian Splatting with Boundary Guidance and Box-Focused Sampling for 3D Object Detection

MonoNeRD: NeRF-like Representations for Monocular 3D Object Detection

DaRF: Boosting Radiance Fields from Sparse Inputs with Monocular Depth Adaptation

StructNeRF: Neural Radiance Fields for Indoor Scenes With Structural Hints

DETR4D: Direct Multi-View 3D Object Detection with Sparse Attention

MVSDet: Multi-View Indoor 3D Object Detection via Efficient Plane Sweeps

DETR3D: 3D Object Detection from Multi-view Images via 3D-to-2D Queries

SAID-NeRF: Segmentation-AIDed NeRF for Depth Completion of Transparent Objects

UniDet3D: Multi-dataset Indoor 3D Object Detection

Three-Dimensional Reconstruction of Indoor Scenes Based on Implicit Neural Representation

NeRFuser: Large-Scale Scene Representation by NeRF Fusion

DSEM-NeRF: Multimodal feature fusion and global-local attention for enhanced 3D scene reconstruction

NeSLAM: Neural Implicit Mapping and Self-Supervised Feature Tracking With Depth Completion and Denoising

MMAF-Net: Multi-view multi-stage adaptive fusion for multi-sensor 3D object detection

Spatial and Semantic Information Enhancement for Indoor 3D Object Detection

SparseDet: A Simple and Effective Framework for Fully Sparse LiDAR-based 3D Object Detection

Suppress-and-Refine Framework for End-to-End 3D Object Detection