Abstract:While 3D object bounding box (bbox) representation has been widely used in autonomous driving perception, it lacks the ability to capture the precise details of an object's intrinsic geometry. Recently, occupancy has emerged as a promising alternative for 3D scene perception. However, constructing a high-resolution occupancy map remains infeasible for large scenes due to computational constraints. Recognizing that foreground objects only occupy a small portion of the scene, we introduce object-centric occupancy as a supplement to object bboxes. This representation not only provides intricate details for detected objects but also enables higher voxel resolution in practical applications. We advance the development of object-centric occupancy perception from both data and algorithm perspectives. On the data side, we construct the first object-centric occupancy dataset from scratch using an automated pipeline. From the algorithmic standpoint, we introduce a novel object-centric occupancy completion network equipped with an implicit shape decoder that manages dynamic-size occupancy generation. This network accurately predicts the complete object-centric occupancy volume for inaccurate object proposals by leveraging temporal information from long sequences. Our method demonstrates robust performance in completing object shapes under noisy detection and tracking conditions. Additionally, we show that our occupancy features significantly enhance the detection results of state-of-the-art 3D object detectors, especially for incomplete or distant objects in the Waymo Open Dataset.

What problem does this paper attempt to address?

The problem that this paper attempts to solve is: in autonomous driving perception, the existing 3D object bounding box (3D bounding box, bbox) representation methods cannot accurately capture the intrinsic geometric details of objects, especially when dealing with objects with irregular shapes. To overcome this limitation, the author introduced object - centric occupancy as a supplement to 3D object detection. This method not only provides fine - grained details of detected objects but also enables a higher voxel resolution in practical applications. Specifically, the main contributions of the paper include: 1. **Proposing object - centric occupancy representation**: Different from the traditional scene - level occupancy representation, this representation focuses on foreground objects, allowing the use of higher - resolution voxels in large - scale scenes, thereby more accurately describing complex object structures. 2. **Constructing the first object - centric occupancy dataset**: Built from scratch through an automated pipeline, it addresses the deficiencies of existing datasets in coordinate system alignment and resolution. 3. **Proposing a sequence - based occupancy completion network**: This network utilizes an implicit shape decoder to generate occupancy of dynamic sizes and aggregates historical observation information through an attention mechanism, effectively handling detection drift and accurately predicting the complete object - centric occupancy volume. 4. **Enhancing 3D object detection performance**: Experimental results show that this method significantly improves the performance of existing 3D object detectors on the Waymo Open Dataset, especially when dealing with incomplete or distant objects. ### Main Technical Details of the Paper #### 1. Dynamic - Size Occupancy Generation - **Implicit Decoder**: Use an implicit shape decoder \( D \), which predicts the occupancy probability according to the latent embedding \( z \) and the query position \( q \): \[ p = D(z, q) \] where \( D: \mathbb{R}^e\times\mathbb{R}^3\rightarrow[0, 1] \) is an MLP, and \( z \) is a fixed - length embedding that describes the geometric structure within the ROI. #### 2. Two - Branch ROI Encoding - **Local Encoder \( E_{\text{local}} \)**: Transform the ROI into a local coordinate system for encoding, preserving the local geometric structure. - **Global Encoder \( E_{\text{global}} \)**: Directly encode the ROI in the global coordinate system, preserving the global motion information. #### 3. Feature Enhancement through Temporal Aggregation - **Causal Attention Mechanism**: Use a causal transformer \( \text{CausalAttn} \) to aggregate temporal information and prevent future information leakage: \[ Z_g'=\text{CausalAttn}(Z_g + \gamma(T)+\phi(B)) \] where \( \gamma(\cdot) \) is the position encoding of the timestamp, and \( \phi(\cdot) \) is an MLP that encodes global bbox information. - **Feature Fusion**: Fuse the enhanced global feature \( Z_g' \) with the local feature \( Z_l \) to obtain the final latent embedding \( Z \): \[ Z = \text{MLP}(\text{Concat}(Z_l, Z_g')) \] #### 4. Occupancy Completion and Detection Optimization - **Occupancy Completion**: Predict the complete object - centric occupancy volume for each proposal by querying the implicit decoder \( D \) at different positions. - **Detection Optimization**: Fuse the final latent embedding \( Z \) with the global ROI feature \( Z_g \) for bounding box and score optimization: \[ Z_{\text{det}}=\text{MLP}(\text{Concat}(Z, Z_g)) \] #### 5. Loss Function - **Overall Training Loss**: \[ L = L_{\text

Towards Flexible 3D Perception: Object-Centric Occupancy Completion Augments 3D Object Detection

Leveraging Front and Side Cues for Occlusion Handling in Monocular 3D Object Detection

Learning Occupancy for Monocular 3D Object Detection

Occ3D: A Large-Scale 3D Occupancy Prediction Benchmark for Autonomous Driving

SurroundOcc: Multi-Camera 3D Occupancy Prediction for Autonomous Driving

3Dopformer: 3D Occupancy Perception from Multi-Camera Images with Directional and Distance Enhancement

GEOcc: Geometrically Enhanced 3D Occupancy Network with Implicit-Explicit Depth Fusion and Contextual Self-Supervision

DAOcc: 3D Object Detection Assisted Multi-Sensor Fusion for 3D Occupancy Prediction

AdaOcc: Adaptive-Resolution Occupancy Prediction

Robust 3D Semantic Occupancy Prediction with Calibration-free Spatial Transformation

Scene as Occupancy

LiDAR-based 4D Occupancy Completion and Forecasting

OpenOccupancy: A Large Scale Benchmark for Surrounding Semantic Occupancy Perception

AdaptiveOcc: Adaptive Octree-based Network for Multi-Camera 3D Semantic Occupancy Prediction in Autonomous Driving

A Simple Framework for 3D Occupancy Estimation in Autonomous Driving

Real-Time 3D Occupancy Prediction via Geometric-Semantic Disentanglement

A Survey on Occupancy Perception for Autonomous Driving: The Information Fusion Perspective

OCC-VO: Dense Mapping via 3D Occupancy-Based Visual Odometry for Autonomous Driving