Abstract:3D semantic occupancy prediction, which seeks to provide accurate and comprehensive representations of environment scenes, is important to autonomous driving systems. For autonomous cars equipped with multi-camera and LiDAR, it is critical to aggregate multi-sensor information into a unified 3D space for accurate and robust predictions. Recent methods are mainly built on the 2D-to-3D transformation that relies on sensor calibration to project the 2D image information into the 3D space. These methods, however, suffer from two major limitations: First, they rely on accurate sensor calibration and are sensitive to the calibration noise, which limits their application in real complex environments. Second, the spatial transformation layers are computationally expensive and limit their running on an autonomous vehicle. In this work, we attempt to exploit a Robust and Efficient 3D semantic Occupancy (REO) prediction scheme. To this end, we propose a calibration-free spatial transformation based on vanilla attention to implicitly model the spatial correspondence. In this way, we robustly project the 2D features to a predefined BEV plane without using sensor calibration as input. Then, we introduce 2D and 3D auxiliary training tasks to enhance the discrimination power of 2D backbones on spatial, semantic, and texture features. Last, we propose a query-based prediction scheme to efficiently generate large-scale fine-grained occupancy predictions. By fusing point clouds that provide complementary spatial information, our REO surpasses the existing methods by a large margin on three benchmarks, including OpenOccupancy, Occ3D-nuScenes, and SemanticKITTI Scene Completion. For instance, our REO achieves 19.8× speedup compared to Co-Occ, with 1.1 improvements in geometry IoU on OpenOccupancy. Our code will be available at https://github.com/ICEORY/REO.

LowRankOcc: Tensor Decomposition and Low-Rank Recovery for Vision-based 3D Semantic Occupancy Prediction

SparseOcc: Rethinking Sparse Latent Representation for Vision-Based Semantic Occupancy Prediction

Up-to-Down Network: Fusing Multi-Scale Context for 3D Semantic Scene Completion

Fully Sparse 3D Occupancy Prediction

OccRWKV: Rethinking Efficient 3D Semantic Occupancy Prediction with Linear Complexity

SelfOcc: Self-Supervised Vision-Based 3D Occupancy Prediction

OccFormer: Dual-path Transformer for Vision-based 3D Semantic Occupancy Prediction

OPUS: Occupancy Prediction Using a Sparse Set

Tensor Low-Rank Reconstruction for Semantic Segmentation

Deep Height Decoupling for Precise Vision-based 3D Occupancy Prediction

HybridOcc: NeRF Enhanced Transformer-based Multi-Camera 3D Occupancy Prediction

Robust 3D Semantic Occupancy Prediction with Calibration-free Spatial Transformation

ViPOcc: Leveraging Visual Priors from Vision Foundation Models for Single-View 3D Occupancy Prediction

MR-Occ: Efficient Camera-LiDAR 3D Semantic Occupancy Prediction Using Hierarchical Multi-Resolution Voxel Representation

LinkOcc: 3D Semantic Occupancy Prediction with Temporal Association

Tri-Perspective View for Vision-Based 3D Semantic Occupancy Prediction

MonoOcc: Digging into Monocular Semantic Occupancy Prediction

OccupancyDETR: Using DETR for Mixed Dense-sparse 3D Occupancy Prediction

OccLoff: Learning Optimized Feature Fusion for 3D Occupancy Prediction

Real-Time 3D Occupancy Prediction via Geometric-Semantic Disentanglement

GaussianFormer: Scene as Gaussians for Vision-Based 3D Semantic Occupancy Prediction