Abstract:Occupancy and 3D object detection are characterized as two standard tasks in modern autonomous driving system. In order to deploy them on a series of edge chips with better precision and time-consuming trade-off, contemporary approaches either deploy standalone models for individual tasks, or design a multi-task paradigm with separate heads. However, they might suffer from deployment difficulties (i.e., 3D convolution, transformer and so on) or deficiencies in task coordination. Instead, we argue that a favorable framework should be devised in pursuit of ease deployment on diverse chips and high precision with little time-consuming. Oriented at this, we revisit the paradigm for interaction between 3D object detection and occupancy prediction, reformulate the model with 2D convolution and prioritize the tasks such that each contributes to other. Thus, we propose a method to achieve fast 3D object detection and occupancy prediction (UltimateDO), wherein the light occupancy prediction head in FlashOcc is married to 3D object detection network, with negligible additional timeconsuming of only 1.1ms while facilitating each other. We instantiate UltimateDO on the challenging nuScenes-series benchmarks.

What problem does this paper attempt to address?

The problems that this paper attempts to solve are: how to efficiently implement occupancy prediction and 3D object detection in modern autonomous driving systems, and enable these two tasks to be easily deployed on various edge chips while maintaining high precision and low time consumption. ### Specific problems: 1. **Task integration and optimization**: - Existing methods either deploy models separately for each task or design multi - task paradigms but use independent head structures. These methods may encounter problems such as deployment difficulties or insufficient task coordination. - The paper proposes a new framework (UltimateDO), which combines a lightweight occupancy prediction head (from FlashOcc) with a 3D object detection network to achieve efficient occupancy prediction and 3D object detection. 2. **Computational resources and deployment challenges**: - Traditional operations such as 3D convolution and Transformer have problems of high computational complexity and large memory consumption when deployed on edge devices. - UltimateDO avoids the complex calculations of 3D convolution and Transformer by using 2D convolution and channel - to - height plug - ins, thereby reducing the computational cost and deployment difficulty. 3. **Mutual promotion of tasks**: - The interaction between occupancy prediction and 3D object detection can improve the overall performance. For example, fine - grained occupancy information can help more accurate 3D object detection, and instance - level 3D bounding boxes can enhance the accuracy of occupancy prediction. - UltimateDO enables the two tasks to promote each other by sharing the BEV feature map, further improving the overall perception performance. ### Solutions: - **Framework design**: The UltimateDO framework unifies occupancy prediction and 3D object detection by grafting the occupancy prediction head onto the BEV neck of BEVDet. This framework only increases the time consumption by 1.1 milliseconds but achieves state - of - the - art performance on the nuScenes dataset. - **Task coordination**: Through appropriate joint training of tasks, ensure that occupancy prediction and 3D object detection can complement each other and improve the overall perception accuracy. - **Computational efficiency**: Adopt 2D convolution and channel - to - height plug - ins to avoid the complex calculations of 3D convolution and Transformer, enabling the model to be efficiently deployed on various edge devices. In summary, this paper aims to solve the problems of efficient implementation and deployment of occupancy prediction and 3D object detection in autonomous driving systems through a novel framework design, and verifies its superiority in performance and efficiency through experiments.

UltimateDO: An Efficient Framework to Marry Occupancy Prediction with 3D Object Detection via Channel2height

Leveraging Front and Side Cues for Occlusion Handling in Monocular 3D Object Detection

FlashOcc: Fast and Memory-Efficient Occupancy Prediction via Channel-to-Height Plugin

AdaOcc: Adaptive-Resolution Occupancy Prediction

DAOcc: 3D Object Detection Assisted Multi-Sensor Fusion for 3D Occupancy Prediction

FastOcc: Accelerating 3D Occupancy Prediction by Fusing the 2D Bird's-Eye View and Perspective View

TEOcc: Radar-camera Multi-modal Occupancy Prediction via Temporal Enhancement

Towards Flexible 3D Perception: Object-Centric Occupancy Completion Augments 3D Object Detection

Real-Time 3D Occupancy Prediction via Geometric-Semantic Disentanglement

A Simple Framework for 3D Occupancy Estimation in Autonomous Driving

Deep Height Decoupling for Precise Vision-based 3D Occupancy Prediction

Occ3D: A Large-Scale 3D Occupancy Prediction Benchmark for Autonomous Driving

DETR3D: 3D Object Detection from Multi-view Images via 3D-to-2D Queries

6DoF-3D: Efficient and accurate 3D object detection using six degrees-of-freedom for autonomous driving

SurroundOcc: Multi-Camera 3D Occupancy Prediction for Autonomous Driving

OccFusion: Depth Estimation Free Multi-sensor Fusion for 3D Occupancy Prediction

OPUS: Occupancy Prediction Using a Sparse Set

Scene as Occupancy

OccupancyDETR: Using DETR for Mixed Dense-sparse 3D Occupancy Prediction