Abstract:In this paper, we propose a weakly-supervised approach for 3D object detection, which makes it possible to train a strong 3D detector with position-level annotations (i.e. annotations of object centers and categories). In order to remedy the information loss from box annotations to centers, our method makes use of synthetic 3D shapes to convert the position-level annotations into virtual scenes with box-level annotations, and in turn utilizes the fully-annotated virtual scenes to complement the real labels. Specifically, we first present a shape-guided label-enhancement method, which assembles 3D shapes into physically reasonable virtual scenes according to the coarse scene layout extracted from position-level annotations. Then we transfer the information contained in the virtual scenes back to real ones by applying a virtual-to-real domain adaptation method, which refines the annotated object centers and additionally supervises the training of detector with the virtual scenes. Since the shape-guided label enhancement method generates virtual scenes by human-heuristic physical constraints, the layout of the fixed virtual scenes may be unreasonable with varied object combinations. To address this, we further present differentiable label enhancement to optimize the virtual scenes including object scales, orientations and locations in a data-driven manner. Moreover, we further propose a label-assisted self-training strategy to fully exploit the capability of detector. By reusing the position-level annotations and virtual scenes, we fuse the information from both domains and generate box-level pseudo labels on the real scenes, which enables us to directly train a detector in fully-supervised manner. Extensive experiments on the widely used ScanNet and Matterport3D datasets show that our approach surpasses current weakly-supervised and semi-supervised methods by a large margin, and achieves comparable detection performance with some popular fully-supervised methods with less than 5% of the labeling labor.

Effective 3D Object Detection Based on Detector and Tracker.

3D-SSD: Learning Hierarchical Features from RGB-D Images for Amodal 3D Object Detection

Recent advances in 3D object detection based on RGB-D: A survey

Reinforcing Lidar-Based 3d Object Detection With Rgb And 3d Information

Stereo RGB and Deeper LIDAR Based Network for 3D Object Detection

Stereo RGB and Deeper LIDAR-Based Network for 3D Object Detection in Autonomous Driving.

Accurate Monocular Object Detection via Color-Embedded 3D Reconstruction for Autonomous Driving

Cross-Modal 3D Object Detection and Tracking for Auto-Driving

An End-to-End Deep Learning Network for 3D Object Detection From RGB-D Data Based on Hough Voting

Training an Open-Vocabulary Monocular 3D Object Detection Model without 3D Data

Ret3D: Rethinking Object Relations for Efficient 3D Object Detection in Driving Scenes

A survey of Object Classification and Detection based on 2D/3D data

MVM3Det: A Novel Method for Multi-view Monocular 3D Detection

Back to Reality: Learning Data-Efficient 3D Object Detector with Shape Guidance.

Object2Scene: Putting Objects in Context for Open-Vocabulary 3D Detection

RGBD Object Tracking: An In-depth Review

Aug3D-RPN: Improving Monocular 3D Object Detection by Synthetic Images with Virtual Depth

Towards Open-set Camera 3D Object Detection

VKP-P3D: Real-Time Monocular Pseudo 3D Object Detection Based on Visible Key Points and Camera Geometry

Multi-Object Tracking Algorithm for RGB-D Images Based on Asymmetric Dual Siamese Networks

Suppress-and-Refine Framework for End-to-End 3D Object Detection