Abstract:Realizing unified 3D object detection, including both indoor and outdoor scenes, holds great importance in applications like robot navigation. However, involving various scenarios of data to train models poses challenges due to their significantly distinct characteristics, \eg, diverse geometry properties and heterogeneous domain distributions. In this work, we propose to address the challenges from two perspectives, the algorithm perspective and data perspective. In terms of the algorithm perspective, we first build a monocular 3D object detector based on the bird's-eye-view (BEV) detection paradigm, where the explicit feature projection is beneficial to addressing the geometry learning ambiguity. In this detector, we split the classical BEV detection architecture into two stages and propose an uneven BEV grid design to handle the convergence instability caused by geometry difference between scenarios. Besides, we develop a sparse BEV feature projection strategy to reduce the computational cost and a unified domain alignment method to handle heterogeneous domains. From the data perspective, we propose to incorporate depth information to improve training robustness. Specifically, we build the first unified multi-modal 3D object detection benchmark MM-Omni3D and extend the aforementioned monocular detector to its multi-modal version, which is the first unified multi-modal 3D object detector. We name the designed monocular and multi-modal detectors as UniMODE and MM-UniMODE, respectively. The experimental results reveal several insightful findings highlighting the benefits of multi-modal data and confirm the effectiveness of all the proposed strategies.

Learning a unified label space

Universal Object Detection with Large Vision Model

Dual Collaborative Visual-Semantic Mapping for Multi-Label Zero-Shot Image Recognition

Simple Multi-dataset Detection

Label Name is Mantra: Unifying Point Cloud Segmentation across Heterogeneous Datasets

Online Pseudo-Label Unified Object Detection for Multiple Datasets Training

Labeling Indoor Scenes with Fusion of Out-of-the-Box Perception Models

UniDetector: Towards Universal Object Detection with Heterogeneous Supervision

Joint Input and Output Space Learning for Multi-Label Image Classification

Supervised High-Level Feature Learning with Label Consistencies for Object Recognition.

Towards Unified 3D Object Detection via Algorithm and Data Unification

Learning Spatial Prior With Automatically Labeled Landmarks

Detecting Everything in the Open World: Towards Universal Object Detection

Automatic image annotation via local multi-label classification

MULTI-LABEL IMAGE RECOGNITION WITH JOINT CLASS-AWARE MAP DISENTANGLING AND LABEL CORRELATION EMBEDDING

Mdalu: Multi-Source Domain Adaptation and Label Unification with Partial Datasets

Exploiting Unlabeled Data with Vision and Language Models for Object Detection

Automated Label Unification for Multi-Dataset Semantic Segmentation with GNNs

Spatial Context-Aware Object-Attentional Network for Multi-Label Image Classification

KTCN: Enhancing Open-World Object Detection with Knowledge Transfer and Class-Awareness Neutralization

Recurrent Attentional Reinforcement Learning for Multi-label Image Recognition