Abstract:OmDet, a novel language‐aware detector, designed to enhance open‐vocabulary and open‐world object detection through a continual learning approach and multi‐dataset vision‐language pre‐training is presented. By using natural language for knowledge representation, the authors successfully increase the "visual vocabularyˮ; and create a unified, language‐conditioned detection framework, which outperforms previous models on object detection and phrase grounding. This promising method proves the effectiveness of joint learning from multiple datasets and presents a path forward for scaling to even larger datasets. The advancement of object detection (OD) in open‐vocabulary and open‐world scenarios is a critical challenge in computer vision. OmDet, a novel language‐aware object detection architecture and an innovative training mechanism that harnesses continual learning and multi‐dataset vision‐language pre‐training is introduced. Leveraging natural language as a universal knowledge representation, OmDet accumulates "visual vocabularies" from diverse datasets, unifying the task as a language‐conditioned detection framework. The multimodal detection network (MDN) overcomes the challenges of multi‐dataset joint training and generalizes to numerous training datasets without manual label taxonomy merging. The authors demonstrate superior performance of OmDet over strong baselines in object detection in the wild, open‐vocabulary detection, and phrase grounding, achieving state‐of‐the‐art results. Ablation studies reveal the impact of scaling the pre‐training visual vocabulary, indicating a promising direction for further expansion to larger datasets. The effectiveness of our deep fusion approach is underscored by its ability to learn jointly from multiple datasets, enhancing performance through knowledge sharing.

Simple Multi-dataset Detection

Universal Object Detection with Large Vision Model

Uni$^2$Det: Unified and Universal Framework for Prompt-Guided Multi-dataset 3D Detection

UniDetector: Towards Universal Object Detection with Heterogeneous Supervision

Uni^2Det: Unified and Universal Framework for Prompt-Guided Multi-dataset 3D Detection

ScaleDet: A Scalable Multi-Dataset Object Detector

Uni3D: A Unified Baseline for Multi-dataset 3D Object Detection

Detecting Everything in the Open World: Towards Universal Object Detection

Learning a unified label space

Detection Hub: Unifying Object Detection Datasets via Query Adaptation on Language Embedding

Anno-incomplete Multi-dataset Detection

Online Pseudo-Label Unified Object Detection for Multiple Datasets Training

One for All: Multi-Domain Joint Training for Point Cloud Based 3D Object Detection

DaTaSeg: Taming a Universal Multi-Dataset Multi-Task Segmentation Model

Towards Unified 3D Object Detection via Algorithm and Data Unification

UniHead: Unifying Multi-Perception for Detection Heads

Language-aware Multiple Datasets Detection Pretraining for DETRs

Multi-Label and Evolvable Dataset Preparation for Web-Based Object Detection

Plain-Det: A Plain Multi-Dataset Object Detector

OmDet: Large‐scale vision‐language multi‐dataset pre‐training with multimodal detection network

Every Dataset Counts: Scaling up Monocular 3D Object Detection with Joint Datasets Training