Abstract:OmDet, a novel language‐aware detector, designed to enhance open‐vocabulary and open‐world object detection through a continual learning approach and multi‐dataset vision‐language pre‐training is presented. By using natural language for knowledge representation, the authors successfully increase the "visual vocabularyˮ; and create a unified, language‐conditioned detection framework, which outperforms previous models on object detection and phrase grounding. This promising method proves the effectiveness of joint learning from multiple datasets and presents a path forward for scaling to even larger datasets. The advancement of object detection (OD) in open‐vocabulary and open‐world scenarios is a critical challenge in computer vision. OmDet, a novel language‐aware object detection architecture and an innovative training mechanism that harnesses continual learning and multi‐dataset vision‐language pre‐training is introduced. Leveraging natural language as a universal knowledge representation, OmDet accumulates "visual vocabularies" from diverse datasets, unifying the task as a language‐conditioned detection framework. The multimodal detection network (MDN) overcomes the challenges of multi‐dataset joint training and generalizes to numerous training datasets without manual label taxonomy merging. The authors demonstrate superior performance of OmDet over strong baselines in object detection in the wild, open‐vocabulary detection, and phrase grounding, achieving state‐of‐the‐art results. Ablation studies reveal the impact of scaling the pre‐training visual vocabulary, indicating a promising direction for further expansion to larger datasets. The effectiveness of our deep fusion approach is underscored by its ability to learn jointly from multiple datasets, enhancing performance through knowledge sharing.

OmDet: Large‐scale vision‐language multi‐dataset pre‐training with multimodal detection network

Universal Object Detection with Large Vision Model

Language-aware Multiple Datasets Detection Pretraining for DETRs

ScaleDet: A Scalable Multi-Dataset Object Detector

CapDet: Unifying Dense Captioning and Open-World Detection Pretraining

MutDet: Mutually Optimizing Pre-training for Remote Sensing Object Detection

DetCLIP: Dictionary-Enriched Visual-Concept Paralleled Pre-training for Open-world Detection

Unlocking Textual and Visual Wisdom: Open-Vocabulary 3D Object Detection Enhanced by Comprehensive Guidance from Text and Image

DST-Det: Simple Dynamic Self-Training for Open-Vocabulary Object Detection

OV-DINO: Unified Open-Vocabulary Detection with Language-Aware Selective Fusion

Contextual Object Detection with Multimodal Large Language Models

Open Vocabulary Object Detection with Proposal Mining and Prediction Equalization

Exploring Multi-Modal Contextual Knowledge for Open-Vocabulary Object Detection

DetCLIPv2: Scalable Open-Vocabulary Object Detection Pre-training via Word-Region Alignment

Plain-Det: A Plain Multi-Dataset Object Detector

UMS2-ODNet: Unified-scale Domain Adaptation Mechanism Driven Object Detection Network with Multi-Scale Attention

Multi-modal Queried Object Detection in the Wild

Open-vocabulary Object Detection via Vision and Language Knowledge Distillation

LaMI-DETR: Open-Vocabulary Detection with Language Model Instruction

Uni$^2$Det: Unified and Universal Framework for Prompt-Guided Multi-dataset 3D Detection