Abstract:Open World Object Detection (OWOD) aims to adapt object detection to an open-world environment, so as to detect unknown objects and learn knowledge incrementally. Existing OWOD methods typically leverage training sets with a relatively small number of known objects. Due to the absence of generic object knowledge, they fail to comprehensively perceive objects beyond the scope of training sets. Recent advancements in large vision models (LVMs), trained on extensive large-scale data, offer a promising opportunity to harness rich generic knowledge for the fundamental advancement of OWOD. Motivated by Segment Anything Model (SAM), a prominent LVM lauded for its exceptional ability to segment generic objects, we first demonstrate the possibility to employ SAM for OWOD and establish the very first SAM-Guided OWOD baseline solution. Subsequently, we identify and address two fundamental challenges in SAM-Guided OWOD and propose a pioneering SAM-Guided Robust Open-world Detector (SGROD) method, which can significantly improve the recall of unknown objects without losing the precision on known objects. Specifically, the two challenges in SAM-Guided OWOD include: (1) Noisy labels caused by the class-agnostic nature of SAM; (2) Precision degradation on known objects when more unknown objects are recalled. For the first problem, we propose a dynamic label assignment (DLA) method that adaptively selects confident labels from SAM during training, evidently reducing the noise impact. For the second problem, we introduce cross-layer learning (CLL) and SAM-based negative sampling (SNS), which enable SGROD to avoid precision loss by learning robust decision boundaries of objectness and classification. Experiments on public datasets show that SGROD not only improves the recall of unknown objects by a large margin (~ 20%), but also preserves highly-competitive precision on known objects. The program codes are available at https://github.com/harrylin-hyl/SGROD.

Text-Guided Unknown Pseudo-Labeling for Open-World Object Detection

Towards Open-Set Semantic Labeling in 3D Point Clouds : Analysis on the Unknown Class.

Unsupervised Recognition of Unknown Objects for Open-World Object Detection

A Semantic Labeling Strategy to Reject Unknown Objects in Large Scale 3d Point Clouds

Annealing-Based Label-Transfer Learning for Open World Object Detection

Recalling Unknowns without Losing Precision: An Effective Solution to Large Model-Guided Open World Object Detection

Detecting the open-world objects with the help of the Brain

GOOD: Exploring Geometric Cues for Detecting Objects in an Open World

From Open Vocabulary to Open World: Teaching Vision Language Models to Detect Novel Objects

OV-DQUO: Open-Vocabulary DETR with Denoising Text Query Training and Open-World Unknown Objects Supervision

USD: Unknown Sensitive Detector Empowered by Decoupled Objectness and Segment Anything Model

CAT: LoCalization and IdentificAtion Cascade Detection Transformer for Open-World Object Detection.

Exploring Orthogonality in Open World Object Detection

UC-OWOD: Unknown-Classified Open World Object Detection

Open-World Object Detection with Instance Representation Learning

Online Pseudo-Label Unified Object Detection for Multiple Datasets Training

OW-Adapter: Human-Assisted Open-World Object Detection with a Few Examples

OPODet: Toward Open World Potential Oriented Object Detection in Remote Sensing Images

Addressing the Challenges of Open-World Object Detection

What Makes Good Open-Vocabulary Detector: A Disassembling Perspective

TagOOD: A Novel Approach to Out-of-Distribution Detection via Vision-Language Representations and Class Center Learning