Abstract:Open World Object Detection (OWOD) aims to adapt object detection to an open-world environment, so as to detect unknown objects and learn knowledge incrementally. Existing OWOD methods typically leverage training sets with a relatively small number of known objects. Due to the absence of generic object knowledge, they fail to comprehensively perceive objects beyond the scope of training sets. Recent advancements in large vision models (LVMs), trained on extensive large-scale data, offer a promising opportunity to harness rich generic knowledge for the fundamental advancement of OWOD. Motivated by Segment Anything Model (SAM), a prominent LVM lauded for its exceptional ability to segment generic objects, we first demonstrate the possibility to employ SAM for OWOD and establish the very first SAM-Guided OWOD baseline solution. Subsequently, we identify and address two fundamental challenges in SAM-Guided OWOD and propose a pioneering SAM-Guided Robust Open-world Detector (SGROD) method, which can significantly improve the recall of unknown objects without losing the precision on known objects. Specifically, the two challenges in SAM-Guided OWOD include: (1) Noisy labels caused by the class-agnostic nature of SAM; (2) Precision degradation on known objects when more unknown objects are recalled. For the first problem, we propose a dynamic label assignment (DLA) method that adaptively selects confident labels from SAM during training, evidently reducing the noise impact. For the second problem, we introduce cross-layer learning (CLL) and SAM-based negative sampling (SNS), which enable SGROD to avoid precision loss by learning robust decision boundaries of objectness and classification. Experiments on public datasets show that SGROD not only improves the recall of unknown objects by a large margin (~ 20%), but also preserves highly-competitive precision on known objects. The program codes are available at https://github.com/harrylin-hyl/SGROD.

Recalling Unknowns without Losing Precision: An Effective Solution to Large Model-Guided Open World Object Detection

USD: Unknown Sensitive Detector Empowered by Decoupled Objectness and Segment Anything Model

LVMUM: Toward Open-World Object Detection with Large Vision Models and Unsupervised Modeling

Spatial Likelihood Voting with Self-Knowledge Distillation for Weakly Supervised Object Detection.

Unsupervised Recognition of Unknown Objects for Open-World Object Detection

Revisiting Open World Object Detection.

KTCN: Enhancing Open-World Object Detection with Knowledge Transfer and Class-Awareness Neutralization

DDOWOD: DiffusionDet for Open-World Object Detection

Detecting the open-world objects with the help of the Brain

Sniffing Threatening Open-World Objects in Autonomous Driving by Open-Vocabulary Models

Open World Object Detection: A Survey

SKDF: A Simple Knowledge Distillation Framework for Distilling Open-Vocabulary Knowledge to Open-world Object Detector

MR-GDINO: Efficient Open-World Continual Object Detection

Annealing-Based Label-Transfer Learning for Open World Object Detection

OW-Adapter: Human-Assisted Open-World Object Detection with a Few Examples

UC-OWOD: Unknown-Classified Open World Object Detection

Text-Guided Unknown Pseudo-Labeling for Open-World Object Detection

S3OD: Single Stage Small Object Detector from Scratch for Remote Sensing Images

Redefining Object Detection for Open-World Settings: A Framework for Simultaneous Identification of Known and Unknown Classes

(Sod)-O-3: Single Stage Small Object Detector From Scratch For Remote Sensing Images

YOLO-UniOW: Efficient Universal Open-World Object Detection