LaMI-DETR: Open-Vocabulary Detection with Language Model Instruction

Penghui Du,Yu Wang,Yifan Sun,Luting Wang,Yue Liao,Gang Zhang,Errui Ding,Yan Wang,Jingdong Wang,Si Liu

2024-07-18

Abstract:Existing methods enhance open-vocabulary object detection by leveraging the robust open-vocabulary recognition capabilities of Vision-Language Models (VLMs), such as CLIP.However, two main challenges emerge:(1) A deficiency in concept representation, where the category names in CLIP's text space lack textual and visual knowledge.(2) An overfitting tendency towards base categories, with the open vocabulary knowledge biased towards base categories during the transfer from VLMs to <a class="link-external link-http" href="http://detectors.To" rel="external noopener nofollow">this http URL</a> address these challenges, we propose the Language Model Instruction (LaMI) strategy, which leverages the relationships between visual concepts and applies them within a simple yet effective DETR-like detector, termed LaMI-DETR.LaMI utilizes GPT to construct visual concepts and employs T5 to investigate visual similarities across categories.These inter-category relationships refine concept representation and avoid overfitting to base categories.Comprehensive experiments validate our approach's superior performance over existing methods in the same rigorous setting without reliance on external training resources.LaMI-DETR achieves a rare box AP of 43.4 on OV-LVIS, surpassing the previous best by 7.8 rare box AP.

Computer Vision and Pattern Recognition

What problem does this paper attempt to address?

The paper aims to address two main challenges in Open Vocabulary Object Detection (OVOD):

1. **Insufficient Concept Representation**: Existing methods mostly rely on the CLIP text encoder to represent category names, but this approach has limitations in capturing the textual and visual semantic similarities between categories. It struggles to distinguish visually similar categories and cannot effectively explore potential new categories.

2. **Overfitting to Base Categories**: Although Visual Language Models (VLMs) perform well on new categories, the open vocabulary detectors are optimized using only base detection data, leading to overfitting to the base categories. This results in misclassifying new objects as background or base categories.

To address the above issues, the paper proposes the LaMI strategy (Language Model Instruction), which leverages the relationships between visual concepts and applies them to a simple yet effective DETR-like detector, called LaMI-DETR. LaMI improves concept representation and avoids overfitting to base categories by utilizing GPT to construct visual concepts and using T5 to study cross-category visual similarity. Experimental results show that LaMI-DETR achieves significant performance improvements on the OV-LVIS benchmark without relying on external training resources, particularly in the detection of rare categories, reaching a box AP of 43.4, surpassing the previous best method by 7.8 box AP.