CORA: Adapting CLIP for Open-Vocabulary Detection with Region Prompting and Anchor Pre-Matching

Xiaoshi Wu,Feng Zhu,Rui Zhao,Hongsheng Li

2023-03-23

Abstract:Open-vocabulary detection (OVD) is an object detection task aiming at detecting objects from novel categories beyond the base categories on which the detector is trained. Recent OVD methods rely on large-scale visual-language pre-trained models, such as CLIP, for recognizing novel objects. We identify the two core obstacles that need to be tackled when incorporating these models into detector training: (1) the distribution mismatch that happens when applying a VL-model trained on whole images to region recognition tasks; (2) the difficulty of localizing objects of unseen classes. To overcome these obstacles, we propose CORA, a DETR-style framework that adapts CLIP for Open-vocabulary detection by Region prompting and Anchor pre-matching. Region prompting mitigates the whole-to-region distribution gap by prompting the region features of the CLIP-based region classifier. Anchor pre-matching helps learning generalizable object localization by a class-aware matching mechanism. We evaluate CORA on the COCO OVD benchmark, where we achieve 41.7 AP50 on novel classes, which outperforms the previous SOTA by 2.4 AP50 even without resorting to extra training data. When extra training data is available, we train CORA$^+$ on both ground-truth base-category annotations and additional pseudo bounding box labels computed by CORA. CORA$^+$ achieves 43.1 AP50 on the COCO OVD benchmark and 28.1 box APr on the LVIS OVD benchmark.

Computer Vision and Pattern Recognition,Artificial Intelligence

What problem does this paper attempt to address?

The paper aims to address the problem of Open-Vocabulary Detection (OVD). Specifically, the goal of OVD is to detect objects in new categories that were not seen during training, rather than just the base categories. The paper points out that current methods rely on large-scale vision-language pre-training models (such as CLIP) to recognize objects in new categories. However, these methods face two core obstacles when integrating vision-language models into the detector training process:

1. **Distribution Mismatch**: There is a distribution mismatch problem when applying a vision-language model trained on whole images to a region recognition task.
2. **Difficulty in Locating Objects of Unseen Categories**: Due to the lack of annotated data for unseen categories, the model finds it difficult to accurately locate objects in these categories.

To address these issues, the authors propose the CORA framework, which is based on a DETR-style design and incorporates the CLIP model. CORA addresses the aforementioned obstacles through two technical means:

1. **Region Prompting**: By prompting region features to adapt the CLIP image encoder, thus alleviating the distribution gap from whole image to region.
2. **Anchor Pre-Matching**: By using a category-aware matching mechanism to help learn a general object localization capability.

Experimental results show that CORA achieves a score of 41.7 AP50 on unseen categories in the COCO OVD benchmark, which is 2.4 AP50 higher than the previous best method. Even without additional training data, CORA demonstrates significant advantages. Furthermore, when additional training data is available, CORA+ further improves performance, achieving 43.1 AP50 on the COCO OVD benchmark and 28.1 box APr on the LVIS OVD benchmark.