TagCLIP: Improving Discrimination Ability of Open-Vocabulary Semantic Segmentation

Jingyao Li,Pengguang Chen,Shengju Qian,Shu Liu,Jiaya Jia

2024-09-03

Abstract:Contrastive Language-Image Pre-training (CLIP) has recently shown great promise in pixel-level zero-shot learning tasks. However, existing approaches utilizing CLIP's text and patch embeddings to generate semantic masks often misidentify input pixels from unseen classes, leading to confusion between novel classes and semantically similar ones. In this work, we propose a novel approach, TagCLIP (Trusty-aware guided CLIP), to address this issue. We disentangle the ill-posed optimization problem into two parallel processes: semantic matching performed individually and reliability judgment for improving discrimination ability. Building on the idea of special tokens in language modeling representing sentence-level embeddings, we introduce a trusty token that enables distinguishing novel classes from known ones in prediction. To evaluate our approach, we conduct experiments on two benchmark datasets, PASCAL VOC 2012, COCO-Stuff 164K and PASCAL Context. Our results show that TagCLIP improves the Intersection over Union (IoU) of unseen classes by 7.4%, 1.7% and 2.1%, respectively, with negligible overheads. The code is available at <a class="link-external link-https" href="https://github.com/dvlab-research/TagCLIP" rel="external noopener nofollow">this https URL</a>.

Computer Vision and Pattern Recognition

What problem does this paper attempt to address?

The paper attempts to address the issue in zero-shot semantic segmentation tasks where existing methods tend to misclassify pixels into known categories when dealing with unseen categories, resulting in poor recognition ability for new categories. Specifically, although existing methods utilize CLIP's text and image embeddings to generate semantic masks, these methods often encounter confusion when handling unseen categories, especially among semantically similar categories.

To solve this problem, the authors propose a new method called TagCLIP (Trusty-aware guided CLIP), which introduces a trusty token to distinguish between known and unknown categories and improve the model's discriminative ability. The specific approach includes:

1. **Decomposing the optimization problem**: Decomposing the originally complex optimization problem into two parallel processes: one for independent semantic matching and the other for assessing the reliability of predictions.
2. **Introducing a trusty token**: Designing an additional trusty token to reflect the prediction tendency of known and unknown categories.
3. **Optimizing the trusty token**: Using a module called Trusty Learner to optimize the trusty token in parallel with semantic matching, enhancing the model's ability to discriminate unseen categories with almost no additional overhead.

With these improvements, experimental results on multiple datasets show that TagCLIP significantly improves the Intersection over Union (IoU) for unseen categories, with almost no additional time and storage costs during inference.