SegCLIP: Patch Aggregation with Learnable Centers for Open-Vocabulary Semantic Segmentation

Huaishao Luo,Junwei Bao,Youzheng Wu,Xiaodong He,Tianrui Li

2023-06-20

Abstract:Recently, the contrastive language-image pre-training, e.g., CLIP, has demonstrated promising results on various downstream tasks. The pre-trained model can capture enriched visual concepts for images by learning from a large scale of text-image data. However, transferring the learned visual knowledge to open-vocabulary semantic segmentation is still under-explored. In this paper, we propose a CLIP-based model named SegCLIP for the topic of open-vocabulary segmentation in an annotation-free manner. The SegCLIP achieves segmentation based on ViT and the main idea is to gather patches with learnable centers to semantic regions through training on text-image pairs. The gathering operation can dynamically capture the semantic groups, which can be used to generate the final segmentation results. We further propose a reconstruction loss on masked patches and a superpixel-based KL loss with pseudo-labels to enhance the visual representation. Experimental results show that our model achieves comparable or superior segmentation accuracy on the PASCAL VOC 2012 (+0.3% mIoU), PASCAL Context (+2.3% mIoU), and COCO (+2.2% mIoU) compared with baselines. We release the code at <a class="link-external link-https" href="https://github.com/ArrowLuo/SegCLIP" rel="external noopener nofollow">this https URL</a>.

Computer Vision and Pattern Recognition,Artificial Intelligence

What problem does this paper attempt to address?

### Problems the Paper Aims to Solve

The paper primarily aims to address the issue of open-vocabulary semantic segmentation. Specifically:

1. **Expensive Pixel-Level Annotations**: Traditional semantic segmentation methods require a large amount of pixel-level annotated data, which is a very time-consuming and costly process in practical applications.
2. **Limited Label Categories**: Existing methods are usually limited to a fixed set of label categories, resulting in weak generalization ability of the model when facing new categories.

To overcome these two limitations, the paper proposes a CLIP-based model called SegCLIP, which achieves open-vocabulary semantic segmentation without additional annotations by leveraging large-scale image-text pairs for training. The main innovations of SegCLIP include:
- Using a learnable center patch aggregation module to dynamically capture semantic regions.
- Proposing a reconstruction loss and superpixel KL loss to enhance visual representation.

Experimental results show that SegCLIP has competitive or even superior segmentation accuracy compared to baseline methods on the PASCAL VOC 2012, PASCAL Context, and COCO datasets.