Abstract:This paper studies the problem of weakly open-vocabulary semantic segmentation (WOVSS), which learns to segment objects of arbitrary classes using mere image-text pairs. Existing works turn to enhance the vanilla vision transformer by introducing explicit grouping recognition, i.e., employing several group tokens/centroids to cluster the image tokens and perform the group-text alignment. Nevertheless, these methods suffer from a granularity inconsistency regarding the usage of group tokens, which are aligned in the all-to-one v.s. one-to-one manners during the training and inference phases, respectively. We argue that this discrepancy arises from the lack of elaborate supervision for each group token. To bridge this granularity gap, this paper explores explicit supervision for the group tokens from the prototypical knowledge. To this end, this paper proposes the non-learnable prototypical regularization (NPR) where non-learnable prototypes are estimated from source features to serve as supervision and enable contrastive matching of the group tokens. This regularization encourages the group tokens to segment objects with less redundancy and capture more comprehensive semantic regions, leading to increased compactness and richness. Based on NPR, we propose the prototypical guidance segmentation network (PGSeg) that incorporates multi-modal regularization by leveraging prototypical sources from both images and texts at different levels, progressively enhancing the segmentation capability with diverse prototypical patterns. Experimental results show that our proposed method achieves state-of-the-art performance on several benchmark datasets. The source code is available at <a class="link-external link-https" href="https://github.com/Ferenas/PGSeg" rel="external noopener nofollow">this https URL</a>.

Class Semantics Modulation for Open-Set Instance Segmentation

In Defense Of Multi-Source Omni-Supervised Efficient Convnet For Robust Semantic Segmentation In Heterogeneous Unseen Domains

Synthetic Instance Segmentation from Semantic Image Segmentation Masks

Open-Vocabulary Instance Segmentation via Robust Cross-Modal Pseudo-Labeling

Multi-Modal Prototypes for Open-World Semantic Segmentation

Rethinking mask heads for partially supervised instance segmentation

Prototype-based Semantic Segmentation

Learning Open-vocabulary Semantic Segmentation Models from Natural Language Supervision.

Open-vocabulary Panoptic Segmentation with Embedding Modulation

Boosting Semantic Segmentation from the Perspective of Explicit Class Embeddings

SSA-Seg: Semantic and Spatial Adaptive Pixel-level Classifier for Semantic Segmentation

Deep Object Co-segmentation via Spatial-Semantic Network Modulation

Exploring Simple Open-Vocabulary Semantic Segmentation

Uncovering Prototypical Knowledge for Weakly Open-Vocabulary Semantic Segmentation

Rethinking Semantic Segmentation: A Prototype View

SegMind: Semisupervised Remote Sensing Image Semantic Segmentation With Masked Image Modeling and Contrastive Learning Method

Continual Segmentation with Disentangled Objectness Learning and Class Recognition

Object Segmentation by Mining Cross-Modal Semantics

Semantic and Spatial Adaptive Pixel-level Classifier for Semantic Segmentation

Unsupervised Universal Image Segmentation

SSUL: Semantic Segmentation with Unknown Label for Exemplar-based Class-Incremental Learning