Abstract:In semantic segmentation, generalizing a visual system to both seen categories and novel categories at inference time has always been practically valuable yet challenging. To enable such functionality, existing methods mainly rely on either providing several support demonstrations from the visual aspect or characterizing the informative clues from the textual aspect (e.g., the class names). Nevertheless, both two lines neglect the complementary intrinsic of low-level visual and high-level language information, while the explorations that consider visual and textual modalities as a whole to promote predictions are still limited. To close this gap, we propose to encompass textual and visual clues as multi-modal prototypes to allow more comprehensive support for open-world semantic segmentation, and build a novel prototype-based segmentation framework to realize this promise. To be specific, unlike the straightforward combination of bi-modal clues, we decompose the high-level language information as multi-aspect prototypes and aggregate the low-level visual information as more semantic prototypes, on basis of which, a fine-grained complementary fusion makes the multi-modal prototypes more powerful and accurate to promote the prediction. Based on an elastic mask prediction module that permits any number and form of prototype inputs, we are able to solve the zero-shot, few-shot and generalized counterpart tasks in one architecture. Extensive experiments on both PASCAL-$5^i$ and COCO-$20^i$ datasets show the consistent superiority of the proposed method compared with the previous state-of-the-art approaches, and a range of ablation studies thoroughly dissects each component in our framework both quantitatively and qualitatively that verify their effectiveness.

POEM: A prototype cross and emphasis network for few-shot semantic segmentation

CLIP-Driven Prototype Network for Few-Shot Semantic Segmentation

Dense Cross-Query-and-Support Attention Weighted Mask Aggregation for Few-Shot Segmentation

APANet: Adaptive Prototypes Alignment Network for Few-Shot Semantic Segmentation

Prior-Enhanced Few-Shot Segmentation with Meta-Prototypes

Learning from the Target: Dual Prototype Network for Few Shot Semantic Segmentation

Prototype Comparison Convolutional Networks for One-Shot Segmentation

Cycle association prototype network for few-shot semantic segmentation

Prototype-based Semantic Segmentation

Semantically Meaningful Class Prototype Learning for One-Shot Image Segmentation

Learning Prototype from Unlabeled Regions for Few-shot Segmentation

Query-guided Prototype Evolution Network for Few-Shot Segmentation

Rethinking Semantic Segmentation: A Prototype View

Semantically Meaningful Class Prototype Learning for One-Shot Image Semantic Segmentation

Multi-Similarity Enhancement Network for Few-Shot Segmentation.

Beyond singular prototype: A prototype splitting strategy for few-shot medical image segmentation

PANet: Few-Shot Image Semantic Segmentation With Prototype Alignment

Multi-Modal Prototypes for Open-World Semantic Segmentation

Prototype Mixture Models for Few-shot Semantic Segmentation

Dynamic Prototype Convolution Network for Few-Shot Semantic Segmentation

APSeg: Auto-Prompt Network for Cross-Domain Few-Shot Semantic Segmentation