Abstract:Deep learning based semantic segmentation solutions have yielded compelling results over the preceding decade. They encompass diverse network architectures (FCN based or attention based), along with various mask decoding schemes (parametric softmax based or pixel-query based). Despite the divergence, they can be grouped within a unified framework by interpreting the softmax weights or query vectors as learnable class prototypes. In light of this prototype view, we reveal inherent limitations within the parametric segmentation regime, and accordingly develop a nonparametric alternative based on non-learnable prototypes. In contrast to previous approaches that entail the learning of a single weight/query vector per class in a fully parametric manner, our approach represents each class as a set of non-learnable prototypes, relying solely upon the mean features of training pixels within that class. The pixel-wise prediction is thus achieved by nonparametric nearest prototype retrieving. This allows our model to directly shape the pixel embedding space by optimizing the arrangement between embedded pixels and anchored prototypes. It is able to accommodate an arbitrary number of classes with a constant number of learnable parameters. Through empirical evaluation with FCN based and Transformer based segmentation models (i.e., HRNet, Swin, SegFormer, Mask2Former) and backbones (i.e., ResNet, HRNet, Swin, MiT), our nonparametric framework shows superior performance on standard segmentation datasets (i.e., ADE20K, Cityscapes, COCO-Stuff), as well as in large-vocabulary semantic segmentation scenarios. We expect that this study will provoke a rethink of the current de facto semantic segmentation model design.

TPSN: Transformer-based Multi-Prototype Search Network for Few-Shot Semantic Segmentation

Learning from the Target: Dual Prototype Network for Few Shot Semantic Segmentation

Intermediate Prototype Mining Transformer for Few-Shot Semantic Segmentation

Cycle association prototype network for few-shot semantic segmentation

APANet: Adaptive Prototypes Alignment Network for Few-Shot Semantic Segmentation

PANet: Few-Shot Image Semantic Segmentation With Prototype Alignment

CLIP-Driven Prototype Network for Few-Shot Semantic Segmentation

Dual-Guided Frequency Prototype Network for Few-Shot Semantic Segmentation

Prototype Refinement Network for Few-Shot Segmentation

Prototype Comparison Convolutional Networks for One-Shot Segmentation

Prototype-based Semantic Segmentation

Semantically Meaningful Class Prototype Learning for One-Shot Image Segmentation

MSDNet: Multi-Scale Decoder for Few-Shot Semantic Segmentation via Transformer-Guided Prototyping

Dynamic Prototype Convolution Network for Few-Shot Semantic Segmentation

Dual Complementary Prototype Learning for Few-Shot Segmentation

A lightweight siamese transformer for few-shot semantic segmentation

Few-shot 3D Point Cloud Semantic Segmentation

Self-Regularized Prototypical Network for Few-Shot Semantic Segmentation

Semantically Meaningful Class Prototype Learning for One-Shot Image Semantic Segmentation

Few-Shot 3D Point Cloud Semantic Segmentation via Stratified Class-Specific Attention Based Transformer Network

Spatial Correlation Fusion Network for Few-Shot Segmentation