Abstract:This work studies the problem of image semantic segmentation. Current approaches focus mainly on mining "local" context, i.e., dependencies between pixels within individual images, by specifically-designed, context aggregation modules (e.g., dilated convolution, neural attention) or structure-aware optimization objectives (e.g., IoU-like loss). However, they ignore "global" context of the training data, i.e., rich semantic relations between pixels across different images. Inspired by recent advance in unsupervised contrastive representation learning, we propose a pixel-wise contrastive algorithm, dubbed as PiCo, for semantic segmentation in the fully supervised learning setting. The core idea is to enforce pixel embeddings belonging to a same semantic class to be more similar than embeddings from different classes. It raises a pixel-wise metric learning paradigm for semantic segmentation, by explicitly exploring the structures of labeled pixels, which were rarely studied before. Our training algorithm is compatible with modern segmentation solutions without extra overhead during testing. We experimentally show that, with famous segmentation models (i.e., DeepLabV3, HRNet, OCRNet, SegFormer, Segmenter, MaskFormer) and backbones (i.e., MobileNet, ResNet, HRNet, MiT, ViT), our algorithm brings consistent performance improvements across diverse datasets (i.e., Cityscapes, ADE20K, PASCAL-Context, COCO-Stuff, CamVid). We expect that this work will encourage our community to rethink the current de facto training paradigm in semantic segmentation. Our code is available at https://github.com/tfzhou/ContrastiveSeg.

Guided Contrastive Boundary Learning for Semantic Segmentation

Contrastive Boundary Learning for Point Cloud Segmentation

Region-aware Contrastive Learning for Semantic Segmentation

Boundary-aware Graph Convolution for Semantic Segmentation.

Boundary-Guided Lightweight Semantic Segmentation With Multi-Scale Semantic Context

Contextrast: Contextual Contrastive Learning for Semantic Segmentation

Exploring Cross-Image Pixel Contrast for Semantic Segmentation

Adversarial Dense Contrastive Learning for Semi-Supervised Semantic Segmentation

Boundary–Inner Disentanglement Enhanced Learning for Point Cloud Semantic Segmentation

Optimizing Spatial Relationships in GCN to Improve the Classification Accuracy of Remote Sensing Images

Push-the-Boundary: Boundary-aware Feature Propagation for Semantic Segmentation of 3D Point Clouds

Directed Mix Contrast for Lidar Point Cloud Segmentation

PRCL: Probabilistic Representation Contrastive Learning for Semi-Supervised Semantic Segmentation

Boundary-aware Contrastive Learning for Semi-supervised Nuclei Instance Segmentation

Positive-Negative Equal Contrastive Loss for Semantic Segmentation

Progressive Learning With Cross-Window Consistency for Semi-Supervised Semantic Segmentation

Cross-Image Pixel Contrasting for Semantic Segmentation

Multimodal Contrastive Learning for Unpaired and Depth-privileged Semantic Segmentation.

Semantic-Aware Contrastive Learning for Multi-Object Medical Image Segmentation

Pixel Contrastive-Consistent Semi-Supervised Semantic Segmentation

Deep Clustering by Semantic Contrastive Learning