Abstract:Unsupervised semantic segmentation (USS) aims at partitioning an image into semantically meaningful segments by learning from a collection of unlabeled images. The effectiveness of current approaches is plagued by difficulties in coordinating representation learning and pixel clustering, modeling the varying feature distributions of different classes, handling outliers and noise, and addressing the pixel class imbalance problem. This paper introduces a novel approach, termed Imbalance-Aware Dense Discriminative Clustering (IDDC), for USS, which addresses all these difficulties in a unified framework. Different from existing approaches, which learn USS in two stages (i.e., generating and updating pseudo masks, or refining and clustering embeddings), IDDC learns pixel-wise feature representation and dense discriminative clustering in an end-to-end and self-supervised manner, through a novel objective function that transfers the manifold structure of pixels in the embedding space of a vision Transformer (ViT) to the label space while tolerating the noise in pixel affinities. During inference, the trained model directly outputs the classification probability of each pixel conditioned on the image. In addition, this paper proposes a new regularizer, based on the Weibull function, to handle pixel class imbalance and cluster degeneration in a single shot. Experimental results demonstrate that IDDC significantly outperforms all previous USS methods on three real-world datasets, COCO-Stuff-27, COCO-Stuff-171, and Cityscapes. Extensive ablation studies validate the effectiveness of each design. Our code is available at https://github.com/MY-LIU100101/IDDC.

Harmonizing Output Imbalance for semantic segmentation on extremely-imbalanced input data

Harmonizing output imbalance for defect segmentation on extremely-imbalanced photovoltaic module cells images

Balance-batch: An Optimized Method for Semantic Segmentation Loss Functions

Striking the Right Balance: Recall Loss for Semantic Segmentation

Penalizing Top Performers: Conservative Loss for Semantic Segmentation Adaptation

Analyzing Overfitting under Class Imbalance in Neural Networks for Image Segmentation

Subclassified Loss: Rethinking Data Imbalance from Subclass Perspective for Semantic Segmentation

Adaptive Spatial-BCE Loss for Weakly Supervised Semantic Segmentation.

March on Data Imperfections: Domain Division and Domain Generalization for Semantic Segmentation.

Delving into Semantic Scale Imbalance

Contour-weighted loss for class-imbalanced image segmentation

Splitting Vs. Merging: Mining Object Regions with Discrepancy and Intersection Loss for Weakly Supervised Semantic Segmentation

Semi-Supervised Semantic Segmentation Via Adaptive Equalization Learning

Threshold-adaptive Unsupervised Focal Loss for Domain Adaptation of Semantic Segmentation

Domain Adaptative Semantic Segmentation by Alleviating Long-tail Problem.

Instance-wise Uncertainty for Class Imbalance in Semantic Segmentation

Improved segmentation under extreme imbalance towards full background images

Balancing Logit Variation for Long-tailed Semantic Segmentation

A Simple Baseline for Semi-supervised Semantic Segmentation with Strong Data Augmentation

Imbalance-Aware Discriminative Clustering for Unsupervised Semantic Segmentation