Abstract:Unsupervised semantic segmentation (USS) aims at partitioning an image into semantically meaningful segments by learning from a collection of unlabeled images. The effectiveness of current approaches is plagued by difficulties in coordinating representation learning and pixel clustering, modeling the varying feature distributions of different classes, handling outliers and noise, and addressing the pixel class imbalance problem. This paper introduces a novel approach, termed Imbalance-Aware Dense Discriminative Clustering (IDDC), for USS, which addresses all these difficulties in a unified framework. Different from existing approaches, which learn USS in two stages (i.e., generating and updating pseudo masks, or refining and clustering embeddings), IDDC learns pixel-wise feature representation and dense discriminative clustering in an end-to-end and self-supervised manner, through a novel objective function that transfers the manifold structure of pixels in the embedding space of a vision Transformer (ViT) to the label space while tolerating the noise in pixel affinities. During inference, the trained model directly outputs the classification probability of each pixel conditioned on the image. In addition, this paper proposes a new regularizer, based on the Weibull function, to handle pixel class imbalance and cluster degeneration in a single shot. Experimental results demonstrate that IDDC significantly outperforms all previous USS methods on three real-world datasets, COCO-Stuff-27, COCO-Stuff-171, and Cityscapes. Extensive ablation studies validate the effectiveness of each design. Our code is available at https://github.com/MY-LIU100101/IDDC.

CLUSTSEG: Clustering for Universal Segmentation

ClusterFormer: Clustering As A Universal Visual Learner

Unsupervised Segmentation Using Cluster Ensembles.

Unsupervised Universal Image Segmentation

A Lightweight Clustering Framework for Unsupervised Semantic Segmentation

ClusterSeg: A Crowd Cluster Pinpointed Nucleus Segmentation Framework with Cross-Modality Datasets

Pixel-level clustering network for unsupervised image segmentation

Learnable Pixel Clustering Via Structure and Semantic Dual Constraints for Unsupervised Image Segmentation.

Deep Markov Clustering for Panoptic Segmentation.

Unsupervised Hierarchical Semantic Segmentation with Multiview Cosegmentation and Clustering Transformers

Image Segmentation Algorithm Based on Superpixel Clustering

Imbalance-Aware Discriminative Clustering for Unsupervised Semantic Segmentation

Tensor multi-view clustering method for natural image segmentation

Enhancing Unsupervised Semantic Segmentation Through Context-Aware Clustering

Automatic Fuzzy Clustering Framework for Image Segmentation

Cluster Sensing Superpixel and Grouping

Semantic Equitable Clustering: A Simple and Effective Strategy for Clustering Vision Tokens

Clustering based Point Cloud Representation Learning for 3D Analysis

CEU-Net: ensemble semantic segmentation of hyperspectral images using clustering

SMAC-Seg: LiDAR Panoptic Segmentation via Sparse Multi-directional Attention Clustering

USE: Universal Segment Embeddings for Open-Vocabulary Image Segmentation