Abstract:We propose an unsupervised image segmentation method using features from pre-trained text-to-image diffusion models. Inspired by classic spectral clustering approaches, we construct adjacency matrices from self-attention layers between image patches and recursively partition using Normalised Cuts. A key insight is that self-attention probability distributions, which capture semantic relations between patches, can be interpreted as a transition matrix for random walks across the image. We leverage this by first using Random Walk Normalized Cuts directly on these self-attention activations to partition the image, minimizing transition probabilities between clusters while maximizing coherence within clusters. Applied recursively, this yields a hierarchical segmentation that reflects the rich semantics in the pre-trained attention layers, without any additional training. Next, we explore other ways to build the NCuts adjacency matrix from features, and how we can use the random walk interpretation of self-attention to capture long-range relationships. Finally, we propose an approach to automatically determine the NCut cost criterion, avoiding the need to tune this manually. We quantitatively analyse the effect incorporating different features, a constant versus dynamic NCut threshold, and incorporating multi-node paths when constructing the NCuts adjacency matrix. We show that our approach surpasses all existing methods for zero-shot unsupervised segmentation, achieving state-of-the-art results on COCO-Stuff-27 and Cityscapes.

Regularized Tree Partitioning and Its Application to Unsupervised Image Segmentation.

Normalized Tree Partitioning for Image Segmentation

Unsupervised image segmentation based on analysis of binary partition tree for salient object extraction

Automatic Segmentation for Medical Image with the Optimized Tree Structured Part Model

Augmented tree partitioning for interactive image segmentation

Automatic Segmentation of Liver from CT Scans with CCP-TSPM Algorithm

A Multiscale Image Segmentation Algorithm Based on Binary Partition Tree

Image Segmentation Using Hierarchical Merge Tree

Improved image segmentation algorithm based on Normalized Cut

Unsupervised Salient Object Segmentation Based on Kernel Density Estimation and Two-Phase Graph Cut

Tree-Cut for Probabilistic Image Segmentation

Explored Normalized Cut With Random Walk Refining Term for Image Segmentation

Unsupervised Segmentation by Diffusing, Walking and Cutting

Segmentation of Individual Tree Points by Combining Marker-Controlled Watershed Segmentation and Spectral Clustering Optimization

A New Linear Quadtree-based Image Segmentation Algorithm

Optimization of Single Tree Segmentation in Point Cloud Data by Machine Learning Method

TreeSegNet: Adaptive Tree CNNs for Subdecimeter Aerial Image Segmentation

A Regularized Convolutional Neural Network for Semantic Image Segmentation

A Hierarchic Method of Individual Tree Canopy Segmentation Combing UAV Image and LiDAR

New Normalized Nonlocal Hybrid Level Set Method for Image Segmentation

Unsupervised Part Segmentation through Disentangling Appearance and Shape