Abstract:Open-vocabulary learning can identify categories marked during training (seen categories) and generalize to categories not annotated in the training set (unseen categories). It could theoretically extend segmentation systems to more universal applications. However, current open-vocabulary segmentation frameworks are primarily suited for specific tasks or require retraining according to the task, and they significantly underperform in inferring seen categories compared to fully supervised frameworks. Therefore, we introduce a universal open-vocabulary segmentation framework based on the latent diffusion process (SegLD), which requires only a single training session on a panoptic dataset to achieve inference across all open-vocabulary segmentation tasks, and reaches SOTA segmentation performance for both seen and unseen categories in every task. Specifically, SegLD comprises two stages: in the first stage, we deploy two parallel latent diffusion processes to deeply fuse the text (image caption or category labels) and image information, further aggregating the multi-scale features output from both latent diffusion processes on a scale basis. In the second stage, we introduce text queries, text list queries, and task queries, facilitating the learning of inter-category and inter-task differences through the computation of contrastive losses between them. Text queries are then further fed into a Transformer Decoder to obtain category-agnostic segmentation masks. Then we establish classification loss functions for the type of text input during training, whether image captions or category labels, to help assign a category label from the open vocabulary to each predicted binary mask. Experimental results show that, with just a single training session, SegLD significantly outperforms other contemporary SOTA fully supervised segmentation frameworks and open-vocabulary segmentation frameworks across all evaluation metrics for both known and unknown categories on the ADE20K, Cityscapes, and COCO datasets. This highlights SegLD’s capability as a universal segmentation framework, with the potential to replace other segmentation frameworks and adapt to various segmentation domains. The project link for SegLD is https://github.com/ZhengHongtao2001/SegLD.

Training-Free Open-Vocabulary Segmentation with Offline Diffusion-Augmented Prototype Generation

FreeSeg-Diff: Training-Free Open-Vocabulary Segmentation with Diffusion Models

Diffusion Models for Open-Vocabulary Segmentation

Open-Vocabulary 3D Semantic Segmentation with Text-to-Image Diffusion Models

Open-vocabulary Object Segmentation with Diffusion Models

TAG: Guidance-free Open-Vocabulary Semantic Segmentation

Diffusion Model is Secretly a Training-free Open Vocabulary Semantic Segmenter

Diffusion Model is Secretly a Training-free Open Vocabulary Semantic Segmenter

Open-Vocabulary Panoptic Segmentation with Text-to-Image Diffusion Models

Open-Vocabulary Attention Maps with Token Optimization for Semantic Segmentation in Diffusion Models

Emergent Open-Vocabulary Semantic Segmentation from Off-the-shelf Vision-Language Models

OpenDAS: Open-Vocabulary Domain Adaptation for 2D and 3D Segmentation

SegLD: Achieving Universal, Zero-Shot and Open-Vocabulary Segmentation Through Multimodal Fusion Via Latent Diffusion Processes

Learning Open-vocabulary Semantic Segmentation Models from Natural Language Supervision.

Exploring Limits of Diffusion-Synthetic Training with Weakly Supervised Semantic Segmentation

Dataset Diffusion: Diffusion-based Synthetic Dataset Generation for Pixel-Level Semantic Segmentation

Text2Seg: Remote Sensing Image Semantic Segmentation via Text-Guided Visual Foundation Models

A Simple Framework for Open-Vocabulary Segmentation and Detection

MaskDiffusion: Exploiting Pre-Trained Diffusion Models for Semantic Segmentation

DaTaSeg: Taming a Universal Multi-Dataset Multi-Task Segmentation Model

OpenSD: Unified Open-Vocabulary Segmentation and Detection