Abstract:In the realm of high-resolution (HR), fine-grained image segmentation, the primary challenge is balancing broad contextual awareness with the precision required for detailed object delineation, capturing intricate details and the finest edges of objects. Diffusion models, trained on vast datasets comprising billions of image-text pairs, such as SD V2.1, have revolutionized text-to-image synthesis by delivering exceptional quality, fine detail resolution, and strong contextual awareness, making them an attractive solution for high-resolution image segmentation. To this end, we propose DiffDIS, a diffusion-driven segmentation model that taps into the potential of the pre-trained U-Net within diffusion models, specifically designed for high-resolution, fine-grained object segmentation. By leveraging the robust generalization capabilities and rich, versatile image representation prior of the SD models, coupled with a task-specific stable one-step denoising approach, we significantly reduce the inference time while preserving high-fidelity, detailed generation. Additionally, we introduce an auxiliary edge generation task to not only enhance the preservation of fine details of the object boundaries, but reconcile the probabilistic nature of diffusion with the deterministic demands of segmentation. With these refined strategies in place, DiffDIS serves as a rapid object mask generation model, specifically optimized for generating detailed binary maps at high resolutions, while demonstrating impressive accuracy and swift processing. Experiments on the DIS5K dataset demonstrate the superiority of DiffDIS, achieving state-of-the-art results through a streamlined inference process. Our code will be made publicly available.

Instance-Aware Diffusion Implicit Process for Box-Based Instance Segmentation

High-Precision Dichotomous Image Segmentation via Probing Diffusion Capacity

BoxInst: High-Performance Instance Segmentation with Box Annotations.

A Simple Latent Diffusion Approach for Panoptic Segmentation and Mask Inpainting

Conditional Convolutions for Instance Segmentation

Deep Level Set for Box-supervised Instance Segmentation in Aerial Images

InstanceDiffusion: Instance-level Control for Image Generation

DFormer: Diffusion-guided Transformer for Universal Image Segmentation

IPoD: Implicit Field Learning with Point Diffusion for Generalizable 3D Object Reconstruction from Single RGB-D Images

Diffusion Features to Bridge Domain Gap for Semantic Segmentation

DiffusionDet: Diffusion Model for Object Detection

ISBNet: a 3D Point Cloud Instance Segmentation Network with Instance-aware Sampling and Box-aware Dynamic Convolution

Edge-and-Mask Integration-Driven Diffusion Models for Medical Image Segmentation

A boundary-preserving conditional convolution network for instance segmentation

Box2Mask: Box-supervised Instance Segmentation via Level-set Evolution

DyCo3D: Robust Instance Segmentation of 3D Point Clouds Through Dynamic Convolution

BoxDiff: Text-to-Image Synthesis with Training-Free Box-Constrained Diffusion

Explicit-implicit priori knowledge-based diffusion model for generative medical image segmentation

Efficient Denoising Diffusion Via Probabilistic Masking

INSTANCE SEGMENTATION BY LEARNING DEEP FEATURE IN EMBEDDING SPACE