Abstract:The goal of interactive image segmentation is to delineate specific regions within an image via visual or language prompts. Low-latency and high-quality interactive segmentation with diverse prompts remain challenging for existing specialist and generalist models. Specialist models, with their limited prompts and task-specific designs, experience high latency because the image must be recomputed every time the prompt is updated, due to the joint encoding of image and visual prompts. Generalist models, exemplified by the Segment Anything Model (SAM), have recently excelled in prompt diversity and efficiency, lifting image segmentation to the foundation model era. However, for high-quality segmentations, SAM still lags behind state-of-the-art specialist models despite SAM being trained with x100 more segmentation masks. In this work, we delve deep into the architectural differences between the two types of models. We observe that dense representation and fusion of visual prompts are the key design choices contributing to the high segmentation quality of specialist models. In light of this, we reintroduce this dense design into the generalist models, to facilitate the development of generalist models with high segmentation quality. To densely represent diverse visual prompts, we propose to use a dense map to capture five types: clicks, boxes, polygons, scribbles, and masks. Thus, we propose SegNext, a next-generation interactive segmentation approach offering low latency, high quality, and diverse prompt support. Our method outperforms current state-of-the-art methods on HQSeg-44K and DAVIS, both quantitatively and qualitatively.

Interactive Segmentation by Considering First-Click Intentional Ambiguity

Interactive Image Segmentation with First Click Attention

FocalClick: Towards Practical Interactive Image Segmentation.

Spatiotemporal Consistent Selection-Correction Network for Deep Interactive Image Segmentation.

Sequential Interactive Image Segmentation

PiClick: Picking the desired mask from multiple candidates in click-based interactive segmentation

One-Click-Based Perception for Interactive Image Segmentation

Rethinking Interactive Image Segmentation with Low Latency, High Quality, and Diverse Prompts

ClickAttention: Click Region Similarity Guided Interactive Segmentation

Refining Segmentation On-the-Fly: An Interactive Framework for Point Cloud Semantic Segmentation

Regional Interactive Image Segmentation Networks

AdaptiveClick: Click-Aware Transformer with Adaptive Focal Loss for Interactive Image Segmentation

AdaptiveClick: Clicks-aware Transformer with Adaptive Focal Loss for Interactive Image Segmentation

CSANet: Cross-self attention guided by semantic click embedding for interactive segmentation

Rethinking click embedding for deep interactive image segmentation

Deep Interactive Image Segmentation Based on Region and Boundary-Click Guidance

AGILE3D: Attention Guided Interactive Multi-object 3D Segmentation

Order-aware Interactive Segmentation

Learning from Exemplars for Interactive Image Segmentation

Interactive Segmentation with Recommendation of Most Informative Regions.