Towards Training-free Open-world Segmentation via Image Prompt Foundation Models

Lv Tang,Peng-Tao Jiang,Hao-Ke Xiao,Bo Li

2024-06-26

Abstract:The realm of computer vision has witnessed a paradigm shift with the advent of foundational models, mirroring the transformative influence of large language models in the domain of natural language processing. This paper delves into the exploration of open-world segmentation, presenting a novel approach called Image Prompt Segmentation (IPSeg) that harnesses the power of vision foundational models. IPSeg lies the principle of a training-free paradigm, which capitalizes on image prompt techniques. Specifically, IPSeg utilizes a single image containing a subjective visual concept as a flexible prompt to query vision foundation models like DINOv2 and Stable Diffusion. Our approach extracts robust features for the prompt image and input image, then matches the input representations to the prompt representations via a novel feature interaction module to generate point prompts highlighting target objects in the input image. The generated point prompts are further utilized to guide the Segment Anything Model to segment the target object in the input image. The proposed method stands out by eliminating the need for exhaustive training sessions, thereby offering a more efficient and scalable solution. Experiments on COCO, PASCAL VOC, and other datasets demonstrate IPSeg's efficacy for flexible open-world segmentation using intuitive image prompts. This work pioneers tapping foundation models for open-world understanding through visual concepts conveyed in images.

Computer Vision and Pattern Recognition

What problem does this paper attempt to address?

The paper attempts to address the problem of how to achieve efficient and flexible object segmentation in open-world scenarios using image prompt technology without additional training.

Specifically, the paper proposes a new framework called Image Prompt Segmentation (IPSeg), which aims to guide the base models (such as DINOv2 and Stable Diffusion) to extract features of the target object through image prompts and generate point prompts, thereby guiding the Segment Anything Model (SAM) to complete the object segmentation task. The core of this method lies in:

1. **Image Prompt**: Users can provide an image containing specific visual concepts as a prompt. The system can identify the same or similar objects in the input image based on the target object in the prompt image.
2. **Feature Extraction and Interaction**: IPSeg extracts features from the prompt image and the input image through two branches, and then matches these two features through a feature interaction module to generate point prompts that highlight the target object.
3. **Training-Free Segmentation**: The entire process does not require additional training of the model, thereby improving the efficiency and scalability of the method.

The paper validates the effectiveness of IPSeg through experiments on multiple datasets, especially showing significant performance improvements on datasets such as COCO and PASCAL VOC. This method not only simplifies the task of open-world object segmentation but also provides new ideas for future research.