Jinrong Zhang,Penghui Wang,Chunxiao Liu,Wei Liu,Dian Jin,Qiong Zhang,Erli Meng,Zhengnan Hu
Abstract:To break through the limitations of pre-training models on fixed categories, Open-Set Object Detection (OSOD) and Open-Set Segmentation (OSS) have attracted a surge of interest from researchers. Inspired by large language models, mainstream OSOD and OSS methods generally utilize text as a prompt, achieving remarkable performance. Following SAM paradigm, some researchers use visual prompts, such as points, boxes, and masks that cover detection or segmentation targets. Despite these two prompt paradigms exhibit excellent performance, they also reveal inherent limitations. On the one hand, it is difficult to accurately describe characteristics of specialized category using textual description. On the other hand, existing visual prompt paradigms heavily rely on multi-round human interaction, which hinders them being applied to fully automated pipeline. To address the above issues, we propose a novel prompt paradigm in OSOD and OSS, that is, \textbf{Image Prompt Paradigm}. This brand new prompt paradigm enables to detect or segment specialized categories without multi-round human intervention. To achieve this goal, the proposed image prompt paradigm uses just a few image instances as prompts, and we propose a novel framework named \textbf{MI Grounding} for this new paradigm. In this framework, high-quality image prompts are automatically encoded, selected and fused, achieving the single-stage and non-interactive inference. We conduct extensive experiments on public datasets, showing that MI Grounding achieves competitive performance on OSOD and OSS benchmarks compared to text prompt paradigm methods and visual prompt paradigm methods. Moreover, MI Grounding can greatly outperform existing method on our constructed specialized ADR50K dataset.
What problem does this paper attempt to address?
The problem that this paper attempts to solve is the limitations of existing open - set object detection (OSOD) and open - set segmentation (OSS) methods when dealing with specific categories. Specifically, the two main existing prompting paradigms - the text - prompting paradigm and the visual - prompting paradigm - have the following problems:
1. **Limitations of the text - prompting paradigm**:
- It is difficult to accurately describe the visual features of specific categories. For example, in X - ray defect detection, some professional terms such as "shrinkage porosity", "dent" and "porosity" are difficult to convey their visual characteristics such as shape, size and texture through text descriptions.
- There may be a mismatch between the text description and the complex visual scene.
2. **Limitations of the visual - prompting paradigm**:
- It is highly dependent on multi - round human interactions, which hinders its application in a fully automated pipeline. For example, the user needs to manually label or check the bounding boxes to ensure that the target area is covered.
- This interaction process makes a single - stage, fully - automatic reasoning pipeline impossible.
To solve these problems, the paper proposes a new prompting paradigm - the **Image Prompt Paradigm**. This paradigm uses a small number of image instances as prompts and can automatically encode, select and fuse high - quality image prompts, thereby achieving single - stage and non - human - intervention - required reasoning. The paper also proposes a new framework named **MI Grounding** to support this new image - prompt paradigm.
### Main contributions
1. **Proposing the image - prompt paradigm**: Different from the existing text and visual - prompt paradigms, the image - prompt paradigm uses a small number of image instances as prompts and can understand specific categories that are difficult to describe in text in a single - stage and non - interactive way.
2. **Proposing the MI Grounding framework**: This framework uses a small number of image prompts for open - set object detection and segmentation, and introduces an Image Prompt Selection Encoder (IPS encoder) to select and integrate high - quality prompts.
3. **Experimental verification**: This method outperforms the mainstream open - set object detection and segmentation methods on multiple datasets, and achieves significant performance improvements on the specially constructed ADR50K dataset.
### ADR50K dataset
To further verify the superiority of the image - prompt paradigm, the authors created an automatic defect recognition dataset named ADR50K. This dataset contains more than 50,000 X - ray defect detection images and provides classification annotations for three defect types: "dent", "shrinkage porosity" and "porosity". In addition, detection and segmentation annotations for all defects are also provided.
### Experimental results
The comparison experiments on the ADR50K dataset show that MI Grounding outperforms the methods based on text - prompts and visual - prompts in both object detection and instance - segmentation tasks. Especially when dealing with specific - category defects, MI Grounding shows higher precision and robustness.
### Summary
By proposing the image - prompt paradigm and the MI Grounding framework, this paper solves the limitations of existing methods when dealing with specific categories, realizes single - stage, fully - automatic open - set object detection and segmentation, and shows its superior performance in practical application scenarios.