Segment Any Anomaly without Training via Hybrid Prompt Regularization

Yunkang Cao,Xiaohao Xu,Chen Sun,Yuqi Cheng,Zongwei Du,Liang Gao,Weiming Shen

2023-05-18

Abstract:We present a novel framework, i.e., Segment Any Anomaly + (SAA+), for zero-shot anomaly segmentation with hybrid prompt regularization to improve the adaptability of modern foundation models. Existing anomaly segmentation models typically rely on domain-specific fine-tuning, limiting their generalization across countless anomaly patterns. In this work, inspired by the great zero-shot generalization ability of foundation models like Segment Anything, we first explore their assembly to leverage diverse multi-modal prior knowledge for anomaly localization. For non-parameter foundation model adaptation to anomaly segmentation, we further introduce hybrid prompts derived from domain expert knowledge and target image context as regularization. Our proposed SAA+ model achieves state-of-the-art performance on several anomaly segmentation benchmarks, including VisA, MVTec-AD, MTD, and KSDD2, in the zero-shot setting. We will release the code at \href{<a class="link-external link-https" href="https://github.com/caoyunkang/Segment-Any-Anomaly" rel="external noopener nofollow">this https URL</a>}{<a class="link-external link-https" href="https://github.com/caoyunkang/Segment-Any-Anomaly" rel="external noopener nofollow">this https URL</a>}.

Computer Vision and Pattern Recognition,Artificial Intelligence

What problem does this paper attempt to address?

### What problem does this paper attempt to solve?

This paper aims to address the problem of Zero-Shot Anomaly Segmentation (ZSAS). Specifically:

1. **Background and Challenges**:
- Existing anomaly segmentation models typically rely on domain-specific fine-tuning, which limits their generalization ability across numerous anomaly patterns.
- In fields such as industrial quality control and medical diagnostics, reliable anomaly segmentation requires distinguishing between the distributions of anomalous and normal data.
- In situations where training data is scarce, many research efforts focus on unsupervised or self-supervised anomaly segmentation methods.

2. **Objectives**:
- Propose a new framework, Segment Any Anomaly + (SAA +), to improve the adaptability of modern foundational models through hybrid prompt regularization.
- Utilize the strong zero-shot generalization capabilities of foundational models (e.g., Segment Anything) and explore how to assemble these models to leverage multimodal prior knowledge for anomaly localization.
- Introduce hybrid prompts derived from domain expert knowledge and target image context as a regularization means to enhance the accuracy of anomaly segmentation.

3. **Main Contributions**:
- Proposed the SAA framework, allowing various foundational models to be collaboratively assembled without training.
- Introduced hybrid prompts as a regularization technique, combining domain expert knowledge and target image context to adapt to anomaly segmentation tasks.
- Achieved state-of-the-art performance on multiple benchmark datasets, including VisA, MVTec-AD, KSDD2, and MTD.

Through these methods, the paper addresses the limitations of existing methods in zero-shot settings and significantly improves the effectiveness of anomaly segmentation.