Do LLMs Understand Visual Anomalies? Uncovering LLM's Capabilities in Zero-shot Anomaly Detection

Jiaqi Zhu,Shaofeng Cai,Fang Deng,Beng Chin Ooi,Junran Wu

2024-09-10

Abstract:Large vision-language models (LVLMs) are markedly proficient in deriving visual representations guided by natural language. Recent explorations have utilized LVLMs to tackle zero-shot visual anomaly detection (VAD) challenges by pairing images with textual descriptions indicative of normal and abnormal conditions, referred to as anomaly prompts. However, existing approaches depend on static anomaly prompts that are prone to cross-semantic ambiguity, and prioritize global image-level representations over crucial local pixel-level image-to-text alignment that is necessary for accurate anomaly localization. In this paper, we present ALFA, a training-free approach designed to address these challenges via a unified model. We propose a run-time prompt adaptation strategy, which first generates informative anomaly prompts to leverage the capabilities of a large language model (LLM). This strategy is enhanced by a contextual scoring mechanism for per-image anomaly prompt adaptation and cross-semantic ambiguity mitigation. We further introduce a novel fine-grained aligner to fuse local pixel-level semantics for precise anomaly localization, by projecting the image-text alignment from global to local semantic spaces. Extensive evaluations on MVTec and VisA datasets confirm ALFA's effectiveness in harnessing the language potential for zero-shot VAD, achieving significant PRO improvements of 12.1% on MVTec and 8.9% on VisA compared to state-of-the-art approaches.

Computer Vision and Pattern Recognition,Multimedia

What problem does this paper attempt to address?

The paper aims to address two main challenges in Visual Anomaly Detection (VAD): 1) existing methods rely on static anomaly prompts, which can easily lead to cross-semantic ambiguity; 2) these methods focus more on global image-level representations, neglecting the local pixel-level image-to-text alignment that is crucial for precise anomaly localization. To tackle these issues, the paper proposes the ALFA (Adaptive Large Language Model Empowered Zero-Shot Visual Anomaly Detection and Fine-Grained Alignment) model.

Specifically, ALFA addresses the above challenges in the following ways:
1. **Dynamic Prompt Adaptation Strategy**: Generates informative anomaly prompts at runtime and adaptively manages a set of prompts for each query image through a context scoring mechanism, avoiding the extensive time and expertise required for traditional hand-crafted prompts.
2. **Fine-Grained Aligner**: Introduces a novel fine-grained aligner to fuse local pixel-level semantics, enabling precise anomaly localization. This aligner projects image-text alignment from the global semantic space to the local semantic space.

Experimental results show that ALFA significantly outperforms existing state-of-the-art methods on the MVTec and VisA datasets, achieving remarkable performance improvements in zero-shot VAD tasks.