Abstract:Visual anomaly detection aims to identify anomalous regions in images through unsupervised learning paradigms, with increasing application demand and value in fields such as industrial inspection and medical lesion detection. Despite significant progress in recent years, there is a lack of comprehensive benchmarks to adequately evaluate the performance of various mainstream methods across different datasets under the practical multi-class setting. The absence of standardized experimental setups can lead to potential biases in training epochs, resolution, and metric results, resulting in erroneous conclusions. This paper addresses this issue by proposing a comprehensive visual anomaly detection benchmark, ADer, which is a modular framework that is highly extensible for new methods. The benchmark includes multiple datasets from industrial and medical domains, implementing fifteen state-of-the-art methods and nine comprehensive metrics. Additionally, we have proposed the GPU-assisted ADEval package to address the slow evaluation problem of metrics like time-consuming mAU-PRO on large-scale data, significantly reducing evaluation time by more than \textit{1000-fold}. Through extensive experimental results, we objectively reveal the strengths and weaknesses of different methods and provide insights into the challenges and future directions of multi-class visual anomaly detection. We hope that ADer will become a valuable resource for researchers and practitioners in the field, promoting the development of more robust and generalizable anomaly detection systems. Full codes are open-sourced at <a class="link-external link-https" href="https://github.com/zhangzjn/ader" rel="external noopener nofollow">this https URL</a>.

What problem does this paper attempt to address?

The main problem that this paper attempts to solve is the lack of a comprehensive and standardized benchmarking framework in the current field of multi - class visual anomaly detection, which leads to biased performance evaluations of different methods on different datasets. Specifically: 1. **Insufficient Datasets and Evaluation Metrics**: Most of the existing research uses small - scale datasets and does not have enough evaluation metrics to comprehensively assess model performance. This makes the comparison between different methods less fair. 2. **Non - uniform Training Settings**: Different studies use different training settings (such as the number of training epochs, resolution, etc.), which may lead to biases in evaluation results and thus draw wrong conclusions. 3. **Low Evaluation Efficiency**: Some complex evaluation metrics (such as mAU - PRO) are very time - consuming when evaluating on large - scale datasets, limiting the feasibility of detailed evaluation. To solve these problems, the paper proposes a comprehensive benchmarking library named ADer, and its main contributions include: 1. **Comprehensive Benchmarking Library**: The ADer library implements 15 state - of - the - art anomaly detection methods and evaluates them on 11 popular datasets, covering industrial, medical, and general fields. At the same time, ADer adopts 9 comprehensive evaluation metrics to ensure a comprehensive and unbiased evaluation of the performance of each method. 2. **GPU - Assisted Evaluation Toolkit**: Developed and open - sourced the ADEval toolkit, which significantly improves the evaluation speed of complex evaluation metrics, especially for large - scale datasets, reducing the evaluation time by more than 1,000 times. 3. **Extensive Experimental Analysis**: Objectively reveals the advantages and disadvantages of different visual anomaly detection methods through a large number of experiments and provides valuable insights into future research directions. 4. **Open - source Resources**: Fully open - sourced the ADer code, making it an important resource for researchers and practitioners and promoting the further development of this field. In conclusion, this paper aims to promote the development and application of multi - class visual anomaly detection technology by establishing a comprehensive, fair, and efficient benchmarking framework.

A Comprehensive Library for Benchmarking Multi-class Visual Anomaly Detection

A Diffusion-Based Framework for Multi-Class Anomaly Detection

BMAD: Benchmarks for Medical Anomaly Detection

MedIAnomaly: A comparative study of anomaly detection in medical images

AnomalyNCD: Towards Novel Anomaly Class Discovery in Industrial Scenarios

IM-IAD: Industrial Image Anomaly Detection Benchmark in Manufacturing

ADBench: Anomaly Detection Benchmark

A Survey of Visual Sensory Anomaly Detection

Multiview Deep Anomaly Detection: A Systematic Exploration

UMAD: University of Macau Anomaly Detection Benchmark Dataset

A Survey on Visual Anomaly Detection: Challenge, Approach, and Prospect

Advancing Video Anomaly Detection: A Concise Review and a New Dataset

Learning Feature Inversion for Multi-class Anomaly Detection under General-purpose COCO-AD Benchmark

Learning Multi-view Anomaly Detection

UniVAD: A Training-free Unified Model for Few-shot Visual Anomaly Detection

A New Comprehensive Benchmark for Semi-supervised Video Anomaly Detection and Anticipation

Are Anomaly Scores Telling the Whole Story? A Benchmark for Multilevel Anomaly Detection

Towards Continual Adaptation in Industrial Anomaly Detection

You Only Train Once: Learning a General Anomaly Enhancement Network with Random Masks for Hyperspectral Anomaly Detection

Supervised Anomaly Detection for Complex Industrial Images