Abstract:Key Performance Indicators (KPIs) are essential time-series metrics for ensuring the reliability and stability of many software systems. They faithfully record runtime states to facilitate the understanding of anomalous system behaviors and provide informative clues for engineers to pinpoint the root causes. The unprecedented scale and complexity of modern software systems, however, make the volume of KPIs explode. Consequently, many traditional methods of KPI anomaly detection become impractical, which serves as a catalyst for the fast development of machine learning-based solutions in both academia and industry. However, there is currently a lack of rigorous comparison among these KPI anomaly detection methods, and re-implementation demands a non-trivial effort. Moreover, we observe that different works adopt independent evaluation processes with different metrics. Some of them may not fully reveal the capability of a model and some are creating an illusion of progress. To better understand the characteristics of different KPI anomaly detectors and address the evaluation issue, in this paper, we provide a comprehensive review and evaluation of twelve state-of-the-art methods, and propose a novel metric called salience. Particularly, the selected methods include five traditional machine learning-based methods and seven deep learning-based methods. These methods are evaluated with five multivariate KPI datasets that are publicly available. A unified toolkit with easy-to-use interfaces is also released. We report the benchmark results in terms of accuracy, salience, efficiency, and delay, which are of practical importance for industrial deployment. We believe our work can contribute as a basis for future academic research and industrial application.

Survey and Benchmark of Anomaly Detection in Business Processes

DBPA: A Benchmark for Transactional Database Performance Anomalies.

An empirical evaluation of process mining algorithms based on structural and behavioral similarities.

MTAD: Tools and Benchmarks for Multivariate Time Series Anomaly Detection

Experimental Comparison and Survey of Twelve Time Series Anomaly Detection Algorithms

ADBench: Anomaly Detection Benchmark

On Recommendation of Process Mining Algorithms

Survey and cross-benchmark comparison of remaining time prediction methods in business process monitoring

Efficient Selection of Process Mining Algorithms

Practical Anomaly Detection over Multivariate Monitoring Metrics for Online Services

Multi-perspective Anomaly Detection in Business Process Execution Events

Anomaly Detection Based on Isolation Mechanisms: A Survey

A Benchmark Dataset for Evaluating Process Similarity Search Methods

Benchmarking Anomaly Detection Methods: Insights From the UCR Time Series Anomaly Archive

Anomaly Correction of Business Processes Using Transformer Autoencoder

Can Tree Based Approaches Surpass Deep Learning in Anomaly Detection? A Benchmarking Study

Data-driven anomaly monitoring methods for multimode processes:overview and perspectives

Detecting Data-Model-Oriented Anomalies In Parallel Business Process

A Survey: Industrial Anomaly Detection based on Data Mining

Kernel Based Sequential Data Anomaly Detection in Business Process Event Logs

DABL: Detecting Semantic Anomalies in Business Processes Using Large Language Models