Abstract:Key Performance Indicators (KPIs) are essential time-series metrics for ensuring the reliability and stability of many software systems. They faithfully record runtime states to facilitate the understanding of anomalous system behaviors and provide informative clues for engineers to pinpoint the root causes. The unprecedented scale and complexity of modern software systems, however, make the volume of KPIs explode. Consequently, many traditional methods of KPI anomaly detection become impractical, which serves as a catalyst for the fast development of machine learning-based solutions in both academia and industry. However, there is currently a lack of rigorous comparison among these KPI anomaly detection methods, and re-implementation demands a non-trivial effort. Moreover, we observe that different works adopt independent evaluation processes with different metrics. Some of them may not fully reveal the capability of a model and some are creating an illusion of progress. To better understand the characteristics of different KPI anomaly detectors and address the evaluation issue, in this paper, we provide a comprehensive review and evaluation of twelve state-of-the-art methods, and propose a novel metric called salience. Particularly, the selected methods include five traditional machine learning-based methods and seven deep learning-based methods. These methods are evaluated with five multivariate KPI datasets that are publicly available. A unified toolkit with easy-to-use interfaces is also released. We report the benchmark results in terms of accuracy, salience, efficiency, and delay, which are of practical importance for industrial deployment. We believe our work can contribute as a basis for future academic research and industrial application.

Enhanced KPI Anomaly Detection: An Unsupervised Hybrid Model with Dynamic Threshold.

Robust and Unsupervised KPI Anomaly Detection Based on Highly Sensitive Conditional Variational Auto-Encoders.

Joint Learning-Based Anomaly Detection on KPI Data

UDTL: Anomaly Detection Based on Unsupervised Deep Transfer Learning

KPI-TSAD: A Time-Series Anomaly Detector for KPI Monitoring in Cloud Applications.

Little Help Makes a Big Difference: Leveraging Active Learning to Improve Unsupervised Time Series Anomaly Detection

Long Short Term Memory Networks Based Anomaly Detection for KPIs

Accurate Anomaly Detection Leveraging Knowledge-enhanced GAT

MTAD: Tools and Benchmarks for Multivariate Time Series Anomaly Detection

Rapid Deployment of Anomaly Detection Models for Large Number of Emerging KPI Streams.

Unsupervised Online Anomaly Detection with Parameter Adaptation for KPI Abrupt Changes.

A General KPI Anomaly Detection Using Attention Models

Anomaly Detection Based on Mining Six Local Data Features and BP Neural Network.

A Multivariate KPIs Anomaly Detection Framework with Dynamic Balancing Loss Training

Intelligent Detection for Key Performance Indicators in Industrial-Based Cyber-Physical Systems

Dynamic Self-Correcting Key Performance Indicator Anomaly Detection Algorithm

Adaptive Thresholding Heuristic for KPI Anomaly Detection

Research of Anomaly detection based on Dynamic Anomaly Detection Enhancement Framework

Large-Scale KPI Anomaly Detection Based on Ensemble Learning and Clustering

Effect Improved for High-Dimensional and Unbalanced Data Anomaly Detection Model Based on KNN-SMOTE-LSTM

Probabilistic Temporal Fusion Transformers for Large-Scale KPI Anomaly Detection