Abstract:Key Performance Indicators (KPIs) are essential time-series metrics for ensuring the reliability and stability of many software systems. They faithfully record runtime states to facilitate the understanding of anomalous system behaviors and provide informative clues for engineers to pinpoint the root causes. The unprecedented scale and complexity of modern software systems, however, make the volume of KPIs explode. Consequently, many traditional methods of KPI anomaly detection become impractical, which serves as a catalyst for the fast development of machine learning-based solutions in both academia and industry. However, there is currently a lack of rigorous comparison among these KPI anomaly detection methods, and re-implementation demands a non-trivial effort. Moreover, we observe that different works adopt independent evaluation processes with different metrics. Some of them may not fully reveal the capability of a model and some are creating an illusion of progress. To better understand the characteristics of different KPI anomaly detectors and address the evaluation issue, in this paper, we provide a comprehensive review and evaluation of twelve state-of-the-art methods, and propose a novel metric called salience. Particularly, the selected methods include five traditional machine learning-based methods and seven deep learning-based methods. These methods are evaluated with five multivariate KPI datasets that are publicly available. A unified toolkit with easy-to-use interfaces is also released. We report the benchmark results in terms of accuracy, salience, efficiency, and delay, which are of practical importance for industrial deployment. We believe our work can contribute as a basis for future academic research and industrial application.

AutoKAD: Empowering KPI Anomaly Detection with Label-Free Deployment

Auto-AD: Autonomous Hyperspectral Anomaly Detection Network Based on Fully Convolutional Autoencoder

Rapid Deployment of Anomaly Detection Models for Large Number of Emerging KPI Streams.

Robust KPI Anomaly Detection for Large-Scale Software Services with Partial Labels

Label-Less: A Semi-Automatic Labelling Tool for KPI Anomalies.

Joint Learning-Based Anomaly Detection on KPI Data

Pre-trained KPI Anomaly Detection Model Through Disentangled Transformer

Unsupervised Online Anomaly Detection with Parameter Adaptation for KPI Abrupt Changes.

Large-Scale KPI Anomaly Detection Based on Ensemble Learning and Clustering

Weakly Supervised Anomaly Detection via Knowledge-Data Alignment

Practical and White-Box Anomaly Detection through Unsupervised and Active Learning

Enhanced KPI Anomaly Detection: An Unsupervised Hybrid Model with Dynamic Threshold.

KPI-TSAD: A Time-Series Anomaly Detector for KPI Monitoring in Cloud Applications.

Accurate Anomaly Detection Leveraging Knowledge-enhanced GAT

MTAD: Tools and Benchmarks for Multivariate Time Series Anomaly Detection

Little Help Makes a Big Difference: Leveraging Active Learning to Improve Unsupervised Time Series Anomaly Detection

Opprentice: Towards Practical And Automatic Anomaly Detection Through Machine Learning

Dynamic Self-Correcting Key Performance Indicator Anomaly Detection Algorithm

ALSR: an Adaptive Label Screening and Relearning Approach for Interval-Oriented Anomaly Detection

Learning to Diagnose: Meta-Learning for Efficient Adaptation in Few-Shot AIOps Scenarios

A Multivariate KPIs Anomaly Detection Framework with Dynamic Balancing Loss Training