Abstract:Key Performance Indicators (KPIs) are essential time-series metrics for ensuring the reliability and stability of many software systems. They faithfully record runtime states to facilitate the understanding of anomalous system behaviors and provide informative clues for engineers to pinpoint the root causes. The unprecedented scale and complexity of modern software systems, however, make the volume of KPIs explode. Consequently, many traditional methods of KPI anomaly detection become impractical, which serves as a catalyst for the fast development of machine learning-based solutions in both academia and industry. However, there is currently a lack of rigorous comparison among these KPI anomaly detection methods, and re-implementation demands a non-trivial effort. Moreover, we observe that different works adopt independent evaluation processes with different metrics. Some of them may not fully reveal the capability of a model and some are creating an illusion of progress. To better understand the characteristics of different KPI anomaly detectors and address the evaluation issue, in this paper, we provide a comprehensive review and evaluation of twelve state-of-the-art methods, and propose a novel metric called salience. Particularly, the selected methods include five traditional machine learning-based methods and seven deep learning-based methods. These methods are evaluated with five multivariate KPI datasets that are publicly available. A unified toolkit with easy-to-use interfaces is also released. We report the benchmark results in terms of accuracy, salience, efficiency, and delay, which are of practical importance for industrial deployment. We believe our work can contribute as a basis for future academic research and industrial application.

Label-Less: A Semi-Automatic Labelling Tool for KPI Anomalies.

Robust and Unsupervised KPI Anomaly Detection Based on Highly Sensitive Conditional Variational Auto-Encoders.

Robust KPI Anomaly Detection for Large-Scale Software Services with Partial Labels

AutoKAD: Empowering KPI Anomaly Detection with Label-Free Deployment

LabelEase: A Semi-Automatic Tool for Efficient and Accurate Trace Labeling in Microservices

Intelligent Detection for Key Performance Indicators in Industrial-Based Cyber-Physical Systems

Towards Efficient Pixel Labeling for Industrial Anomaly Detection and Localization

Rapid Deployment of Anomaly Detection Models for Large Number of Emerging KPI Streams.

Practical and White-Box Anomaly Detection through Unsupervised and Active Learning

Label-Efficient Interactive Time-Series Anomaly Detection

MTAD: Tools and Benchmarks for Multivariate Time Series Anomaly Detection

Opprentice: Towards Practical And Automatic Anomaly Detection Through Machine Learning

Little Help Makes a Big Difference: Leveraging Active Learning to Improve Unsupervised Time Series Anomaly Detection

An Interactive Visual Analytics Approach for Network Anomaly Detection Through Smart Labeling

Robust and Rapid Clustering of KPIs for Large-Scale Anomaly Detection

MAPL: Memory Augmentation and Pseudo-Labeling for Semi-Supervised Anomaly Detection

Learning thresholds for PV change detection from operators' labels.

LabelSens: Enabling Real-time Sensor Data Labelling at the point of Collection on Edge Computing

Label Critic: Design Data Before Models

Learning Image Labels On-the-fly for Training Robust Classification Models

Label Smarter, Not Harder: CleverLabel for Faster Annotation of Ambiguous Image Classification with Higher Quality