Abstract:Debugging performance anomalies in real-world databases is challenging. Causal inference techniques enable qualitative and quantitative root cause analysis of performance downgrade. Nevertheless, causality analysis is practically challenging, particularly due to limited observability. Recently, chaos engineering has been applied to test complex real-world software systems. Chaos frameworks like Chaos Mesh mutate a set of chaos variables to inject catastrophic events (e.g., network slowdowns) to "stress" software systems. The systems under chaos stress are then tested using methods like differential testing to check if they retain their normal functionality (e.g., SQL query output is always correct under stress). Despite its ubiquity in the industry, chaos engineering is now employed mostly to aid software testing rather for performance debugging. This paper identifies novel usage of chaos engineering on helping developers diagnose performance anomalies in databases. Our presented framework, PERFCE, comprises an offline phase and an online phase. The offline phase learns the statistical models of the target database system, whilst the online phase diagnoses the root cause of monitored performance anomalies on the fly. During the offline phase, PERFCE leverages both passive observations and proactive chaos experiments to constitute accurate causal graphs and structural equation models (SEMs). When observing performance anomalies during the online phase, causal graphs enable qualitative root cause identification (e.g., high CPU usage) and SEMs enable quantitative counterfactual analysis (e.g., determining "when CPU usage is reduced to 45\%, performance returns to normal"). PERFCE notably outperforms prior works on common synthetic datasets, and our evaluation on real-world databases, MySQL and TiDB, shows that PERFCE is highly accurate and moderately expensive.

FluxInfer: Automatic Diagnosis of Performance Anomaly for Online Database System

Schema-Driven Performance Evaluation for Highly Concurrent Scenarios.

A DATA FUSION METHOD FOR ON-LINE PERFORMANCE CALCULATION OF TURBOGENERATOR

Distance Based Root Cause Analysis and Change Impact Analysis of Performance Regressions

FluxRank: A Widely-Deployable Framework to Automatically Localizing Root Cause Machines for Software Service Failure Mitigation

CoFlux

Generic and Robust Performance Diagnosis Via Causal Inference for OLTP Database Systems

A fine-grained robust performance diagnosis framework for run-time cloud applications

Diagnosing Root Causes of Intermittent Slow Queries in Cloud Databases

Enhancing Online Index Tuning with a Learned Tuning Diagnostic.

Detecting Performance Anomaly with Correlation Analysis for Internetware

Online Anomaly Detection in HPC Systems

PerfCE: Performance Debugging on Databases with Chaos Engineering-Enhanced Causality Analysis

DBPA: A Benchmark for Transactional Database Performance Anomalies.

Adaptive Performance Anomaly Detection for Online Service Systems via Pattern Sketching

Rapid Deployment of Anomaly Detection Models for Large Number of Emerging KPI Streams.

#FluxFlow: Visual Analysis of Anomalous Information Spreading on Social Media

HotSpot: Anomaly Localization for Additive KPIs with Multi-Dimensional Attributes.

Practical Anomaly Detection over Multivariate Monitoring Metrics for Online Services

Performance anomaly detection using isolation‐trees in heterogeneous workloads of web applications in computing clouds

SubAnom: Efficient Subgraph Anomaly Detection Framework over Dynamic Graphs