Abstract:Debugging performance anomalies in real-world databases is challenging. Causal inference techniques enable qualitative and quantitative root cause analysis of performance downgrade. Nevertheless, causality analysis is practically challenging, particularly due to limited observability. Recently, chaos engineering has been applied to test complex real-world software systems. Chaos frameworks like Chaos Mesh mutate a set of chaos variables to inject catastrophic events (e.g., network slowdowns) to "stress" software systems. The systems under chaos stress are then tested using methods like differential testing to check if they retain their normal functionality (e.g., SQL query output is always correct under stress). Despite its ubiquity in the industry, chaos engineering is now employed mostly to aid software testing rather for performance debugging. This paper identifies novel usage of chaos engineering on helping developers diagnose performance anomalies in databases. Our presented framework, PERFCE, comprises an offline phase and an online phase. The offline phase learns the statistical models of the target database system, whilst the online phase diagnoses the root cause of monitored performance anomalies on the fly. During the offline phase, PERFCE leverages both passive observations and proactive chaos experiments to constitute accurate causal graphs and structural equation models (SEMs). When observing performance anomalies during the online phase, causal graphs enable qualitative root cause identification (e.g., high CPU usage) and SEMs enable quantitative counterfactual analysis (e.g., determining "when CPU usage is reduced to 45\%, performance returns to normal"). PERFCE notably outperforms prior works on common synthetic datasets, and our evaluation on real-world databases, MySQL and TiDB, shows that PERFCE is highly accurate and moderately expensive.

Chaos Engineering

Chaos Engineering: A Multi-Vocal Literature Review

A Platform for Automating Chaos Experiments

Automating chaos experiments in production

Application Research and Discussing of Detective Systems Based on Chaos Theory

Chaos Engineering for Enhanced Resilience of Cyber-Physical Systems

Chaos Engineering of Ethereum Blockchain Clients

On Evaluating Self-Adaptive and Self-Healing Systems using Chaos Engineering

CHESS: A Framework for Evaluation of Self-adaptive Systems based on Chaos Engineering

Chaos as a Software Product Line—A platform for improving open hybrid‐cloud systems resiliency

Abstraction Engineering

Observability and Chaos Engineering on System Calls for Containerized Applications in Docker

SciOps: Achieving Productivity and Reliability in Data-Intensive Research

Chaos-model Based Framework for Embedded Software Development

PerfCE: Performance Debugging on Databases with Chaos Engineering-Enhanced Causality Analysis

Reliability and Fault-Tolerance by Choreographic Design

Enhancing DevSecOps practice with Large Language Models and Security Chaos Engineering

From Lemons to Peaches: Improving Security ROI through Security Chaos Engineering

Automated metrics calculation in a dynamic heterogeneous environment

Application of Chaos in Chemical Process Optimization

The Essence Theory of Software Engineering - Large-Scale Classroom Experiences from 450+ Software Engineering BSc Students