Abstract:As business of Alibaba expands across the world among various industries, higher standards are imposed on the service quality and reliability of big data cloud computing platforms which constitute the infrastructure of Alibaba Cloud. However, root cause analysis in these platforms is non-trivial due to the complicated system architecture. In this paper, we propose a root cause analysis framework called CloudRCA which makes use of heterogeneous multi-source data including Key Performance Indicators (KPIs), logs, as well as topology, and extracts important features via state-of-the-art anomaly detection and log analysis techniques. The engineered features are then utilized in a Knowledge-informed Hierarchical Bayesian Network (KHBN) model to infer root causes with high accuracy and efficiency. Ablation study and comprehensive experimental comparisons demonstrate that, compared to existing frameworks, CloudRCA 1) consistently outperforms existing approaches in f1-score across different cloud systems; 2) can handle novel types of root causes thanks to the hierarchical structure of KHBN; 3) performs more robustly with respect to algorithmic configurations; and 4) scales more favorably in the data and feature sizes. Experiments also show that a cross-platform transfer learning mechanism can be adopted to further improve the accuracy by more than 10%. CloudRCA has been integrated into the diagnosis system of Alibaba Cloud and employed in three typical cloud computing platforms including MaxCompute, Realtime Compute and Hologres. It saves Site Reliability Engineers (SREs) more than 20% in the time spent on resolving failures in the past twelve months and improves service reliability significantly.

Hierarchical Diagnostic Approach for Performance Problems in Cloud Computing Platforms

Magnifier: Online Detection of Performance Problems in Large-Scale Cloud Computing Systems

Visual Analysis of Cloud Computing Performance Using Behavioral Lines

Toward Fine-Grained, Unsupervised, Scalable Performance Diagnosis for Production Cloud Computing Systems

Localizing Root Causes of Performance Anomalies in Cloud Computing Systems by Analyzing Request Trace Logs

Examination Data Analysis and Evaluation Platform Based on Cloud Computing

CloudRCA: A Root Cause Analysis Framework for Cloud Computing Platforms

Root Cause Analysis of Anomalies of Multitier Services in Public Clouds.

CloudDet: Interactive Visual Analysis of Anomalous Performances in Cloud Computing Systems

A fine-grained robust performance diagnosis framework for run-time cloud applications

Distributed Predictive Modeling Framework for Prediction and Diagnosis of Key Performance Index in Plant-Wide Processes

Hierarchical Process Diagnosis Analysis and Its Application

Anomaly detecting and ranking of the cloud computing platform by multi-view learning

Diagnosing Performance Issues for Large-Scale Microservice Systems with Heterogeneous Graph

Using Hardware Counter-Based Performance Model to Diagnose Scaling Issues of HPC Applications.

P-Tracer: Path-Based Performance Profiling in Cloud Computing Systems

Hierarchical Fault Diagnosis Model of a Large-scale Power System Based on Dispatch Integrated Information Platform

Automatic Detecting Performance Bugs in Cloud Computing Systems via Learning Latency Specification Model

A Causality Mining and Knowledge Graph Based Method of Root Cause Diagnosis for Performance Anomaly in Cloud Applications

Diagnosing Performance Issues in Microservices with Heterogeneous Data Source

A Real-Time Trace-Level Root-Cause Diagnosis System in Alibaba Datacenters