Fast Outage Analysis of Large-scale Production Clouds with Service Correlation Mining
Yaohui Wang,Guozheng Li,Zijian Wang,Yu Kang,Yangfan Zhou,Hongyu Zhang,Feng Gao,Jeffrey Sun,Li Yang,Pochian Lee,Zhangwei Xu,Pu Zhao,Bo Qiao,Liqun Li,Xu Zhang,Qingwei Lin
DOI: https://doi.org/10.1109/icse43902.2021.00085
2021-01-01
Abstract:Cloud-based services are surging into popularity in recent years. However, outages, i.e., severe incidents that always impact multiple services, can dramatically affect user experience and incur severe economic losses. Locating the root-cause service, i.e., the service that contains the root cause of the outage, is a crucial step to mitigate the impact of the outage. In current industrial practice, this is generally performed in a bootstrap manner and largely depends on human efforts: the service that directly causes the outage is identified first, and the suspected root cause is traced back manually from service to service during diagnosis until the actual root cause is found. Unfortunately, production cloud systems typically contain a large number of interdependent services. Such a manual root cause analysis is often time-consuming and labor-intensive. In this work, we propose COT, the first outage triage approach that considers the global view of service correlations. COT mines the correlations among services from outage diagnosis data. After learning from historical outages, COT can infer the root cause of emerging ones accurately. We implement COT and evaluate it on a real-world dataset containing one year of data collected from Microsoft Azure, one of the representative cloud computing platforms in the world. Our experimental results show that COT can reach a triage accuracy of 82.1%~83.5%, which outperforms the state-of-the-art triage approach by 28.0%~29.7%.