Abstract:The growing complexity of cloud based software systems has resulted in incident management becoming an integral part of the software development lifecycle. Root cause analysis (RCA), a critical part of the incident management process, is a demanding task for on-call engineers, requiring deep domain knowledge and extensive experience with a team's specific services. Automation of RCA can result in significant savings of time, and ease the burden of incident management on on-call engineers. Recently, researchers have utilized Large Language Models (LLMs) to perform RCA, and have demonstrated promising results. However, these approaches are not able to dynamically collect additional diagnostic information such as incident related logs, metrics or databases, severely restricting their ability to diagnose root causes. In this work, we explore the use of LLM based agents for RCA to address this limitation. We present a thorough empirical evaluation of a ReAct agent equipped with retrieval tools, on an out-of-distribution dataset of production incidents collected at Microsoft. Results show that ReAct performs competitively with strong retrieval and reasoning baselines, but with highly increased factual accuracy. We then extend this evaluation by incorporating discussions associated with incident reports as additional inputs for the models, which surprisingly does not yield significant performance improvements. Lastly, we conduct a case study with a team at Microsoft to equip the ReAct agent with tools that give it access to external diagnostic services that are used by the team for manual RCA. Our results show how agents can overcome the limitations of prior work, and practical considerations for implementing such a system in practice.

What problem does this paper attempt to address?

The paper aims to address the issue of incident management caused by the increasing complexity of software systems in cloud services, particularly the most critical part—Root Cause Analysis (RCA). With the proliferation of cloud services and technological advancements, the complexity of software systems continues to rise, making incident management increasingly important and challenging. Root Cause Analysis, as a key step in the incident management process, is crucial for timely and effective problem resolution. However, it requires engineers handling the incidents to have deep domain knowledge and extensive experience to diagnose and identify the root cause of the incidents. To alleviate the workload of manual root cause analysis, researchers have begun exploring automated methods. In recent years, studies based on large language models (LLMs) have shown potential in root cause analysis. Although these methods have achieved initial success, they often fail to dynamically gather additional diagnostic information (such as logs, metrics, or database data), which limits their ability to diagnose the root cause. The goal of this paper is to overcome this limitation by utilizing LLM-based agents that can reason, plan, and interact with the external environment to gather new information. Specifically, the paper explores the following points: 1. **Evaluate whether LLM-based agents can effectively perform root cause analysis without fine-tuning**. To this end, the authors conducted experiments using a real production incident dataset and performed a detailed empirical evaluation of an agent named ReAct. 2. **Investigate whether discussion comments in historical incident reports can help improve the performance of LLM-based methods in root cause analysis**. These discussions may contain diagnostic steps that are beneficial for future incident analysis. 3. **Demonstrate the effectiveness of LLM-based agents in root cause analysis when provided with specialized diagnostic tools used by specific teams**. The authors conducted a practical case study in collaboration with a team from Microsoft to evaluate the agent. In summary, this paper aims to experimentally evaluate the root cause analysis capabilities of LLM-based agents under different conditions and explore their potential and challenges in real-world application scenarios.

Exploring LLM-based Agents for Root Cause Analysis

Exploring LLM-based Agents for Root Cause Analysis

RCAgent: Cloud Root Cause Analysis by Autonomous Agents with Tool-Augmented Large Language Models

Automated Root Causing of Cloud Incidents using In-Context Learning with GPT-4

Microservice Root Cause Analysis with Limited Observability Through Intervention Recognition in the Latent Space

ReST meets ReAct: Self-Improvement for Multi-Step Reasoning LLM Agent

AI based Multiagent Approach for Requirements Elicitation and Analysis

X-lifecycle Learning for Cloud Incident Management using LLMs

mABC: multi-Agent Blockchain-Inspired Collaboration for root cause analysis in micro-services architecture

Multi-Agent Collaboration in Incident Response with Large Language Models

Agent Q: Advanced Reasoning and Learning for Autonomous AI Agents

LEMMA-RCA: A Large Multi-modal Multi-domain Dataset for Root Cause Analysis

ReAct: Synergizing Reasoning and Acting in Language Models

Towards Efficient LLM Grounding for Embodied Multi-Agent Collaboration

TraceDiag: Adaptive, Interpretable, and Efficient Root Cause Analysis on Large-Scale Microservice Systems

ConceptAgent: LLM-Driven Precondition Grounding and Tree Search for Robust Task Planning and Execution

Automated program and software defect root cause analysis using machine learning techniques

LogRCA: Log-based Root Cause Analysis for Distributed Services

On the Brittle Foundations of ReAct Prompting for Agentic Large Language Models

AgentBoard: An Analytical Evaluation Board of Multi-turn LLM Agents

From Facts to Insights: A Study on the Generation and Evaluation of Analytical Reports for Deciphering Earnings Calls