MAGIS: LLM-Based Multi-Agent Framework for GitHub Issue Resolution

Wei Tao,Yucheng Zhou,Yanlin Wang,Wenqiang Zhang,Hongyu Zhang,Yu Cheng

2024-06-27

Abstract:In software development, resolving the emergent issues within GitHub repositories is a complex challenge that involves not only the incorporation of new code but also the maintenance of existing code. Large Language Models (LLMs) have shown promise in code generation but face difficulties in resolving Github issues, particularly at the repository level. To overcome this challenge, we empirically study the reason why LLMs fail to resolve GitHub issues and analyze the major factors. Motivated by the empirical findings, we propose a novel LLM-based Multi-Agent framework for GitHub Issue reSolution, MAGIS, consisting of four agents customized for software evolution: Manager, Repository Custodian, Developer, and Quality Assurance Engineer agents. This framework leverages the collaboration of various agents in the planning and coding process to unlock the potential of LLMs to resolve GitHub issues. In experiments, we employ the SWE-bench benchmark to compare MAGIS with popular LLMs, including GPT-3.5, GPT-4, and Claude-2. MAGIS can resolve 13.94% GitHub issues, significantly outperforming the baselines. Specifically, MAGIS achieves an eight-fold increase in resolved ratio over the direct application of GPT-4, the advanced LLM.

Software Engineering,Artificial Intelligence

What problem does this paper attempt to address?

### The Problem This Paper Attempts to Solve This paper primarily focuses on how to effectively address issues in GitHub repositories during the software development process. Specifically, it attempts to solve the following problems: 1. **Limitations of Existing LLMs in Solving GitHub Issues**: - Large Language Models (LLMs) perform well in code generation but struggle with solving GitHub issues at the repository level. - Existing LLMs find it difficult to handle long context inputs, especially when multiple files need to be modified. 2. **Increasing the Success Rate of Solving GitHub Issues**: - Currently, the success rate of LLMs in solving GitHub issues is very low, even when provided with the file paths that need modification. - The paper proposes a new multi-agent framework called MAGIS to improve the efficiency of solving GitHub issues. 3. **Optimizing the Code Modification Process**: - By analyzing factors that affect LLMs in solving GitHub issues, such as locating files, line numbers, and code complexity, the paper proposes an improved method to better address these challenges. ### Summary This paper aims to overcome the limitations of existing LLMs in solving GitHub issues by introducing a multi-agent framework called MAGIS, thereby significantly increasing the success rate of solving these issues. Experimental results show that MAGIS outperforms other existing LLM models in solving GitHub issues.

MAGIS: LLM-Based Multi-Agent Framework for GitHub Issue Resolution

MAgIC: Investigation of Large Language Model Powered Multi-Agent in Cognition, Adaptability, Rationality and Collaboration

GitAgent: Facilitating Autonomous Agent with GitHub by Tool Extension

Lingma SWE-GPT: An Open Development-Process-Centric Language Model for Automated Software Improvement

Encouraging Divergent Thinking in Large Language Models through Multi-Agent Debate

MetaGPT: Meta Programming for A Multi-Agent Collaborative Framework

ML-Bench: Evaluating Large Language Models and Agents for Machine Learning Tasks on Repository-Level Code

Enhancing LLMs for Power System Simulations: A Feedback-driven Multi-agent Framework

GraphTeam: Facilitating Large Language Model-based Graph Analysis via Multi-Agent Collaboration

MetaGPT: Meta Programming for Multi-Agent Collaborative Framework

STRIDE: A Tool-Assisted LLM Agent Framework for Strategic and Interactive Decision-Making

Multi-Agent Collaboration: Harnessing the Power of Intelligent LLM Agents

An LLM Agent for Automatic Geospatial Data Analysis

MegaAgent: A Practical Framework for Autonomous Cooperation in Large-Scale LLM Agent Systems

MAG-V: A Multi-Agent Framework for Synthetic Data Generation and Verification

On the effectiveness of Large Language Models for GitHub Workflows

Automated Commit Message Generation with Large Language Models: An Empirical Study and Beyond

SMoA: Improving Multi-agent Large Language Models with Sparse Mixture-of-Agents

Enhancing Multi-Agent Consensus through Third-Party LLM Integration: Analyzing Uncertainty and Mitigating Hallucinations in Large Language Models

SciAgent: Tool-augmented Language Models for Scientific Reasoning

A Unified Debugging Approach via LLM-Based Multi-Agent Synergy