From LLMs to LLM-based Agents for Software Engineering: A Survey of Current, Challenges and Future

Haolin Jin,Linghan Huang,Haipeng Cai,Jun Yan,Bo Li,Huaming Chen

2024-08-05

Abstract:With the rise of large language models (LLMs), researchers are increasingly exploring their applications in var ious vertical domains, such as software engineering. LLMs have achieved remarkable success in areas including code generation and vulnerability detection. However, they also exhibit numerous limitations and shortcomings. LLM-based agents, a novel tech nology with the potential for Artificial General Intelligence (AGI), combine LLMs as the core for decision-making and action-taking, addressing some of the inherent limitations of LLMs such as lack of autonomy and self-improvement. Despite numerous studies and surveys exploring the possibility of using LLMs in software engineering, it lacks a clear distinction between LLMs and LLM based agents. It is still in its early stage for a unified standard and benchmarking to qualify an LLM solution as an LLM-based agent in its domain. In this survey, we broadly investigate the current practice and solutions for LLMs and LLM-based agents for software engineering. In particular we summarise six key topics: requirement engineering, code generation, autonomous decision-making, software design, test generation, and software maintenance. We review and differentiate the work of LLMs and LLM-based agents from these six topics, examining their differences and similarities in tasks, benchmarks, and evaluation metrics. Finally, we discuss the models and benchmarks used, providing a comprehensive analysis of their applications and effectiveness in software engineering. We anticipate this work will shed some lights on pushing the boundaries of LLM-based agents in software engineering for future research.

Software Engineering,Artificial Intelligence,Computation and Language

What problem does this paper attempt to address?

### Problems the Paper Attempts to Solve

This paper aims to explore the application of large language models (LLMs) and their derivative agents (LLM-based agents) in the field of software engineering, and to provide a systematic review of current research progress, challenges, and future development directions.

#### Main Objectives
1. **Overview of the Applications of LLMs and LLM-based Agents**:
- Compare the performance differences between LLMs and LLM-based agents in software engineering.
- Analyze their applications in six key areas: requirements engineering, code generation, autonomous decision-making, software design, test generation, and software maintenance.

2. **Evaluation Benchmarks and Metrics**:
- Summarize commonly used datasets and metrics for evaluating the performance of LLMs and LLM-based agents.
- Provide a comprehensive analytical framework to assess the effectiveness and applicability of these technologies in software engineering.

3. **Distinguishing the Capabilities of LLMs and LLM-based Agents**:
- Clearly identify the differences between the two, including task execution capabilities, adaptability, and the ability to utilize external tools.
- Explore how LLM-based agents overcome the limitations of LLMs in context understanding, autonomous learning, etc.

4. **Promoting Future Research Directions**:
- Provide guidance for future research through a comprehensive analysis of existing research results.
- Emphasize the importance of standardized benchmarks to promote the further development of LLM-based agents in software engineering.

#### Research Questions
- **RQ1**: What are the latest technologies and practices of LLMs and LLM-based agents in software engineering?
- **RQ2**: What are the key differences in task execution between LLMs and LLM-based agents in software engineering applications?
- **RQ3**: What are the most commonly used benchmark datasets and evaluation metrics when assessing the performance of LLMs and LLM-based agents in software engineering tasks?
- **RQ4**: What experimental models and methodologies are primarily adopted in software engineering research utilizing LLMs?

Through these questions, the paper aims to reveal the current state of practical applications of LLMs and LLM-based agents in software engineering, providing valuable reference information for researchers and practitioners.