Agent-Driven Automatic Software Improvement

Fernando Vallecillos Ruiz

DOI: https://doi.org/10.1145/3661167.3661171

2024-06-24

Abstract:With software maintenance accounting for 50% of the cost of developing software, enhancing code quality and reliability has become more critical than ever. In response to this challenge, this doctoral research proposal aims to explore innovative solutions by focusing on the deployment of agents powered by Large Language Models (LLMs) to perform software maintenance tasks. The iterative nature of agents, which allows for continuous learning and adaptation, can help surpass common challenges in code generation. One distinct challenge is the last-mile problems, errors at the final stage of producing functionally and contextually relevant code. Furthermore, this project aims to surpass the inherent limitations of current LLMs in source code through a collaborative framework where agents can correct and learn from each other's errors. We aim to use the iterative feedback in these systems to further fine-tune the LLMs underlying the agents, becoming better aligned to the task of automated software improvement. Our main goal is to achieve a leap forward in the field of automatic software improvement by developing new tools and frameworks that can enhance the efficiency and reliability of software development.

Software Engineering

What problem does this paper attempt to address?

The problem that this paper attempts to solve is the improvement of efficiency and quality in software maintenance and enhancement. Specifically, the author points out that software maintenance accounts for 50% of the cost of software development, and as the amount of code increases, there are more and more errors, vulnerabilities, and inefficiencies in the software. These problems not only affect performance and user experience but also bring economic and security risks.

To solve these problems, the paper proposes an innovative method: using intelligent agents based on large - language models (LLMs) for automatic software improvement. Through this method, the research aims to achieve the following goals:

1. **Improve code quality and reliability**: Through iterative learning and adaptability, intelligent agents can continuously improve the quality of code generation and reduce errors and vulnerabilities.
2. **Solve the "last - mile" problem**: That is, the functional and context - related errors that occur in the final stage of code generation.
3. **Surpass the limitations of existing LLMs**: Through a multi - agent collaboration framework, agents can correct and learn from each other, thereby overcoming the limitations of existing LLMs in source - code processing.
4. **Optimize automated software - improvement tools**: Develop new tools and frameworks to improve the efficiency and reliability of software development.

### Main contributions of the paper

1. **Single - agent system vs. one - time LLM use**: Research whether a single - agent system can outperform a one - time - used LLM in code - improvement tasks.
2. **Synergistic effect of multi - agent systems**: Explore the collaborative working effect of multiple agents in code - improvement tasks, especially how to find the optimal solution through the interaction of multi - expert systems in complex scenarios.
3. **Fine - tuning techniques in the iterative process**: Develop new fine - tuning techniques and methods to continuously optimize the capabilities of agents through the iterative process of code generation and revision.

### Research challenges

- **Selection of experimental design and evaluation metrics**: It is necessary to establish a robust experimental environment and evaluation metrics to fairly compare the performance of single - agent systems and one - time LLMs.
- **Coordination and evaluation of multi - agent systems**: The complexity of multi - agent systems makes their coordination and performance evaluation more challenging.
- **Datasets and methods for iterative fine - tuning**: Create datasets that integrate the iterative process of generation and revision, and develop and validate new fine - tuning techniques.

Through these studies, the paper hopes to provide new tools and methods for the field of automatic software improvement, thereby significantly improving the efficiency and reliability of software development.