Exploring the Integration of Large Language Models in Industrial Test Maintenance Processes

Ludvig Lemner,Linnea Wahlgren,Gregory Gay,Nasser Mohammadiha,Jingxiong Liu,Joakim Wennerberg

2024-09-10

Abstract:Much of the cost and effort required during the software testing process is invested in performing test maintenance - the addition, removal, or modification of test cases to keep the test suite in sync with the system-under-test or to otherwise improve its quality. Tool support could reduce the cost - and improve the quality - of test maintenance by automating aspects of the process or by providing guidance and support to developers. In this study, we explore the capabilities and applications of large language models (LLMs) - complex machine learning models adapted to textual analysis - to support test maintenance. We conducted a case study at Ericsson AB where we explored the triggers that indicate the need for test maintenance, the actions that LLMs can take, and the considerations that must be made when deploying LLMs in an industrial setting. We also proposed and demonstrated implementations of two multi-agent architectures that can predict which test cases require maintenance following a change to the source code. Collectively, these contributions advance our theoretical and practical understanding of how LLMs can be deployed to benefit industrial test maintenance processes.

Software Engineering,Artificial Intelligence,Machine Learning

What problem does this paper attempt to address?

The paper attempts to address the issue of how to leverage large language models (LLMs) to support and automate test maintenance during the software testing process. Specifically, the paper focuses on the following aspects:

1. **Identifying Test Maintenance Triggers**: Investigating which changes in the source code or development process trigger the need for test maintenance. These changes may include specific code modifications, requirement changes, or updates to other related artifacts.

2. **Exploring the Application of LLMs**: Examining the potential applications of the current generation of LLMs or LLM agents in the test maintenance process. This includes:
- Which triggers can be handled by LLMs or LLM agents.
- What effective test maintenance actions LLMs or LLM agents can perform based on these triggers.
- Factors to consider when deploying LLMs in an industrial environment.

3. **Evaluating Prototype Performance**: Designing and implementing prototypes of two multi-LLM architectures to predict which test cases need maintenance after source code changes, and evaluating the performance of these prototypes, including precision, recall, and F1 score.

Through these research questions, the paper aims to advance the theoretical and practical understanding of how to integrate LLMs into the industrial test maintenance process, thereby reducing the need for manual test maintenance and improving the quality and efficiency of test maintenance.