LLM as Runtime Error Handler: A Promising Pathway to Adaptive Self-Healing of Software Systems

Zhensu Sun,Haotian Zhu,Bowen Xu,Xiaoning Du,Li Li,David Lo
2024-08-02
Abstract:Unanticipated runtime errors, lacking predefined handlers, can abruptly terminate execution and lead to severe consequences, such as data loss or system crashes. Despite extensive efforts to identify potential errors during the development phase, such unanticipated errors remain a challenge to to be entirely eliminated, making the runtime mitigation measurements still indispensable to minimize their impact. Automated self-healing techniques, such as reusing existing handlers, have been investigated to reduce the loss coming through with the execution termination. However, the usability of existing methods is retained by their predefined heuristic rules and they fail to handle diverse runtime errors adaptively. Recently, the advent of Large Language Models (LLMs) has opened new avenues for addressing this problem. Inspired by their remarkable capabilities in understanding and generating code, we propose to deal with the runtime errors in a real-time manner using LLMs. Specifically, we propose Healer, the first LLM-assisted self-healing framework for handling runtime errors. When an unhandled runtime error occurs, Healer will be activated to generate a piece of error-handling code with the help of its internal LLM and the code will be executed inside the runtime environment owned by the framework to obtain a rectified program state from which the program should continue its execution. Our exploratory study evaluates the performance of Healer using four different code benchmarks and three state-of-the-art LLMs, GPT-3.5, GPT-4, and CodeQwen-7B. Results show that, without the need for any fine-tuning, GPT-4 can successfully help programs recover from 72.8% of runtime errors, highlighting the potential of LLMs in handling runtime errors.
Software Engineering,Artificial Intelligence,Cryptography and Security
What problem does this paper attempt to address?
The problem that this paper attempts to solve is the serious impact of unforeseen runtime errors on software systems. Specifically, due to the lack of predefined handlers, such errors may cause the program to terminate suddenly and lead to serious consequences such as data loss or system crashes. Although a great deal of effort has been made during the development stage to identify potential errors, completely eliminating all unforeseen runtime errors remains a huge challenge. Therefore, it becomes crucial to take effective mitigation measures at runtime to minimize the impact of these errors. ### Main Problem Summary 1. **Unforeseen Runtime Errors**: These errors, due to the lack of predefined handlers, may cause the program to terminate suddenly and result in serious consequences. 2. **Limitations of Existing Methods**: Existing automated self - healing techniques rely on predefined heuristic rules and are unable to adapt to a wide variety of runtime errors. 3. **Requirement for Real - Time Processing**: A method that can respond immediately and repair when an error occurs is needed to ensure that the program can continue to run normally. ### Solution Proposed in the Paper The paper proposes a framework named Healer, which utilizes large - language models (LLMs) to achieve real - time, adaptive runtime error handling. The main features of Healer are as follows: - **Generate Handling Code in Real - Time**: When an unforeseen runtime error occurs, Healer will generate the corresponding error - handling code according to the error context (such as the error message and the program state). - **Dynamic Adaptability**: Through the understanding ability of LLMs, Healer can provide personalized solutions for each unforeseen runtime error, rather than just relying on predefined rules. - **Low Latency and High Efficiency**: The experimental results show that the delay introduced by Healer in normal code execution is very small (less than 1 millisecond per program), and the additional overhead in handling errors is also acceptable (the time for LLM to generate handling code is less than 4 seconds). ### Why Choose LLMs? LLMs have strong code - understanding and - generating capabilities. They can generate appropriate error - handling code in real - time based on understanding the error context. Compared with traditional rule - based self - healing methods, LLMs can more flexibly deal with various different runtime errors, thus improving the self - healing ability and robustness of the system. ### Experimental Verification To evaluate the effectiveness of Healer, the researchers used four different code benchmark test sets and three state - of - the - art LLMs (GPT - 3.5, GPT - 4, and CodeQwen - 7B). The experimental results show that GPT - 4 successfully helped the program recover from 72.8% of runtime errors without any fine - tuning, demonstrating the great potential of LLMs in handling runtime errors. ### Summary The main contribution of this paper is that it first proposes a new idea of using LLMs to handle runtime errors, designs an LLM - assisted runtime error - handling framework named Healer, and proves its feasibility through experiments. This not only opens up a new direction for the research of self - healing software systems but also provides a new technical approach for improving software reliability and robustness.