LDB: A Large Language Model Debugger via Verifying Runtime Execution Step-by-step

Lily Zhong,Zilong Wang,Jingbo Shang

2024-06-04

Abstract:Large language models (LLMs) are leading significant progress in code generation. Beyond one-pass code generation, recent works further integrate unit tests and program verifiers into LLMs to iteratively refine the generated programs. However, these works consider the generated programs as an indivisible entity, which falls short for LLMs in debugging the programs, especially when the programs contain complex logic flows and data operations. In contrast, when human developers debug programs, they typically set breakpoints and selectively examine runtime execution information. The execution flow and the intermediate variables play a crucial role in the debugging process, yet they are underutilized in the existing literature on code generation. In this study, we introduce Large Language Model Debugger (LDB), a novel debugging framework that enables LLMs to refine their generated programs with the runtime execution information. Specifically, LDB segments the programs into basic blocks and tracks the values of intermediate variables after each block throughout the runtime execution. This allows LLMs to concentrate on simpler code units within the overall execution flow, verify their correctness against the task description block by block, and efficiently pinpoint any potential errors. Experiments demonstrate that LDB consistently enhances the baseline performance by up to 9.8% across the HumanEval, MBPP, and TransCoder benchmarks, archiving new state-of-the-art performance in code debugging for various LLM selections.

Software Engineering,Artificial Intelligence,Computation and Language

What problem does this paper attempt to address?

The paper aims to address the debugging challenges encountered by Large Language Models (LLMs) during the code generation process. Specifically, existing methods treat the generated program as an indivisible whole and rely on post-execution feedback for debugging, which proves inadequate when dealing with complex logic flows and data operations. To solve this problem, the paper proposes LDB (Large Language Model Debugger), a novel debugging framework that incrementally verifies and improves programs generated by LLMs by tracking runtime execution information.

The main contributions of LDB are as follows:
1. **Introduction of runtime execution information**: LDB is the first to incorporate runtime execution information into the debugging process of LLMs, allowing the model to focus on simpler code units, thereby effectively pinpointing potential errors.
2. **Basic block decomposition**: By using control flow graphs to decompose the program into basic blocks, LDB tracks the intermediate variable values at the end of each basic block, enabling the verification of each code block against the task description.
3. **Experimental validation**: In three code generation benchmarks (HumanEval, MBPP, and TransCoder), LDB demonstrated its effectiveness in debugging generated programs across different LLM backends and achieved state-of-the-art performance.

Through these improvements, LDB not only enhances the accuracy of code generation but also detects errors previously missed by other methods when debugging programs generated by more powerful code generators such as GPT-4 and Reflexion, further augmenting the capability of code generation.