Abstract:The increasing development of large language models (LLMs) in code generation has drawn significant attention among researchers. To enhance LLM-based code generation ability, current efforts are predominantly directed towards collecting high-quality datasets and leveraging diverse training technologies. However, there is a notable lack of comprehensive studies examining the limitations and boundaries of these existing methods. To bridge this gap, we conducted an extensive empirical study evaluating the performance of three leading closed-source LLMs and four popular open-source LLMs on three commonly used benchmarks. Our investigation, which evaluated the length, cyclomatic complexity and API number of the generated code, revealed that these LLMs face challenges in generating successful code for more complex problems, and tend to produce code that is shorter yet more complicated as compared to canonical solutions. Additionally, we developed a taxonomy of bugs for incorrect codes that includes three categories and 12 sub-categories, and analyze the root cause for common bug types. Furthermore, to better understand the performance of LLMs in real-world projects, we manually created a real-world benchmark comprising 140 code generation tasks. Our analysis highlights distinct differences in bug distributions between actual scenarios and existing benchmarks. Finally, we propose a novel training-free iterative method that introduces self-critique, enabling LLMs to critique and correct their generated code based on bug types and compiler feedback. Experimental results demonstrate that our approach can significantly mitigate bugs and increase the passing rate by 29.2% after two iterations, indicating substantial potential for LLMs to handle more complex problems.

Are Large Language Models Memorizing Bug Benchmarks?

Benchmarking Benchmark Leakage in Large Language Models

The Vulnerability of Language Model Benchmarks: Do They Accurately Reflect True LLM Performance?

Detecting Memorization in Large Language Models

Training on the Benchmark Is Not All You Need

Traces of Memorisation in Large Language Models for Code

What's Wrong with Your Code Generated by Large Language Models? An Extensive Study

Inadequacies of Large Language Model Benchmarks in the Era of Generative Artificial Intelligence

An evaluation on large language model outputs: Discourse and memorization

Don't Make Your LLM an Evaluation Benchmark Cheater

An Exploratory Study on Using Large Language Models for Mutation Testing

Undesirable Memorization in Large Language Models: A Survey

Understanding Memorisation in LLMs: Dynamics, Influencing Factors, and Implications

Evaluating Diverse Large Language Models for Automatic and General Bug Reproduction

DebugBench: Evaluating Debugging Capability of Large Language Models

A Careful Examination of Large Language Model Performance on Grade School Arithmetic

Inference-Time Decontamination: Reusing Leaked Benchmarks for Large Language Model Evaluation

Bugs in Large Language Models Generated Code: An Empirical Study