Effi-Code: Unleashing Code Efficiency in Language Models

Dong Huang,Guangtao Zeng,Jianbo Dai,Meng Luo,Han Weng,Yuhao Qing,Heming Cui,Zhijiang Guo,Jie M. Zhang
2024-10-19
Abstract:As the use of large language models (LLMs) for code generation becomes more prevalent in software development, it is critical to enhance both the efficiency and correctness of the generated code. Existing methods and models primarily focus on the correctness of LLM-generated code, ignoring efficiency. In this work, we present Effi-Code, an approach to enhancing code generation in LLMs that can improve both efficiency and correctness. We introduce a Self-Optimization process based on Overhead Profiling that leverages open-source LLMs to generate a high-quality dataset of correct and efficient code samples. This dataset is then used to fine-tune various LLMs. Our method involves the iterative refinement of generated code, guided by runtime performance metrics and correctness checks. Extensive experiments demonstrate that models fine-tuned on the Effi-Code show significant improvements in both code correctness and efficiency across task types. For example, the pass@1 of DeepSeek-Coder-6.7B-Instruct generated code increases from \textbf{43.3\%} to \textbf{76.8\%}, and the average execution time for the same correct tasks decreases by \textbf{30.5\%}. Effi-Code offers a scalable and generalizable approach to improving code generation in AI systems, with potential applications in software development, algorithm design, and computational problem-solving. The source code of Effi-Code was released in \url{<a class="link-external link-https" href="https://github.com/huangd1999/Effi-Code" rel="external noopener nofollow">this https URL</a>}.
Computation and Language,Software Engineering
What problem does this paper attempt to address?
The problem that this paper attempts to solve is how to simultaneously improve the efficiency and correctness of code during the code - generation process of large - language models (LLMs). Existing methods and models mainly focus on improving the correctness of code generated by LLM, while ignoring the efficiency of the code. This has led to the fact that the code generated by LLM is usually inferior to the classic solutions written by human developers in terms of execution time and memory usage. For example, in the EffiBench benchmark test, even for the code generated by the most advanced LLMs such as GPT - 4 - Turbo, its average and worst - case execution times are 1.69 times and 45.49 times that of the classic solutions respectively. To solve this problem, the author proposes EFFI - CODE, a method to enhance the code generated by LLM through a self - optimization process (based on overhead analysis). This method first collects source code from eight existing open - source datasets and conducts strict pre - processing and cleaning. Then, it evaluates the code efficiency of each task by generating test cases. After multiple optimization cycles, the generated optimized code and its related metadata form the final fine - tuning dataset EFFI - CODE. This dataset is used to fine - tune various LLMs to generate more efficient and correct code. The experimental results show that the models fine - tuned with EFFI - CODE not only improve the correctness of the code in multiple benchmark tests such as HumanEval and EffiBench (for example, the pass@1 of DeepSeek - Coder - 6.7B - Instruct is increased from 43.3% to 76.8%), but also significantly reduce the average execution time of the same correct tasks (by 30.5%). These results prove the effectiveness of EFFI - CODE in improving the efficiency of code generation.