CodeGeeX: A Pre-Trained Model for Code Generation with Multilingual Benchmarking on HumanEval-X

Qinkai Zheng,Xiao Xia,Xu Zou,Yuxiao Dong,Shan Wang,Yufei Xue,Zihan Wang,Lei Shen,Andi Wang,Yang Li,Teng Su,Zhilin Yang,Jie Tang
2024-07-10
Abstract:Large pre-trained code generation models, such as OpenAI Codex, can generate syntax- and function-correct code, making the coding of programmers more productive and our pursuit of artificial general intelligence closer. In this paper, we introduce CodeGeeX, a multilingual model with 13 billion parameters for code generation. CodeGeeX is pre-trained on 850 billion tokens of 23 programming languages as of June 2022. Our extensive experiments suggest that CodeGeeX outperforms multilingual code models of similar scale for both the tasks of code generation and translation on HumanEval-X. Building upon HumanEval (Python only), we develop the HumanEval-X benchmark for evaluating multilingual models by hand-writing the solutions in C++, Java, JavaScript, and Go. In addition, we build CodeGeeX-based extensions on Visual Studio Code, JetBrains, and Cloud Studio, generating 4.7 billion tokens for tens of thousands of active users per week. Our user study demonstrates that CodeGeeX can help to increase coding efficiency for 83.4% of its users. Finally, CodeGeeX is publicly accessible and in Sep. 2022, we open-sourced its code, model weights (the version of 850B tokens), API, extensions, and HumanEval-X at <a class="link-external link-https" href="https://github.com/THUDM/CodeGeeX" rel="external noopener nofollow">this https URL</a>.
Machine Learning,Artificial Intelligence,Software Engineering
What problem does this paper attempt to address?
### Problems the Paper Aims to Solve The paper primarily addresses the following issues: 1. **Developing a Multilingual Code Generation Model**: The paper introduces CodeGeeX, a multilingual code generation model with 13 billion parameters. This model has been pre-trained on 23 programming languages and performs excellently in code generation and translation tasks. 2. **Building a Multilingual Code Generation Benchmark**: To evaluate the performance of multilingual code generation models, the authors developed the HumanEval-X benchmark. This benchmark includes 164 programming problems, covering five major languages (C++, Java, JavaScript, Go, and Python), totaling 820 problem-solution pairs. 3. **Improving Programming Efficiency**: By building CodeGeeX plugins in integrated development environments (IDEs) such as Visual Studio Code, JetBrains, and Tencent Cloud Studio, the authors demonstrate that the model can significantly improve users' programming efficiency. User surveys indicate that 83.4% of users believe CodeGeeX has enhanced their programming efficiency. 4. **Verification of Functional Correctness**: Unlike traditional methods that only use string similarity metrics (such as BLEU), the HumanEval-X benchmark verifies the functional correctness of the generated code by actually running test cases, thereby more accurately reflecting the quality of the code. In summary, the paper aims to develop a multilingual code generation model and demonstrate its superior performance through comprehensive experiments and benchmarks to improve programming efficiency and code quality.