McEval: Massively Multilingual Code Evaluation

Linzheng Chai,Shukai Liu,Jian Yang,Yuwei Yin,Ke Jin,Jiaheng Liu,Tao Sun,Ge Zhang,Changyu Ren,Hongcheng Guo,Zekun Wang,Boyang Wang,Xianjie Wu,Bing Wang,Tongliang Li,Liqun Yang,Sufeng Duan,Zhoujun Li

2024-06-12

Abstract:Code large language models (LLMs) have shown remarkable advances in code understanding, completion, and generation tasks. Programming benchmarks, comprised of a selection of code challenges and corresponding test cases, serve as a standard to evaluate the capability of different LLMs in such tasks. However, most existing benchmarks primarily focus on Python and are still restricted to a limited number of languages, where other languages are translated from the Python samples (e.g. MultiPL-E) degrading the data diversity. To further facilitate the research of code LLMs, we propose a massively multilingual code benchmark covering 40 programming languages (McEval) with 16K test samples, which substantially pushes the limits of code LLMs in multilingual scenarios. The benchmark contains challenging code completion, understanding, and generation evaluation tasks with finely curated massively multilingual instruction corpora McEval-Instruct. In addition, we introduce an effective multilingual coder mCoder trained on McEval-Instruct to support multilingual programming language generation. Extensive experimental results on McEval show that there is still a difficult journey between open-source models and closed-source LLMs (e.g. GPT-series models) in numerous languages. The instruction corpora, evaluation benchmark, and leaderboard are available at \url{<a class="link-external link-https" href="https://mceval.github.io/" rel="external noopener nofollow">this https URL</a>}.

Programming Languages

What problem does this paper attempt to address?

The paper aims to address the following key issues:

1. **Developing a large-scale multilingual code evaluation benchmark**: Current code evaluation benchmarks mainly focus on Python, and datasets in other languages are often translated from Python, which limits data diversity and multilingual support. Therefore, the researchers propose a large-scale multilingual code evaluation benchmark named MCEVAL, covering 40 programming languages and containing 16,000 test samples.

2. **Creating a multilingual instruction corpus**: To support multilingual programming language generation, the paper also constructs a large-scale multilingual instruction corpus named MCEVAL-INSTRUCT, which includes high-quality code snippets from 40 languages.

3. **Training a multilingual encoder MCODER**: Based on MCEVAL-INSTRUCT, the paper introduces the training process of the multilingual encoder MCODER, a powerful baseline model for multilingual programming tasks.

4. **Evaluating the performance of existing models**: The paper systematically evaluates the performance of a series of models (including open-source and closed-source models) on MCEVAL, particularly focusing on their ability to generate, interpret, and complete code tasks in a multilingual environment.

In summary, the goal of the paper is to advance the development of large language models (LLMs) for code, particularly enhancing their capabilities in multilingual scenarios, and to this end, it constructs a comprehensive set of tools and resources. By introducing tools such as MCEVAL, MCEVAL-INSTRUCT, and MCODER, the researchers aim to improve the ability of code LLMs to understand and generate code in different programming languages. Additionally, through extensive experimental validation, the paper reveals the gap between existing open-source and closed-source models in terms of multilingual support.