CodeScope: An Execution-based Multilingual Multitask Multidimensional Benchmark for Evaluating LLMs on Code Understanding and Generation

Weixiang Yan,Haitian Liu,Yunkun Wang,Yunzhe Li,Qian Chen,Wen Wang,Tingyu Lin,Weishan Zhao,Li Zhu,Hari Sundaram,Shuiguang Deng

2024-06-07

Abstract:Large Language Models (LLMs) have demonstrated remarkable performance on assisting humans in programming and facilitating programming automation. However, existing benchmarks for evaluating the code understanding and generation capacities of LLMs suffer from severe limitations. First, most benchmarks are insufficient as they focus on a narrow range of popular programming languages and specific tasks, whereas real-world software development scenarios show a critical need to implement systems with multilingual and multitask programming environments to satisfy diverse requirements. Second, most benchmarks fail to consider the actual executability and the consistency of execution results of the generated code. To bridge these gaps between existing benchmarks and expectations from practical applications, we introduce CodeScope, an execution-based, multilingual, multitask, multidimensional evaluation benchmark for comprehensively measuring LLM capabilities on coding tasks. CodeScope covers 43 programming languages and eight coding tasks. It evaluates the coding performance of LLMs from three dimensions (perspectives): length, difficulty, and efficiency. To facilitate execution-based evaluations of code generation, we develop MultiCodeEngine, an automated code execution engine that supports 14 programming languages. Finally, we systematically evaluate and analyze eight mainstream LLMs and demonstrate the superior breadth and challenges of CodeScope for evaluating LLMs on code understanding and generation tasks compared to other benchmarks. The CodeScope benchmark and code are publicly available at <a class="link-external link-https" href="https://github.com/WeixiangYAN/CodeScope" rel="external noopener nofollow">this https URL</a>.

Computation and Language,Artificial Intelligence,Software Engineering

What problem does this paper attempt to address?

This paper proposes a new benchmark called CodeScope to evaluate the capabilities of large language models (LLMs) in code understanding and generation tasks. Existing benchmarks have three main issues: narrow focus (usually on a few popular programming languages and specific tasks), lack of consideration for code feasibility and consistency, and reliance on superficial similarity metrics. CodeScope is an executable-based, multi-language, multi-task, multi-dimensional evaluation benchmark that covers 43 programming languages and 8 programming tasks, evaluating the performance of LLMs from three aspects: length, difficulty, and efficiency. Additionally, it includes an automated code execution engine called MultiCodeEngine that supports 14 programming languages. The paper also systematically evaluates eight mainstream LLMs, demonstrating the broadness and challenges of CodeScope in evaluating LLMs in code understanding and generation tasks. The open-source link for CodeScope is provided at the end of the paper.

CodeScope: An Execution-based Multilingual Multitask Multidimensional Benchmark for Evaluating LLMs on Code Understanding and Generation

BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructions

xCodeEval: A Large Scale Multilingual Multitask Benchmark for Code Understanding, Generation, Translation and Retrieval

McEval: Massively Multilingual Code Evaluation

ComplexCodeEval: A Benchmark for Evaluating Large Code Models on More Complex Code

SpecEval: Evaluating Code Comprehension in Large Language Models via Program Specifications

CRUXEval-X: A Benchmark for Multilingual Code Reasoning, Understanding and Execution

Escalating LLM-based Code Translation Benchmarking into the Class-level Era

MMCode: Benchmarking Multimodal Large Language Models for Code Generation with Visually Rich Programming Problems

CodeEditorBench: Evaluating Code Editing Capability of Large Language Models

CodeMMLU: A Multi-Task Benchmark for Assessing Code Understanding Capabilities of CodeLLMs

CodeJudge-Eval: Can Large Language Models be Good Judges in Code Understanding?

EvoCodeBench: An Evolving Code Generation Benchmark with Domain-Specific Evaluations

Unseen Horizons: Unveiling the Real Capability of LLM Code Generation Beyond the Familiar

FullStack Bench: Evaluating LLMs as Full Stack Coders

EvoCodeBench: An Evolving Code Generation Benchmark Aligned with Real-World Code Repositories

Evaluating and Aligning CodeLLMs on Human Preference

ML-Bench: Evaluating Large Language Models and Agents for Machine Learning Tasks on Repository-Level Code

CS-Bench: A Comprehensive Benchmark for Large Language Models towards Computer Science Mastery

ML-Bench: Large Language Models Leverage Open-source Libraries for Machine Learning Tasks

CoderUJB: An Executable and Unified Java Benchmark for Practical Programming Scenarios