Abstract:Recently, an increasing number of AI-driven programming assistants powered by code LLMs have been integrated into various real-world software development environments, significantly boosting developer productivity. However, existing code generation benchmarks primarily focus on general-purpose scenarios, leaving the code generation performance of LLMs for specific application domains largely unknown. In this paper, we introduce a new benchmark, MultiCodeBench, to fill this gap. MultiCodeBench comprises 2,400 programming tasks, covering 12 popular software development domains and 15 programming languages. Specifically, we perform in-depth research to identify these 12 application domains. Given that each domain may involve multiple technical frameworks, and that different frameworks present distinct challenges in the coding process, we categorize the commonly used frameworks and platforms within each domain. We then sample programming problems from GitHub repositories related to these subdomains. To ensure the quality of the tasks and mitigate data leakage issues, we invite annotators to rewrite the docstrings for each task in MultiCodeBench. Additionally, we build a static analysis-based dependency parsing tool to extract the dependencies in the ground truth for each task, enabling deeper performance analysis. Through extensive experiments on MultiCodeBench with eleven representative mainstream LLMs, we reveal the code generation performance of the LLMs across different application domains, providing practical insights for developers in downstream fields when selecting LLMs. Furthermore, we analyze the reasons behind the models' failures in completing software application development tasks, offering guidance for model developers to enhance domain-specific code generation capabilities.

What problem does this paper attempt to address?

The problem that this paper attempts to solve is: **evaluate the code - generation performance of large - language models (LLMs) in different application domains, filling the gap in existing benchmarks that mainly focus on general scenarios while ignoring specific application domains**. Specifically, although existing code - generation benchmarks (such as HumanEval, MBPP, etc.) can effectively evaluate the general code - generation ability of LLMs, they know very little about the code - generation performance in specific software application domains. Different application domains involve different programming languages, specific requirements, third - party libraries and development platforms, all of which will pose unique requirements and challenges to LLMs in code - generation tasks in specific domains. To address this problem, the authors introduced a new benchmark - **MultiCodeBench**, which aims to evaluate the code - generation performance of LLMs in 12 specific application domains and 15 programming languages. Through this benchmark, the authors hope to reveal the performance of LLMs in different application domains, provide practical guidance for developers to choose LLMs suitable for their domains, and provide directions for model developers to improve the code - generation ability of LLMs in specific domains. ### Main contributions: 1. **Introducing MultiCodeBench**: a new code - generation benchmark that covers 12 specific application domains and 15 programming languages. 2. **Revealing the code - generation performance of mainstream LLMs in specific application domains through extensive experiments**, providing constructive suggestions and practical insights. 3. **Summarizing the reasons for the poor performance of LLMs in specific - domain code - generation tasks through extensive error analysis**, providing directions for improving the capabilities of LLMs. ### Construction process of MultiCodeBench: 1. **Investigation of specific application domains**: Identify 12 currently popular application domains by analyzing posts on technology websites, developer communities and blogs since January 1, 2020. 2. **Selection of high - quality projects**: Select high - quality open - source projects based on the technical frameworks and development platforms of each sub - domain. 3. **Sampling of programming problems**: Manually screen highly domain - specific programming problems from the selected projects. 4. **Manual annotation of docstrings and dependency resolution**: Invite professional developers to rewrite the docstrings of each programming problem and use static analysis tools to resolve dependencies, ensuring the quality of tasks and reducing data leakage problems. Through these efforts, MultiCodeBench provides a comprehensive and detailed benchmark for evaluating the code - generation performance of LLMs in different application domains.

How Well Do LLMs Generate Code for Different Application Domains? Benchmark and Evaluation

BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructions

EvoCodeBench: An Evolving Code Generation Benchmark with Domain-Specific Evaluations

DOMAINEVAL: An Auto-Constructed Benchmark for Multi-Domain Code Generation

EvoCodeBench: An Evolving Code Generation Benchmark Aligned with Real-World Code Repositories

CodeScope: An Execution-based Multilingual Multitask Multidimensional Benchmark for Evaluating LLMs on Code Understanding and Generation

Evaluating Large Language Models in Class-Level Code Generation

A Survey of Large Language Models for Code: Evolution, Benchmarking, and Future Trends

On the Effectiveness of Large Language Models in Domain-Specific Code Generation

Evaluating and Aligning CodeLLMs on Human Preference

Top Leaderboard Ranking = Top Coding Proficiency, Always? EvoEval: Evolving Coding Benchmarks via LLM

DevEval: Evaluating Code Generation in Practical Software Projects

CodeApex: A Bilingual Programming Evaluation Benchmark for Large Language Models

What's Wrong with Your Code Generated by Large Language Models? An Extensive Study

Teaching Code LLMs to Use Autocompletion Tools in Repository-Level Code Generation

A Comprehensive Framework for Evaluating API-oriented Code Generation in Large Language Models

Unseen Horizons: Unveiling the Real Capability of LLM Code Generation Beyond the Familiar

ML-Bench: Evaluating Large Language Models and Agents for Machine Learning Tasks on Repository-Level Code

CodeAgent: Enhancing Code Generation with Tool-Integrated Agent Systems for Real-World Repo-level Coding Challenges

On Evaluating the Efficiency of Source Code Generated by LLMs

Towards more realistic evaluation of LLM-based code generation: an experimental study and beyond