Abstract:With the increasing code reasoning capabilities of existing large language models (LLMs) and breakthroughs in reasoning models like OpenAI o1 and o3, there is a growing need to develop more challenging and comprehensive benchmarks that effectively test their sophisticated competition-level coding abilities. Existing benchmarks, like LiveCodeBench and USACO, fall short due to the unavailability of private test cases, lack of support for special judges, and misaligned execution environments. To bridge this gap, we introduce CodeElo, a standardized competition-level code generation benchmark that effectively addresses all these challenges for the first time. CodeElo benchmark is mainly based on the official CodeForces platform and tries to align with the platform as much as possible. We compile the recent six months of contest problems on CodeForces with detailed information such as contest divisions, problem difficulty ratings, and problem algorithm tags. We introduce a unique judging method in which problems are submitted directly to the platform and develop a reliable Elo rating calculation system that aligns with the platform and is comparable with human participants but has lower variance. By testing on our CodeElo, we provide the Elo ratings of 30 existing popular open-source and 3 proprietary LLMs for the first time. The results show that o1-mini and QwQ-32B-Preview stand out significantly, achieving Elo ratings of 1578 and 1261, respectively, while other models struggle even with the easiest problems, placing in the lowest 25 percent among all human participants. Detailed analysis experiments are also conducted to provide insights into performance across algorithms and comparisons between using C++ and Python, which can suggest directions for future studies.

What problem does this paper attempt to address?

### What problems does this paper attempt to solve? This paper aims to solve the problem that current large - language models (LLMs) lack standardized, comprehensive, and challenging benchmark tests in the evaluation of programming competition capabilities. Specifically, some existing programming competition benchmark tests have the following deficiencies: 1. **Incomplete test cases**: Many programming competition platforms hide their test cases, resulting in existing benchmark tests being able to rely only on public or automatically generated test cases. These test cases are often small and not robust enough, and are prone to misjudgment. 2. **Lack of special judgment support**: About 30% of programming competition questions do not have a unique correct answer and require a specific judgment program (called special judgment) to evaluate the correctness of the solution, while existing benchmark tests usually do not support this special judgment. 3. **Inconsistent execution environment**: Differences in the execution efficiency of different machines will affect the results, especially in algorithm competitions, where running time is a key factor. 4. **Lack of a standard scoring system comparable to humans**: Existing benchmark tests fail to provide a standard scoring system comparable to human contestants. To solve these problems, the author proposes a new benchmark test platform - **CODE ELO**, which is based on the CodeForces platform and can effectively evaluate the performance of LLMs in programming competitions and provide Elo scores comparable to human contestants. Specific improvements include: - **Zero misjudgment**: By directly submitting the code generated by the model to the CodeForces platform for evaluation, the accuracy of the results is ensured. - **Support for special judgment**: Utilize the special judgment mechanism of the CodeForces platform to ensure accurate evaluation of complex problems. - **Consistent execution environment**: All evaluations are carried out on the same platform, avoiding differences caused by different machine performances. - **Standardized Elo scoring system**: Introduce an Elo scoring system that is consistent with the CodeForces platform but has a lower variance, which is used to evaluate and compare the performance of different models. Through these improvements, CODE ELO can measure the capabilities of LLMs in programming competitions more comprehensively and accurately, and provide valuable references for future research.

CodeElo: Benchmarking Competition-level Code Generation of LLMs with Human-comparable Elo Ratings

EvoCodeBench: An Evolving Code Generation Benchmark with Domain-Specific Evaluations

EvoCodeBench: An Evolving Code Generation Benchmark Aligned with Real-World Code Repositories

Top Leaderboard Ranking = Top Coding Proficiency, Always? EvoEval: Evolving Coding Benchmarks via LLM

Benchmarking Language Model Creativity: A Case Study on Code Generation

Evaluating and Aligning CodeLLMs on Human Preference

CRUXEval-X: A Benchmark for Multilingual Code Reasoning, Understanding and Execution

NaturalCodeBench: Examining Coding Performance Mismatch on HumanEval and Natural User Prompts

BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructions

CodeJudge-Eval: Can Large Language Models be Good Judges in Code Understanding?

LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code

InfiCoder-Eval: Systematically Evaluating the Question-Answering Capabilities of Code Large Language Models.

CodeScope: An Execution-based Multilingual Multitask Multidimensional Benchmark for Evaluating LLMs on Code Understanding and Generation

ComplexCodeEval: A Benchmark for Evaluating Large Code Models on More Complex Code

DOMAINEVAL: An Auto-Constructed Benchmark for Multi-Domain Code Generation

CodeApex: A Bilingual Programming Evaluation Benchmark for Large Language Models

How Well Do LLMs Generate Code for Different Application Domains? Benchmark and Evaluation

InfiBench: Evaluating the Question-Answering Capabilities of Code Large Language Models

CodeBenchGen: Creating Scalable Execution-based Code Generation Benchmarks

OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problems

GAOKAO-Eval: Does high scores truly reflect strong capabilities in LLMs?