Abstract:Code generation benchmarks such as HumanEval are widely adopted to evaluate LLMs' capabilities. However, after consolidating the latest 24 benchmarks, we noticed three significant imbalances. First, imbalanced programming language. 95.8% of benchmarks involve Python, while only 5 benchmarks involve Java. Second, imbalanced code granularity. Function-/statement-level benchmarks account for over 83.3% of benchmarks. Only a mere handful extends to class-/project-levels, and all are limited to Python. Third, lacking advanced features. Existing benchmarks primarily assess basic coding skills, while overlooking advanced Object-Oriented Programming (OOP) features (i.e., encapsulation, inheritance, and polymorphism). To fill these gaps, we propose JavaBench, a project-level Java benchmark that exercises OOP features. It comprises four Java projects with 389 methods in 106 Java classes. The test coverage is up to 92%, and JavaBench is attested by 282 undergraduate students, reaching a 90.93/100 average score (i.e., pass rate against the test suite), ensuring the quality of documentation, code skeleton, and tests. To better evaluate LLM's capability against JavaBench, we introduce a systematic evaluation design covering three context settings and five synthesis strategies at two granularities using three hierarchical metrics. Our extensive experiment yields several interesting findings. First, we noticed that regarding project-level Java programming, LLMs are far behind undergraduate students (no project can be correctly completed by any studied LLMs, and at most 41.17% Pass@5 in a more relaxed evaluation). Second, using method signature as prompt context may strike an ideal balance for project-level code generation. JavaBench is publicly available at <a class="link-external link-https" href="https://github.com/java-bench/JavaBench" rel="external noopener nofollow">this https URL</a>.

Adaptive Java optimisation using instance-based learning.

Compiler Autotuning through Multiple Phase Learning

JACO: JAva Code Layout Optimizer Enabling Continuous Optimization without Pausing Application Services

An Intermediate Language Level Optimization Framework for Dynamic Binary Translation.

Dynamic Inlining Based on Continuous Monitoring

Compiler Auto-tuning through Multiple Phase Learning

Towards Compile-Time-Reducing Compiler Optimization Selection via Machine Learning

The mechanism and analysis of optimizing programs based on byte code specialization

Pattern-driven performance optimization at runtime: experiment on JEE systems.

Pattern-Based Peephole Optimizations with Java JIT Tests

Learning Performance-Improving Code Edits

Deep Parameter Optimisation

A Sample Profile-Based Optimization Method with Better Precision

Towards co-designed optimizations in parallel frameworks: A MapReduce case study

Improving productivity in large scale testing at the compiler level by changing the intermediate language from C++ to Java

DJXPerf: Identifying Memory Inefficiencies via Object-centric Profiling for Java

CoderUJB: An Executable and Unified Java Benchmark for Practical Programming Scenarios

Compiling Java just in time

Tuning adaptive computations for the performance improvement of applications in JEE server

MCompiler: A Synergistic Compilation Framework

JavaBench: A Benchmark of Object-Oriented Code Generation for Evaluating Large Language Models