Code generation benchmarks such as HumanEval are widely adopted to evaluate
LLMs' capabilities. However, after consolidating the latest 24 benchmarks, we
noticed three significant imbalances. First, imbalanced programming language.
95.8% of benchmarks involve Python, while only 5 benchmarks involve Java.
Second, imbalanced code granularity. Function-/statement-level benchmarks
account for over 83.3% of benchmarks. Only a mere handful extends to
class-/project-levels, and all are limited to Python. Third, lacking advanced
features. Existing benchmarks primarily assess basic coding skills, while
overlooking advanced Object-Oriented Programming (OOP) features (i.e.,
encapsulation, inheritance, and polymorphism).
To fill these gaps, we propose JavaBench, a project-level Java benchmark that
exercises OOP features. It comprises four Java projects with 389 methods in 106
Java classes. The test coverage is up to 92%, and JavaBench is attested by 282
undergraduate students, reaching a 90.93/100 average score (i.e., pass rate
against the test suite), ensuring the quality of documentation, code skeleton,
and tests. To better evaluate LLM's capability against JavaBench, we introduce
a systematic evaluation design covering three context settings and five
synthesis strategies at two granularities using three hierarchical metrics. Our
extensive experiment yields several interesting findings. First, we noticed
that regarding project-level Java programming, LLMs are far behind
undergraduate students (no project can be correctly completed by any studied
LLMs, and at most 41.17% Pass@5 in a more relaxed evaluation). Second, using
method signature as prompt context may strike an ideal balance for
project-level code generation. JavaBench is publicly available at
this https URL

通过比较 LML 的编码能力与学生的能力，研究发现在项目级别的 Java 编程中，LML 远远落后于学生，并提出了一种新的、覆盖广泛的评估方法。