Nowadays, the fields of code and natural language processing are evolving
rapidly. In particular, models become better at processing long context windows
- supported context sizes have increased by orders of magnitude over the last
few years. However, there is a shortage of benchmarks for code processing that
go beyond a single file of context, while the most popular ones are limited to
a single method. With this work, we aim to close this gap by introducing Long
Code Arena, a suite of six benchmarks for code processing tasks that require
project-wide context. These tasks cover different aspects of code processing:
library-based code generation, CI builds repair, project-level code completion,
commit message generation, bug localization, and module summarization. For each
task, we provide a manually verified dataset for testing, an evaluation suite,
and open-source baseline solutions based on popular LLMs to showcase the usage
of the dataset and to simplify adoption by other researchers. We publish the
benchmark page on HuggingFace Spaces with the leaderboard, links to HuggingFace
Hub for all the datasets, and link to the GitHub repository with baselines:
this https URL

通过引入 Long Code Arena 作为一套六个代码处理任务的基准，我们旨在填补超出单个上下文文件的代码处理基准的空白。这些任务涵盖了代码处理的不同方面：基于库的代码生成、CI 构建修复、项目级代码补全、提交消息生成、错误定位和模块摘要。