Large Language Models (LLMs) have greatly advanced code auto-completion
systems, with a potential for substantial productivity enhancements for
developers. However, current benchmarks mainly focus on single-file tasks,
leaving an assessment gap for more complex, real-world, multi-file programming
scenarios. To fill this gap, we introduce RepoBench, a new benchmark
specifically designed for evaluating repository-level code auto-completion
systems. RepoBench consists of three interconnected evaluation tasks:
RepoBench-R (Retrieval), RepoBench-C (Code Completion), and RepoBench-P
(Pipeline). Each task respectively measures the system's ability to retrieve
the most relevant code snippets from other files as cross-file context, predict
the next line of code with cross-file and in-file context, and handle complex
tasks that require a combination of both retrieval and next-line prediction.
RepoBench aims to facilitate a more complete comparison of performance and
encouraging continuous improvement in auto-completion systems. RepoBench is
publicly available at this https URL

该研究介绍一种名为 RepoBench 的新基准，旨在评估特定于存储库的代码自动完成系统，该基准包含三个相互关联的评估任务：RepoBench-R（检索），RepoBench-C（代码完成）和 RepoBench-P（管道），每个任务分别衡量系统从其他文件中检索最相关的代码片段的能力作为跨文件上下文，使用跨文件和文件内上下文预测代码的下一行以及处理需要检索和下一行预测组合的复杂任务。