Large Language Models demonstrate the ability to solve various programming tasks, including code generation. Typically, the performance of LLMs is measured on benchmarks with small or medium-sized context windows of thousands of lines of code. At the same time, in real-world software projects, repositories can span up to millions of LoC. This paper closes this gap by contributing to the long context code generation benchmark (YABLoCo). The benchmark featured a test set of 215 functions selected from four large repositories with thousands of functions. The dataset contained metadata of functions, contexts of the functions with different levels of dependencies, docstrings, functions bodies, and call graphs for each repository. This paper presents three key aspects of the contribution. First, the benchmark aims at function body generation in large repositories in C and C++, two languages not covered by previous benchmarks. Second, the benchmark contains large repositories from 200K to 2,000K LoC. Third, we contribute a scalable evaluation pipeline for efficient computing of the target metrics and a tool for visual analysis of generated code. Overall, these three aspects allow for evaluating code generation in large repositories in C and C++.

本研究针对现有基准中对小型和中型上下文窗口的局限性，提出了长上下文代码生成基准YABLoCo。该基准包括来自四个大型存储库的215个函数的测试集，涵盖了C和C++语言，以支持在大规模代码库中的函数体生成。研究的关键发现是，该基准能够有效评估大规模代码生成的潜力，并提供可扩展的评估管道和可视化分析工具。

YABLoCo：另一个长上下文代码生成基准