We present RepoST, a scalable method to construct environments that provide execution feedback for repository-level code generation for both training and evaluation. Unlike existing works that aim to build entire repositories for execution, which is challenging for both human and LLMs, we provide execution feedback with sandbox testing, which isolates a given target function and its dependencies to a separate script for testing. Sandbox testing reduces the complexity of external dependencies and enables constructing environments at a large scale. We use our method to construct RepoST-Train, a large-scale train set with 7,415 functions from 832 repositories. Training with the execution feedback provided by RepoST-Train leads to a performance gain of 5.5% Pass@1 on HumanEval and 3.5% Pass@1 on RepoEval. We also build an evaluation dataset, RepoST-Eval, and benchmark 12 code generation models.

本研究解决了现有方法在执行完整仓库时的复杂性问题，提出了一种可扩展的RepoST方法，通过沙箱测试提供执行反馈，以构建针对特定函数的测试环境。关键发现是，通过RepoST-Train构建的大规模训练集显著提升了代码生成模型的性能，表明该方法在代码生成领域具有重要的实际应用价值。

RepoST：可扩展的仓库级编码环境构建与沙箱测试