The instruction-following ability of Large Language Models (LLMs) has
cultivated a class of LLM-based systems capable of approaching complex tasks
such as making edits to large code repositories. Due to the high sensitivity
and unpredictability of LLM behavior in response to changes in prompting,
robust evaluation tools are needed to drive future iteration of these systems.
We propose RES-Q, a natural language instruction-based benchmark for evaluating
$\textbf{R}$epository $\textbf{E}$diting $\textbf{S}$ystems, which consists of
100 repository editing tasks derived from real GitHub commits. Given an edit
instruction and a code repository, RES-Q evaluates an LLM system's ability to
gather information and construct an edit that satisfies the criteria set by the
instruction. We argue that evaluating LLMs in this way addresses issues with
traditional benchmarks and provides a more holistic assessment of a model's
abilities. We evaluate various state-of-the-art LLMs as language agents in a
repository-editing system built on Qurrent OS, our language agent development
software. Despite their 1% pass@1 performance difference on HumanEval, we find
Claude Sonnet 3.5 outperforms GPT-4o by 12% pass@1 on RES-Q, indicating RES-Q's
capacity to differentiate model capability as traditional benchmarks approach
saturation. We further investigate token efficiency, performance relationships
with existing benchmarks, and interesting disparities between closed and
open-source LLMs. Code and dataset are available at
this https URL

通过提出的基于自然语言指令的基准测试 RES-Q，对大型语言模型的指令遵循能力和代码仓库编辑系统进行了评估，发现模型能力存在差异，并提出了评估工具的需求。