A significant amount of research is focused on developing and evaluating
large language models for a variety of code synthesis tasks. These include
synthesizing code from natural language instructions, synthesizing tests from
code, and synthesizing explanations of code. In contrast, the behavior of
instructional code editing with LLMs is understudied. These are tasks in which
the model is instructed to update a block of code provided in a prompt. The
editing instruction may ask for a feature to added or removed, describe a bug
and ask for a fix, ask for a different kind of solution, or many other common
code editing tasks.
We introduce a carefully crafted benchmark of code editing tasks and use it
evaluate several cutting edge LLMs. Our evaluation exposes a significant gap
between the capabilities of state-of-the-art open and closed models. For
example, even GPT-3.5-Turbo is 8.8% better than the best open model at editing
code.
We also introduce a new, carefully curated, permissively licensed training
set of code edits coupled with natural language instructions. Using this
training set, we show that we can fine-tune open Code LLMs to significantly
improve their code editing capabilities.

通过一个仔细设计的代码编辑任务基准以及结合自然语言指令的训练集，我们评估了几个最先进的大型语言模型，并揭示了现有开源和闭源模型之间的显著差距。同时，我们展示了通过精细调优开源代码语言模型可以显著提高其代码编辑能力。