The integration of Large Language Models (LLMs) into Development Environments
(IDEs) has become a focal point in modern software development. LLMs such as
OpenAI GPT-3.5/4 and Code Llama offer the potential to significantly augment
developer productivity by serving as intelligent, chat-driven programming
assistants. However, utilizing LLMs out of the box is unlikely to be optimal
for any given scenario. Rather, each system requires the LLM to be honed to its
set of heuristics to ensure the best performance. In this paper, we introduce
the Copilot evaluation harness: a set of data and tools for evaluating
LLM-guided IDE interactions, covering various programming scenarios and
languages. We propose our metrics as a more robust and information-dense
evaluation than previous state of the art evaluation systems. We design and
compute both static and execution based success metrics for scenarios
encompassing a wide range of developer tasks, including code generation from
natural language (generate), documentation generation from code (doc), test
case generation (test), bug-fixing (fix), and workspace understanding and query
resolution (workspace). These success metrics are designed to evaluate the
performance of LLMs within a given IDE and its respective parameter space. Our
learnings from evaluating three common LLMs using these metrics can inform the
development and validation of future scenarios in LLM guided IDEs.

利用 Large Language Models（LLMs）提高开发人员在开发环境（IDEs）中的生产力是现代软件开发的重点。本文介绍了 Copilot 评估工具，用于评估 LLM 引导的 IDE 交互，在不同编程场景和语言中提供了更稳健和信息丰富的评估指标。通过这些指标，我们评估了三种常见 LLMs 的性能，为未来的 LLM 引导 IDE 场景的开发和验证提供了经验教训。