While language models (LMs) have proven remarkably adept at generating code,
many programs are challenging for LMs to generate using their parametric
knowledge alone. Providing external contexts such as library documentation can
facilitate generating accurate and functional code. Despite the success of
retrieval-augmented generation (RAG) in various text-oriented tasks, its
potential for improving code generation remains under-explored. In this work,
we conduct a systematic, large-scale analysis by asking: in what scenarios can
retrieval benefit code generation models? and what challenges remain? We first
curate a comprehensive evaluation benchmark, CodeRAG-Bench, encompassing three
categories of code generation tasks, including basic programming, open-domain,
and repository-level problems. We aggregate documents from five sources for
models to retrieve contexts: competition solutions, online tutorials, library
documentation, StackOverflow posts, and GitHub repositories. We examine
top-performing models on CodeRAG-Bench by providing contexts retrieved from one
or multiple sources. While notable gains are made in final code generation by
retrieving high-quality contexts across various settings, our analysis reveals
room for improvement -- current retrievers still struggle to fetch useful
contexts especially with limited lexical overlap, and generators fail to
improve with limited context lengths or abilities to integrate additional
contexts. We hope CodeRAG-Bench serves as an effective testbed to encourage
further development of advanced code-oriented RAG methods.

研究发现传统的语言模型在生成代码方面有一定困难，通过提供外部上下文信息如库文档可以改善代码生成效果。通过系统性的大规模分析，发现在何种场景下使用检索增强生成模型能够对代码生成产生帮助，并且存在改进的空间。通过创建了一个综合评估基准，CodeRAG-Bench，涵盖了基本编程、开放域和代码库级别的代码生成任务，实现从五个来源中检索上下文，结果显示在不同设置中通过检索高质量上下文能够提升最终代码生成效果，同时也揭示了当前检索器在词汇重叠有限的情况下仍然难以获取有用的上下文，并且生成器在上下文长度有限或者无法整合额外上下文的能力上存在不足。希望 CodeRAG-Bench 能够成为有效的测试平台，促进进一步开发针对代码导向的检索增强生成方法。

CodeRAG-Bench：检索能增强代码生成吗？

CodeRAG-Bench: Can Retrieval Augment Code Generation?

Recent advancements in code-fluent Large Language Models (LLMs) enabled the
research on repository-level code editing. In such tasks, the model navigates
and modifies the entire codebase of a project according to request. Hence, such
tasks require efficient context retrieval, i.e., navigating vast codebases to
gather relevant context. Despite the recognized importance of context
retrieval, existing studies tend to approach repository-level coding tasks in
an end-to-end manner, rendering the impact of individual components within
these complicated systems unclear. In this work, we decouple the task of
context retrieval from the other components of the repository-level code
editing pipelines. We lay the groundwork to define the strengths and weaknesses
of this component and the role that reasoning plays in it by conducting
experiments that focus solely on context retrieval. We conclude that while the
reasoning helps to improve the precision of the gathered context, it still
lacks the ability to identify its sufficiency. We also outline the ultimate
role of the specialized tools in the process of context gathering. The code
supplementing this paper is available at
this https URL

这项研究探讨了使用代码流畅的大型语言模型（LLMs）进行存储库级代码编辑的最新进展，着重解耦上下文检索的任务以评估其组件的优劣，并指出推理在上下文获取中提高准确性的作用，但仍无法确定其充分性及专用工具在该过程中的最终角色。

关于仓库级代码编辑中推理的重要性

On The Importance of Reasoning for Context Retrieval in Repository-Level  Code Editing

Long-form question answering (LFQA) poses a challenge as it involves
generating detailed answers in the form of paragraphs, which go beyond simple
yes/no responses or short factual answers. While existing QA models excel in
questions with concise answers, LFQA requires handling multiple topics and
their intricate relationships, demanding comprehensive explanations. Previous
attempts at LFQA focused on generating long-form answers by utilizing relevant
contexts from a corpus, relying solely on the question itself. However, they
overlooked the possibility that the question alone might not provide sufficient
information to identify the relevant contexts. Additionally, generating
detailed long-form answers often entails aggregating knowledge from diverse
sources. To address these limitations, we propose an LFQA model with iterative
Planning, Retrieval, and Generation. This iterative process continues until a
complete answer is generated for the given question. From an extensive
experiment on both an open domain and a technical domain QA dataset, we find
that our model outperforms the state-of-the-art models on various textual and
factual metrics for the LFQA task.

提出了一种具有迭代规划、检索和生成过程的长篇问答（Long-form question answering，LFQA）模型，该模型通过迭代过程直到生成给定问题的完整答案，解决了既有模型在处理复杂问题和知识聚合方面的限制，并在各种文本和实际指标上优于现有模型。