Retrieval-Augmented Generation (RAG) is a technique that enhances the
capabilities of large language models (LLMs) by incorporating external
knowledge sources. This method addresses common LLM limitations, including
outdated information and the tendency to produce inaccurate "hallucinated"
content. However, the evaluation of RAG systems is challenging, as existing
benchmarks are limited in scope and diversity. Most of the current benchmarks
predominantly assess question-answering applications, overlooking the broader
spectrum of situations where RAG could prove advantageous. Moreover, they only
evaluate the performance of the LLM component of the RAG pipeline in the
experiments, and neglect the influence of the retrieval component and the
external knowledge database. To address these issues, this paper constructs a
large-scale and more comprehensive benchmark, and evaluates all the components
of RAG systems in various RAG application scenarios. Specifically, we have
categorized the range of RAG applications into four distinct types-Create,
Read, Update, and Delete (CRUD), each representing a unique use case. "Create"
refers to scenarios requiring the generation of original, varied content.
"Read" involves responding to intricate questions in knowledge-intensive
situations. "Update" focuses on revising and rectifying inaccuracies or
inconsistencies in pre-existing texts. "Delete" pertains to the task of
summarizing extensive texts into more concise forms. For each of these CRUD
categories, we have developed comprehensive datasets to evaluate the
performance of RAG systems. We also analyze the effects of various components
of the RAG system, such as the retriever, the context length, the knowledge
base construction, and the LLM. Finally, we provide useful insights for
optimizing the RAG technology for different scenarios.

Retrieval-Augmented Generation (RAG) 技术通过整合外部知识源提升了大型语言模型 (LLM) 的能力，克服了 LLM 存在的过时信息和生成不准确 “幻象” 内容的问题。本文构建了一个大规模且更全面的基准测试，评估了 RAG 系统的所有组成部分在不同应用场景中的性能，并对 RAG 技术在不同场景下的优化提供了有用的见解。