Retrieval-Augmented Generation (RAG) has emerged as a pivotal innovation in
natural language processing, enhancing generative models by incorporating
external information retrieval. Evaluating RAG systems, however, poses distinct
challenges due to their hybrid structure and reliance on dynamic knowledge
sources. We consequently enhanced an extensive survey and proposed an analysis
framework for benchmarks of RAG systems, RAGR (Retrieval, Generation,
Additional Requirement), designed to systematically analyze RAG benchmarks by
focusing on measurable outputs and established truths. Specifically, we
scrutinize and contrast multiple quantifiable metrics of the Retrieval and
Generation component, such as relevance, accuracy, and faithfulness, of the
internal links within the current RAG evaluation methods, covering the possible
output and ground truth pairs. We also analyze the integration of additional
requirements of different works, discuss the limitations of current benchmarks,
and propose potential directions for further research to address these
shortcomings and advance the field of RAG evaluation. In conclusion, this paper
collates the challenges associated with RAG evaluation. It presents a thorough
analysis and examination of existing methodologies for RAG benchmark design
based on the proposed RGAR framework.

Retrieval-Augmented Generation (RAG) 系统的评估和分析框架（RGAR）提供了一种基于可测输出和已建立真实性来系统分析 RAG 系统基准的方法，并讨论了当前基准的局限性和进一步研究的潜在方向。