Evaluating retrieval-augmented generation (RAG) presents challenges,
particularly for retrieval models within these systems. Traditional end-to-end
evaluation methods are computationally expensive. Furthermore, evaluation of
the retrieval model's performance based on query-document relevance labels
shows a small correlation with the RAG system's downstream performance. We
propose a novel evaluation approach, eRAG, where each document in the retrieval
list is individually utilized by the large language model within the RAG
system. The output generated for each document is then evaluated based on the
downstream task ground truth labels. In this manner, the downstream performance
for each document serves as its relevance label. We employ various downstream
task metrics to obtain document-level annotations and aggregate them using
set-based or ranking metrics. Extensive experiments on a wide range of datasets
demonstrate that eRAG achieves a higher correlation with downstream RAG
performance compared to baseline methods, with improvements in Kendall's $\tau$
correlation ranging from 0.168 to 0.494. Additionally, eRAG offers significant
computational advantages, improving runtime and consuming up to 50 times less
GPU memory than end-to-end evaluation.

评估检索增强生成（RAG）面临挑战，传统的端到端评估方法计算开销高，我们提出了一种新的评估方法 eRAG，通过使用每个检索列表中的文档，基于下游任务的真实标签评估生成的输出。实验证明 eRAG 与下游 RAG 的性能呈较高相关性，并且具有显著的计算优势。