Large language models have achieved remarkable success on general NLP tasks,
but they may fall short for domain-specific problems. Recently, various
Retrieval-Augmented Large Language Models (RALLMs) are proposed to address this
shortcoming. However, existing evaluation tools only provide a few baselines
and evaluate them on various domains without mining the depth of domain
knowledge. In this paper, we address the challenges of evaluating RALLMs by
introducing the R-Eval toolkit, a Python toolkit designed to streamline the
evaluation of different RAG workflows in conjunction with LLMs. Our toolkit,
which supports popular built-in RAG workflows and allows for the incorporation
of customized testing data on the specific domain, is designed to be
user-friendly, modular, and extensible. We conduct an evaluation of 21 RALLMs
across three task levels and two representative domains, revealing significant
variations in the effectiveness of RALLMs across different tasks and domains.
Our analysis emphasizes the importance of considering both task and domain
requirements when choosing a RAG workflow and LLM combination. We are committed
to continuously maintaining our platform at this https URL
to facilitate both the industry and the researchers.

我们提出了 R-Eval 工具包，它是一个 Python 工具包，旨在简化对不同 RAG 工作流与 LLMs 的评估，通过在特定领域上支持流行的内置 RAG 工作流，并允许将定制化测试数据结合在一起。我们评估了 21 个 RALLMs 在三个任务级别和两个代表性领域上的表现，揭示了在不同任务和领域中 RALLMs 的效果差异的重要性，强调了在选择 RAG 工作流和 LLM 组合时考虑任务和领域要求的重要性。