Retrieval Augmented Generation (RAG) systems have seen huge popularity in
augmenting Large-Language Model (LLM) outputs with domain specific and time
sensitive data. Very recently a shift is happening from simple RAG setups that
query a vector database for additional information with every user input to
more sophisticated forms of RAG. However, different concrete approaches compete
on mostly anecdotal evidence at the moment. In this paper we present a rigorous
dataset creation and evaluation workflow to quantitatively compare different
RAG strategies. We use a dataset created this way for the development and
evaluation of a boolean agent RAG setup: A system in which a LLM can decide
whether to query a vector database or not, thus saving tokens on questions that
can be answered with internal knowledge. We publish our code and generated
dataset online.

检索增强生成（RAG）系统在通过领域特定和时间敏感数据增强大型语言模型（LLM）输出方面非常受欢迎。本文提出了一种严格的数据集创建和评估工作流程，从而定量比较了不同的 RAG 策略，用于布尔代理 RAG 设置的开发和评估。我们在线发布了我们的代码和生成的数据集。