Federated search systems aggregate results from multiple search engines,
selecting appropriate sources to enhance result quality and align with user
intent. With the increasing uptake of Retrieval-Augmented Generation (RAG)
pipelines, federated search can play a pivotal role in sourcing relevant
information across heterogeneous data sources to generate informed responses.
However, existing datasets, such as those developed in the past TREC FedWeb
tracks, predate the RAG paradigm shift and lack representation of modern
information retrieval challenges. To bridge this gap, we present FeB4RAG, a
novel dataset specifically designed for federated search within RAG frameworks.
This dataset, derived from 16 sub-collections of the widely used \beir
benchmarking collection, includes 790 information requests (akin to
conversational queries) tailored for chatbot applications, along with top
results returned by each resource and associated LLM-derived relevance
judgements. Additionally, to support the need for this collection, we
demonstrate the impact on response generation of a high quality federated
search system for RAG compared to a naive approach to federated search. We do
so by comparing answers generated through the RAG pipeline through a
qualitative side-by-side comparison. Our collection fosters and supports the
development and evaluation of new federated search methods, especially in the
context of RAG pipelines.

对于 RAG 框架中的联邦搜索，我们提出了 FeB4RAG 数据集，它是一个新颖的数据集，专门为联邦搜索而设计，包括 790 个针对聊天机器人应用的信息请求以及与每个资源返回的排名前几个结果和相关的 LLM 推导的相关性判断。通过定性的并排比较，我们演示了高质量联邦搜索系统对于 RAG 中响应生成的影响，相比于简单的联邦搜索方法。