One type of question that is commonly found in day-to-day scenarios is
``fan-out'' questions, complex multi-hop, multi-document reasoning questions
that require finding information about a large number of entities. However,
there exist few resources to evaluate this type of question-answering
capability among large language models. To evaluate complex reasoning in LLMs
more fully, we present FanOutQA, a high-quality dataset of fan-out
question-answer pairs and human-annotated decompositions with English Wikipedia
as the knowledge base. We formulate three benchmark settings across our dataset
and benchmark 7 LLMs, including GPT-4, LLaMA 2, Claude-2.1, and Mixtral-8x7B,
finding that contemporary models still have room to improve reasoning over
inter-document dependencies in a long context. We provide our dataset and
open-source tools to run models to encourage evaluation at this https URL

通过创建 FanOutQA 数据集以及进行评估测试，研究发现当处理复杂的多文档依赖关系时，当代的语言模型如 GPT-4、LLaMA 2、Claude-2.1 和 Mixtral-8x7B 仍存在推理能力的提升空间。