We develop a unified system to answer directly from text open-domain
questions that may require a varying number of retrieval steps. We employ a
single multi-task transformer model to perform all the necessary subtasks --
retrieving supporting facts, reranking them, and predicting the answer from all
retrieved documents -- in an iterative fashion. We avoid crucial assumptions of
previous work that do not transfer well to real-world settings, including
exploiting knowledge of the fixed number of retrieval steps required to answer
each question or using structured metadata like knowledge bases or web links
that have limited availability. Instead, we design a system that can answer
open-domain questions on any text collection without prior knowledge of
reasoning complexity. To emulate this setting, we construct a new benchmark,
called BeerQA, by combining existing one- and two-step datasets with a new
collection of 530 questions that require three Wikipedia pages to answer,
unifying Wikipedia corpora versions in the process. We show that our model
demonstrates competitive performance on both existing benchmarks and this new
benchmark. We make the new benchmark available at this https URL

通过使用单个多任务 transformer 模型，我们设计了一个统一的系统，可以迭代地检索支持事实，重新排序它们，并从所有检索的文档中预测答案，从而回答各种需要不同数量的检索步骤的开放领域问题。我们构建了一个新的基准测试 BeerQA，并证明我们的模型在现有基准测试和这个新基准测试上都表现出有竞争力的表现。

从文本中回答具有不同推理步骤的开放领域问题

Answering Open-Domain Questions of Varying Reasoning Steps from Text

Has there been real progress in multi-hop question-answering? Models often
exploit dataset artifacts to produce correct answers, without connecting
information across multiple supporting facts. This limits our ability to
measure true progress and defeats the purpose of building multi-hop QA
datasets. We make three contributions towards addressing this. First, we
formalize such undesirable behavior as disconnected reasoning across subsets of
supporting facts. This allows developing a model-agnostic probe for measuring
how much any model can cheat via disconnected reasoning. Second, using a notion
of \emph{contrastive support sufficiency}, we introduce an automatic
transformation of existing datasets that reduces the amount of disconnected
reasoning. Third, our experiments suggest that there hasn't been much progress
in multi-hop QA in the reading comprehension setting. For a recent large-scale
model (XLNet), we show that only 18 points out of its answer F1 score of 72 on
HotpotQA are obtained through multifact reasoning, roughly the same as that of
a simpler RNN baseline. Our transformation substantially reduces disconnected
reasoning (19 points in answer F1). It is complementary to adversarial
approaches, yielding further reductions in conjunction.

对多跳问题回答中的断开推理进行了形式化描述，并提出了一种自动转换现有数据集的方法来减少断开推理，表明在阅读理解设置中，快速大规模模型也存在重大局限性。