Although counterfactual reasoning is a fundamental aspect of intelligence,
the lack of large-scale counterfactual open-domain question-answering (QA)
benchmarks makes it difficult to evaluate and improve models on this ability.
To address this void, we introduce the first such dataset, named IfQA, where
each question is based on a counterfactual presupposition via an "if" clause.
For example, if Los Angeles was on the east coast of the U.S., what would be
the time difference between Los Angeles and Paris? Such questions require
models to go beyond retrieving direct factual knowledge from the Web: they must
identify the right information to retrieve and reason about an imagined
situation that may even go against the facts built into their parameters. The
IfQA dataset contains over 3,800 questions that were annotated annotated by
crowdworkers on relevant Wikipedia passages. Empirical analysis reveals that
the IfQA dataset is highly challenging for existing open-domain QA methods,
including supervised retrieve-then-read pipeline methods (EM score 36.2), as
well as recent few-shot approaches such as chain-of-thought prompting with
GPT-3 (EM score 27.4). The unique challenges posed by the IfQA benchmark will
push open-domain QA research on both retrieval and counterfactual reasoning
fronts.

本文介绍了第一个 counterfactual open-domain question-answering（QA）数据集（IfQA），并通过在该数据集上进行实证分析证明了如果就文本假设进行推理和检索是一项具有挑战性的任务。