Sensitivity to false assumptions (or false premises) in information-seeking
questions is critical for robust question-answering (QA) systems. Recent work
has shown that false assumptions in naturally occurring questions pose
challenges to current models, with low performance on both generative QA and
simple detection tasks (Kim et al. 2023). However, the focus of existing work
on naturally occurring questions leads to a gap in the analysis of model
behavior on the long tail of the distribution of possible questions. To this
end, we introduce Syn-(QA)$^2$, a set of two synthetically generated QA
datasets: one generated using perturbed relations from Wikidata, and the other
by perturbing HotpotQA (Yang et al. 2018). Our findings from evaluating a range
of large language models are threefold: (1) false assumptions in QA are
challenging, echoing the findings of prior work, (2) the binary detection task
is challenging even compared to the difficulty of generative QA itself,
possibly due to the linguistic structure of the problem, and (3) the detection
task is more challenging with long-tail questions compared to naturally
occurring questions, highlighting the utility of our synthetic datasets and
generation method.

通过评估一系列大型语言模型，我们的研究发现：(1) 问题回答中的错误假设具有挑战性，与以往研究结果相一致；(2) 与生成型问题回答相比，二值检测任务本身更具挑战性，可能是由于问题的语言结构；(3) 长尾问题的检测任务比自然存在的问题更具挑战性，突显了我们的合成数据集和生成方法的实用性。