Prior work has uncovered a set of common problems in state-of-the-art
context-based question answering (QA) systems: a lack of attention to the
context when the latter conflicts with a model's parametric knowledge, little
robustness to noise, and a lack of consistency with their answers. However,
most prior work focus on one or two of those problems in isolation, which makes
it difficult to see trends across them. We aim to close this gap, by first
outlining a set of -- previously discussed as well as novel -- desiderata for
QA models. We then survey relevant analysis and methods papers to provide an
overview of the state of the field. The second part of our work presents
experiments where we evaluate 15 QA systems on 5 datasets according to all
desiderata at once. We find many novel trends, including (1) systems that are
less susceptible to noise are not necessarily more consistent with their
answers when given irrelevant context; (2) most systems that are more
susceptible to noise are more likely to correctly answer according to a context
that conflicts with their parametric knowledge; and (3) the combination of
conflicting knowledge and noise can reduce system performance by up to 96%. As
such, our desiderata help increase our understanding of how these models work
and reveal potential avenues for improvements.

我们的研究针对现有的基于上下文的问答系统中存在的一系列问题，包括在模型参数知识与上下文冲突时缺乏对上下文的关注、对噪声的鲁棒性不足以及答案不一致性等。通过总结之前的一些已讨论过的以及新颖的需求，系统地评估了 15 个问答系统在 5 个数据集上的性能。实验结果表明，噪声敏感的系统在给定无关上下文的情况下答案一致性不一定更好，而噪声敏感度较高的系统更可能根据与其参数知识冲突的上下文正确回答问题。而冲突的知识和噪声的结合可能会导致系统性能降低高达 96%。因此，我们的需求分析有助于增加对这些模型工作原理的理解，并揭示了潜在的改进方向。