Recent work on Open Domain Question Answering has shown that there is a large
discrepancy in model performance between novel test questions and those that
largely overlap with training questions. However, it is unclear which aspects
of novel questions make them challenging. Drawing upon studies on systematic
generalization, we introduce and annotate questions according to three
categories that measure different levels and kinds of generalization: training
set overlap, compositional generalization (comp-gen), and novel-entity
generalization (novel-entity). When evaluating six popular parametric and
non-parametric models, we find that for the established Natural Questions and
TriviaQA datasets, even the strongest model performance for
comp-gen/novel-entity is 13.1/5.4% and 9.6/1.5% lower compared to that for the
full test set -- indicating the challenge posed by these types of questions.
Furthermore, we show that whilst non-parametric models can handle questions
containing novel entities relatively well, they struggle with those requiring
compositional generalization. Lastly, we find that key question difficulty
factors are: cascading errors from the retrieval component, frequency of
question pattern, and frequency of the entity.

研究了开放域问答中新颖测试问题和训练问题之间的性能差异，并通过三个不同级别和类型的一般化问题的类别进行了评估，发现目前的最强模型在复合一般化和新实体一般化方面的性能比全测试集差 13.1％和 5.4％。此外， 本文还表明虽然非参数模型可以相对良好地处理包含新颖实体的问题，但对于需要合成一般化的问题，则非常困难。