Open-domain question answering (QA) is known to involve several underlying
knowledge and reasoning challenges, but are models actually learning such
knowledge when trained on benchmark tasks? To investigate this, we introduce
several new challenge tasks that probe whether state-of-the-art QA models have
general knowledge about word definitions and general taxonomic reasoning, both
of which are fundamental to more complex forms of reasoning and are widespread
in benchmark datasets. As an alternative to expensive crowd-sourcing, we
introduce a methodology for automatically building datasets from various types
of expert knowledge (e.g., knowledge graphs and lexical taxonomies), allowing
for systematic control over the resulting probes and for a more comprehensive
evaluation. We find automatically constructing probes to be vulnerable to
annotation artifacts, which we carefully control for. Our evaluation confirms
that transformer-based QA models are already predisposed to recognize certain
types of structural lexical knowledge. However, it also reveals a more nuanced
picture: their performance degrades substantially with even a slight increase
in the number of hops in the underlying taxonomic hierarchy, or as more
challenging distractor candidate answers are introduced. Further, even when
these models succeed at the standard instance-level evaluation, they leave much
room for improvement when assessed at the level of clusters of semantically
connected probes (e.g., all Isa questions about a concept).

通过引入几个新的挑战任务，探索了目前最先进的 QA 模型是否具有有关单词定义和词汇推理的通用知识，结果表明，基于 Transformer 的 QA 模型已经更容易识别某些结构词汇知识，但在分类结构的层次增加或更具挑战性的干扰先决条件问题涉及到的问题上存在较大的改进空间。