The predictions of question answering (QA)systems are typically evaluated
against manually annotated finite sets of one or more answers. This leads to a
coverage limitation that results in underestimating the true performance of
systems, and is typically addressed by extending over exact match (EM) with
pre-defined rules or with the token-level F1 measure. In this paper, we present
the first systematic conceptual and data-driven analysis to examine the
shortcomings of token-level equivalence measures.
To this end, we define the asymmetric notion of answer equivalence (AE),
accepting answers that are equivalent to or improve over the reference, and
publish over 23k human judgments for candidates produced by multiple QA systems
on SQuAD. Through a careful analysis of this data, we reveal and quantify
several concrete limitations of the F1 measure, such as a false impression of
graduality, or missing dependence on the question.
Since collecting AE annotations for each evaluated model is expensive, we
learn a BERT matching (BEM) measure to approximate this task. Being a simpler
task than QA, we find BEM to provide significantly better AE approximations
than F1, and to more accurately reflect the performance of systems.
Finally, we demonstrate the practical utility of AE and BEM on the concrete
application of minimal accurate prediction sets, reducing the number of
required answers by up to x2.6.

这篇研究论文通过定量分析 token-level 等价度量的局限性，提出了一个不对称的答案等价度概念，结合 BERT matching 算法，其结果比传统的 F1 方法更为准确，可用于问题回答的实际应用中来找到最小准确预测集。