Question answering (QA) can only make progress if we know if an answer is
correct, but for many of the most challenging and interesting QA examples,
current evaluation metrics to determine answer equivalence (AE) often do not
align with human judgments, particularly more verbose, free-form answers from
large language models (LLM). There are two challenges: a lack of data and that
models are too big: LLM-based scorers can correlate better with human judges,
but this task has only been tested on limited QA datasets, and even when
available, update of the model is limited because LLMs are large and often
expensive. We rectify both of these issues by providing clear and consistent
guidelines for evaluating AE in machine QA adopted from professional human QA
contests. We also introduce a combination of standard evaluation and a more
efficient, robust, and lightweight discriminate AE classifier-based matching
method (CFMatch, smaller than 1 MB), trained and validated to more accurately
evaluate answer correctness in accordance with adopted expert AE rules that are
more aligned with human judgments.

通过采用职业人员 QA 竞赛来评估机器 QA 中答案的正确性，结合标准评估和一种更高效、稳健、轻量级的基于偏差 AE 分类器的匹配方法（CFMatch，小于 1 MB），培训和验证以更准确地评估答案的正确性，使得 QA 能够进一步发展。

CFMatch: 自动化答案等效评估与专家判断在开放域问答中的对齐

CFMatch: Aligning Automated Answer Equivalence Evaluation with Expert  Judgments For Open-Domain Question Answering

The predictions of question answering (QA)systems are typically evaluated
against manually annotated finite sets of one or more answers. This leads to a
coverage limitation that results in underestimating the true performance of
systems, and is typically addressed by extending over exact match (EM) with
pre-defined rules or with the token-level F1 measure. In this paper, we present
the first systematic conceptual and data-driven analysis to examine the
shortcomings of token-level equivalence measures.
To this end, we define the asymmetric notion of answer equivalence (AE),
accepting answers that are equivalent to or improve over the reference, and
publish over 23k human judgments for candidates produced by multiple QA systems
on SQuAD. Through a careful analysis of this data, we reveal and quantify
several concrete limitations of the F1 measure, such as a false impression of
graduality, or missing dependence on the question.
Since collecting AE annotations for each evaluated model is expensive, we
learn a BERT matching (BEM) measure to approximate this task. Being a simpler
task than QA, we find BEM to provide significantly better AE approximations
than F1, and to more accurately reflect the performance of systems.
Finally, we demonstrate the practical utility of AE and BEM on the concrete
application of minimal accurate prediction sets, reducing the number of
required answers by up to x2.6.

这篇研究论文通过定量分析 token-level 等价度量的局限性，提出了一个不对称的答案等价度概念，结合 BERT matching 算法，其结果比传统的 F1 方法更为准确，可用于问题回答的实际应用中来找到最小准确预测集。