The evaluation of question answering models compares ground-truth annotations with model predictions. However, as of today, this comparison is mostly lexical-based and therefore misses out on answers that have no lexical overlap but are still semantically similar, thus treating correct answers as false. This underestimation of the true performance of models hinders user acceptance in applications and complicates a fair comparison of different models. Therefore, there is a need for an evaluation metric that is based on semantics instead of pure string similarity. In this short paper, we present SAS, a cross-encoder-based metric for the estimation of semantic answer similarity, and compare it to seven existing metrics. To this end, we create an English and a German three-way annotated evaluation dataset containing pairs of answers along with human judgment of their semantic similarity, which we release along with an implementation of the SAS metric and the experiments. We find that semantic similarity metrics based on recent transformer models correlate much better with human judgment than traditional lexical similarity metrics on our two newly created datasets and one dataset from related work.

本论文提出了一种基于交叉编码器的语义答案相似度估计评估度量标准（SAS），与七种现有度量标准进行比较，并通过人类判断的语义相似性评估数据集检验其性能。结果表明，基于最近的Transformer模型的语义相似度度量标准与传统的词汇相似度度量标准在新创建的数据集和相关工作的一个数据集上的人类评价更加相关。通过语义相似性度量标准可以提高问答模型评测的准确性。

用于评估问答模型的语义答案相似度